How to train a reinforcement learning agent from raw pixels? How would you train a reinforcement learning agent from raw pixels?

For example, if you have 3 stacked images to sense motion, then how would you pass them to neural networks to output Q-learning values?

If you pass that batch output, it would be a batch of values, so from here it is impossible to deduce which ones are the true Q-values for that state.

Currently, I am watching a YouTuber: Machine Learning with Phil, and he did it very differently. On the 13th minute, he defined a network that outputs a batch of values rather than Q-values for 6 states. In short, he outputs a matrix rather than a vector.

How would you train a reinforcement learning agent from raw pixels? For example, if you have 3 stacked images to sense motion, then how would you pass them to neural networks to output Q-learning values?

A Convolutional Neural Network (CNN) structure is a standard neural network architecture when working with 2D pixel input in reinforcement learning, and it is the technique used in the original DQN paper (see paragraphs 1 & 3 of Section 4.1 of https://arxiv.org/abs/1312.5602). CNNs typically take 3-dimensional input, where the first two dimensions are height and width of your images, and the third is rgb color. The technique in the paper was to convert each RGB frame (or image) to greyscale format (so it has only 1 color channel/dimension instead of 3) and instead use the rgb_color dimension as a frames dimension that is indexed by each stacked frame.

  Currently, I am watching a YouTuber: Machine Learning with Phil, and he did it very differently. On the 13th minute, he defined a network that outputs a batch of values rather than Q-values for 6 states. In short, he outputs a matrix rather than a vector.

Later in the tutorial series, he most likely will discuss the training of the neural network. During training, you need to find the q-values of a batch of sets of stacked frames. Specifically, each element of the batch is a set of stacked frames. In other words, a set of stacked frames is treated as a single observation, so a batch of sets of stacked frames is a batch of observations.

To find these q-values, you will perform a forward pass of the batch of observations through the neural network. A forward pass of a single observation (set of stacked frames) through the neural network yields a vector of q-values (one for each action). Thus, a forward pass of a batch of observations (batch of stacked frames) will yield a matrix of q-values (one vector of q-values for each observation (or set of stacked frames)). This technique is used because many standard neural network libraries are designed to perform a forward pass on a batch of inputs through the neural network much faster than performing a forward pass on each input separately.


How to choose a suitable threshold value for shi-tomasi corner detection algorithm? While implementing shi-tomasi corner detection algorithm i got stuck at deciding a suitable threshold for corner detection.

In shi-tomasi algorithm all those points that qualify min(λ1 , λ2) > threshold are considered as corner points. (where λ1, λ2 are eigenvalues )

My question is what is a suitable criteria to decide that threshold ?

The threshold is typically chosen empirically, so there is no exact answer.
It's dependent on how many corners you wish to select, and how strict you want the detection, which could depend on the use case, the dataset, and the block size of the algorithm. If you’re not sure what to choose for the threshold, I would suggest using a scheme relative to your images to deduce the best threshold, based on what performs best for your use case.
For example, let the criteria for selection, or scoring function, be $R$, which in the Shi-Tomasi case is $R = min(\lambda_1, \lambda_2)$. We could choose some value $q \in (0,1)$, so that the threshold becomes
$t = q\max_p{R}$,
where the max $R$ is calculated over all points $p$, an approach similar to OpenCV Good Features to Track.
You could examine the corners detected and adjust $q$ based on how stringently you wanted to select corners. If you don't want to waste the time manually fine-tuning the threshold, you could check out some automated approaches in the literature, such as Automated thresholding for low-complexity corner detection


Why does the cost function contain a 2 at the denominator? A cost function used in machine is often the following

$$C = \frac{1}{2} \| y - \hat{y} \| ^2$$

Why is there $\frac{1}{2}$ in front of the squared distance?

To simplify the derivative, probably. Otherwise there will be constant 2 in it.


Strategy of using intermediate layers of a neural network as features? There is a popular strategy of using a neural network trained on one task to produce features for another related task by "chopping off" the top of the network and sewing the bottom onto some other modeling pipeline.

Word2Vec models employ this strategy, for example.

Is there an industry-popular term for this strategy?  Are there any good resources that discuss its use in general terms?

This is an example of Transfer Learning, wherein instead of starting the learning process from scratch, a pre-trained model is used and appended with custom layers to achieve additional functionality.


Is A2C loss function taking smaller steps for larger mistakes? A2C loss is usually defined as advantage * (-log(actor_predictions)) * target where target is a one-hot vector (with some clipping/noise/etc...) with the selected target.

Does this mean that we get larger losses for smaller mistakes?

If for example the agent has predicted $\pi(a|s)=0.9$ but the advantage is negative, this would mean a larger mistake than if the agent predicted that $\pi(a|s)=0.1$, however, putting the numbers in the formula means a larger loss for the 0.1 prediction.

Assuming advantage=-1, advantage * (-log(actor_predictions)) * target would mean:

$$
-1 * (-log(0.9)) * 1 = log(0.9)=-0.045
$$
$$
-1 * (-log(0.1)) * 1 = log(0.1)=-1
$$

Is my understanding correct?

So this question is due to a misunderstanding I had with how the loss function works in A2C.

Usually loss functions are always positives, but in A2C it can be negative as well. I thought that minimizing the loss function means moving it closer to zero, but in fact reducing the loss is indifferent to the sign of the loss value and is about making the loss value smaller.

In other words, the loss for 0.1 is smaller than the loss for 0.9 prediction for a disadvantageous action.

When the advantage is positive, $-log (p) * advantage$ will be smaller if $p$ is growing, and when the advantage is negative, $-log (p) * advantage$ will be smaller if $p$ is decreasing.


How to create neural network that predicates result of exam? Actually, I am "fresh-water", and I've never known what is neural network. Now I am trying understand how to design simple neuronetwork for this problem:

I'd like to make up such neural network that after learning it could predicate a mark of passed exam (for example, math). There is such factors that influence on a mark:

Chosen topic (integral, derivative, series)
Perfomance (low, medium, high)
Does a student work? (Yes, No, flexible schedule)
Have a student ever gotten through a add course? (Yes, No)

The output is a mark (A,B,C,D,E,F)
I don't know should I add few layers between inputs and output

Moreover, I have few results from past years:

(integral, low, Yes, No, E)
(integral, medium, Yes, Yes, B)
(series, high, No, Yes, A)
and so on. What do I need to know else for designing this NN?

You can add as many layers (with any arbitrary number of nodes) as you want.
Please note that as you add more learning parameters (layers and nodes), your model complexity increases. This means the model can potentially learn a more complex input-output relationship. However, it also increases the risk of overfitting. Overfitting generally happens when the model you build is more complex than the data you have. In such a scenario, the model memorizes the data instead of learning from it. In other words, it can produce a very good result for the same data as it was trained on but cannot generalize well. So, it performs poorly when the inputs are slightly different from what is used to be fed at the training stage.
In practice, you may try different architectures and parameter configurations, and measure the generalization capacity of the models (via cross-validation for example) to choose the best model. In English, a generalizable model is the one that performs (almost) equally good on both training/validation and testing sets.


What are the common pitfalls that we could face when training neural networks? Apart from the vanishing or exploding gradient problems, what are other problems or pitfalls that we could face when training neural networks?

I can't say that it is the biggest problem of deep neural network but it is one of the big problem with deep neural network.

Other issue which happens a lot is overfitting on the training data hence network behaves badly on test set which can be solved using regression. So you have to make sure that network is generalised enough.

If the network is trying to have a classification then if the outputs are mutually inclusive or exclusive, depending upon that you define your loss function which will be categorical cross entropy for mutually inclusive output and binary cross entropy for mutually exclusive.

Other thing which I can think is about all zero initialization. Initializing the network weights as zero can lead to same gradient computation and due to which network does not learn most of the times.


Were AI strategies identified at go or starcraft games and how? When an AI is trained to play an opposing game, such as chess or go, it can become very strong.
I have read in an article (non-scientific) the claim that AI strategies were identified by scientists while an AI was bound to play go games, as well as starcraft games. However it did not tell what these strategies actually were, how they were identified, nor did it explain the configuration in which AI played (AI vs AI? AI vs human?)
Can someone explain it to me? I am familiar with go, not with starcraft, so an explanation about go is appreciated.
I also note that the chess game is not mentioned. Is there any specific feature for chess that makes them inappropriate for strategies? Or is it the behavior of an AI in the chess game that does not allow to identify strategy?
I understand there are plenty of definitions for strategy, and the article did not give one. So let's focus on following significance: Strategy is a group of principles that tell which fields are important to fight for and which are not. A strategy gives long term rewards, which is opposite to tactics with short term rewards obtained thanks to calculation on a specific issue. With this definition, go game stand for a strategic game with a few, well known tactical situations such as line versus line.

Truth be told, I have no idea how to play Go, but luckily this is a AI forum and not a Go forum. Addressing your questions about the specific strategies that AI discovered, there's a paper released by OpenAI titled "Mastering the game of Go without human knowledge" (https://deepmind.com/research/publications/mastering-game-go-without-human-knowledge). Here, there's a section titled "Knowledge learned by AlphaGo Zero." In it, they mention that AlphaGo Zero discovered some variants of common corner sequences. I think it may be worth checking out. The section also lists some other common Go strategies that AlphaGo Zero learned. I believe during the training process, AlphaGo Zero was trained using self-play, where the AI played itself over and over again to get better.
To address your questions about chess. I don't think there's anything limiting AI from learning strategy in chess or anything about chess that limits AI from learning an effective strategy. Check out this paper for more: https://deepmind.com/research/publications/Mastering-Atari-Go-Chess-and-Shogi-by-Planning-with-a-Learned-Model
Here the authors present a general reinforcement learning model capable of mastering multiple common board games. I'm not too familiar with strategies in Star Craft, however, here's a link containing some other projects where strategies that emerge by the AI are shown: https://openai.com/blog/competitive-self-play/


Categorizing text into dynamic amount of categories I'm looking for a supervized system/approach, that could learn how to categorize incoming texts/documents, where new categories can be added over time and the training set will be small. The trained model should not be static and should be able to evolve with adding new categories or evaluating new documents.

For each document it should first give it's suggestion that can be then corrected.

Transfer Learning allows you to add new categories to be predicted to the output layer without needing to re-train the entire model every time a new category needs to be classified. Rather, the weights of all initial layers up to the last few layers of the network can be frozen and only the last one or two layers can be made trainable for fine-tuning the classification rather than training from scratch. This approach would be relatively fast than other methods since the dataset in your case is small.

In order for model to flag the need of a new output class by inferring on a test document, you could include a class "Unknown" in addition to existing N classes (hence, the output layer now contains N+1 classes). If the model predicts "Unknown" with highest probability, you could add a new class to the output layer after examining the test data.


WEKA - SimpleKMeans - Manually choose intitial centres Is there a way in the WEKA explorer to manually select the initial centres when using SimpleKMeans clustering?

there is an -init option for that

Initialization method to use.
  0 = random, 1 = k-means++, 2 = canopy, 3 = farthest first.
  (default = 0)
kmeans++ will give you an option for you to initialize centres.


Which neural network should I use to approximate a specific function? We have convolutional neural networks and recurrent neural networks for analysing  respectively images and sequential data.

How do I determine which neural network architecture is more appropriate to approximate a certain function? For example, suppose I want to approximate the function $f(x,y) = \sin(2\pi x)\sin(2\pi y)$ with domain $\Omega = [0,1]\times [0,1]$, that is, $x$ and $y$ can be between $0$ and $1$ (inclusive). For example, which kind of activation functions would be better suited for approximating this specific function?

If the concept class specified is

$$f(x, y) = k \, \sin(2 \pi f_x x) \, sin(2 \pi f_y y) \\ \land 0 < x < 1 \\ \land 0 < y < 1 \; \text{,}$$

and the optimum fit to example data is expected occur when $k \approx 1 \land f_x \approx 1 \land f_y \approx 1$, then it is not an AI problem. It is a problem that can be solved with a least squares convergence, probably in conjunction with a Fourier transform.

If nothing is known about $f(x, y)$ except continuity and that it is single valued with respect to $(x, y)$, then few conclusions can be drawn about best approach. In such a case, the domain of $x$ and $y$ are irrelevant because they can be normalized. Furthermore, the tree of operations, such as $\sin()$ and multiplication, are irrelevant too, because the function could just as easily be

$$f(x, y) = \ln(x) + \Gamma(y) - k \, \text{.}$$

The question indicates the design involves CNN and RNN components for analyzing images and sequential data. It is not clear whether the CNN is for the discovery of objects or waves (given the $\sin()$ in the function mentioned) and whether those objects move between frames so that the RNN must detect motion.

Nothing is given about the pool of example data available or planned to be available or the expected outputs of the system. If data is sequential, where is $t$ in the function? What is the objective of image analysis?

Although a deep MLP (multilayer perceptron) with SGD can learn an arbitrary function, it is hardly an architecture, the mention of images, CNN, RNN, and sequential data, MLP with SGD does not seem to match.

Regarding activation functions, the inner layer functions would depend on the higher level design requirements. The activation functions of the last layer of a single artificial network is usually chosen to match the data type and range of desired output for each output channel (dimension).

If the objective of this question is to take images and sequential data and produce something useful without a priori defining what useful means, then it is an unsolved AI problem thus far and no known topology comprised of artificial networks and other AI building blocks provide a solution. The autonomous development of internal concepts of usefulness would need to be developed mathematically and algorithmically and become practically speed optimized in hardware and software first.


Why does Monte Carlo work when a real opponent's behavior may not be random I am learning about Monte Carlo algorithms and struggling to understand the following:

If simulations are based on random moves, how can the modeling of the opponent's behavior work well?

For example, if I have a node with 100 children, 99 of which lead to an instant WIN, whereas the last one leads to an instant LOSS.

In reality, the opponent would never play any of the 99 losing moves for him (assuming they are obvious as they are the last moves), and would always play the winning one. But the Monte Carlo algorithm would still see this node as extremely favorable (99/100 wins for me), because it sees each of the 100 moves as equally probable.

Is my understanding wrong, or does it mean that in most games such situations do not occur and randomness is a good approximation of opponent behavior?

First, we need to distinguish plain Monte-Carlo from Monte-Carlo Tree Search. They're different things.

Monte-Carlo search, in the context of game AI search algorithms, is typically understood to mean that we search randomly many times, and average the results, and nothing else. If this is all we're doing, then yes, your understanding is correct. This is also sometimes referred to as "plain Monte-Carlo (search)" or "pure Monte-Carlo (search)", to make it explicitly clear that we're not doing any tree search as in Monte-Carlo Tree Search (sometimes when we just say "Monte-Carlo" in the context of game AI, people will automatically assume Monte-Carlo Tree Search, due to how popular it is).

Monte-Carlo Tree Search does a lot more than just that though. It gradually builds up a search tree (through the Expansion step), and within that search tree (which is growing over time), it uses a much more sophsticated strategy for traversal than pure random (the Selection step).

  For example suppose i have a node with 100 children, 99 of which leading to an instant WIN, and the last one leading to an instant LOSS.

Suppose that this node you're talking about, the one with those 100 children, is relatively close to the root node. Then, it is likely that it and all 100 of its children will end up "growing into the search tree" that is slowly built up by the Expansion step. Once they have been added to the search tree, the Selection step will make sure that the vast majority of further iterations visiting this part of the tree will select the instant loss (assuming the opponent is to move in this node). In the limit (after an infinite amount of search time), this bias towards selecting the loss node will be so large that the average evaluation tends to the (correct) minimax evaluation.

Another way to view the idea of evaluation through many random sequences of play is the following; the idea is that if we're in a very strong position, a good game state, then we're more likely to win than to lose if both players start playing at random. Consider, for example, a game of chess where Player 1 has many pieces left, and Player 2 only has a few pieces left. Imagine both players were to play completely at random from that game state onwards. On average, which player would you expect to win more often? Probably Player 1.

When we're considering game states that are still far away from terminal states, this basic idea tends to work relatively well. Obviously not always correct, it's still a heuristic, but it can kind of work. When we're already very close to a terminal state that can be reached through one highly specific sequence of play, yeah, we might miss that through random actions; this is where we need the more informed policy of the Selection step.


What is the difference between feature extraction with or without data augmentation? Here's an extract from Chollet's book "Deep Learning with Python" about using pre-trained CNN to predict class from a photo set (p. 146):

At this point, there are two ways you could proceed:

Running the convolutional base over your dataset, recording its output to a Numpy array on disk, and then using this data as input to
a standalone, densely connected classifier similar to those you saw in
part 1 of this book. This solution is fast and cheap to run, because
it only requires running the convolutional base once for every input
image, and the convolutional base is by far the most expensive part of
the pipeline. But for the same reason, this technique won’t allow you
to use data augmentation.

Extending the model you have (conv_base) by adding Dense layers on top, and running the whole thing end to end on the input data. This
will allow you to use data augmentation, because every input image
goes through the convolutional base every time it’s seen by the model.
But for the same reason, this technique is far more expensive than the
first.

The first method is called (1) and the second is (2).
If I use data augmentation to expand my data set, then could (1) be as good ad (2)? If no, why?

There are two ways that you could perform data augmentation:

Up front, by expanding the input dataset into a larger one, performing a range of changes to each input then storing the result. This appears to be what you are suggesting.

Just in time, by sampling from possible augmentations on each epoch, or even per sample when building a mini-batch. This appears to be what Chollet is suggesting.

Chollet's approach allows for augmentation to include finer degrees of augmentation that are different each time an input is considered, e.g. rotations of any angle, selecting a slightly different area from a larger image each time. For your approach you could consider the same set of augmentations but they would have to be "frozen in" at the time of building a dataset, and you would not be able to consider all possible variations for each image because it would make the dataset too large.
Both approaches are valid, and both would be called data augmentation. Chollet's approach obtains better re-use of image samples in the long term, and may result in better generalisation in the final trained network. Your approach may allow for more efficient use of CPU time to obtain a result that passes a threshold in accuracy. In some cases the difference between approaches may be minor compared to that caused by other changes in hyperparameters.


How does Pinterest decipher what's on unmarked pictures and categorize them? According to this article, Pinterest acquired VisualGraph, an image recognition and visual search technology startup.

How does Pinterest apply VisualGraph technology for machine vision, image recognition and visual search in order to classify the images?

In short, how do they predict the image categories? Based on what features?

One of the Pinterest's white paper about Human Curation and Convnets powering item-to-item recommendationsarxiv describes implementation of convolutional neural network (CNN) based visual features (VGG2014, Faster R-CNN). This demonstrates the effectiveness of it (such image or object representations) which can improve user engagement. The visual features are computed using the process described in the previous study about visual search at Pinterest and can be used for more targeted features to be computed for related pins.

Here are the examples of detected visual objects from Pinterest's object detection pipeline:

Image source: Human Curation and Convnets: Powering
Item-to-Item Recommendations on Pinterest, Page 4, Fig. 6

The images are categorized by using dominant visual objects (individual objects seen in the image which passes a confidence threshold in Faster R-CNN) using fine-tuned VGG reranking variant. This allows Pinterest to introduce features such real-time recommendations for the users.

Check also this blog entry: Building a scalable machine vision pipeline.


Correcting 'bad' translations in a sequence-to-sequence neural machine translation model In working with basic sequence-to-sequence models for machine translation I have been able to achieve decent results. But inevitably some translations are not optimal or just flat-out incorrect. I am wondering if there is some way of "correcting" the model when it makes mistakes while not compromising the desirable behavior on translations where it previously performed well.

As an experiment, I took a model that I had previously trained and gathered several examples of translations where it performed poorly. I then took those examples and put them into their own small training set where I provided more desirable translations than what the model was outputting. I then trained the old model on this new small training set very briefly (3-6 training steps was all it took to "learn" the new material). When I tested the new model it translated those several examples in the exact way I had specified. But as I should have anticipated the model overcompensated to "memorize" those handful of new examples  and thus I noticed it started to perform poorly on translations that it had previously been excellent.

Is there some way to avoid this behavior short of simply retraining the model from scratch on an updated data set? I think I understand intuitively that the nature of neural networks would not lend itself to small precise corrections (i.e. when the weighting of just a few neurons change the performance of the entire model will change) but maybe there is a way around it, perhaps with some type of hybrid reinforcement learning approach.

Update:

This paper speaks of approaches to incrementally improving neural machine translation models

The basic process you're describing sounds a lot like Boosting, which is also well covered in Chapter 18 of Russell & Norvig, or like active learning.

You're correct that training the model with emphasis on a specific subset of the data is likely to lead to mistakes somewhere else. Boosting gets around this in a clever way.

It begins by training a model as normal.
The new model's performance on the training set is measured.
A new dataset is generated, where examples that were misclassified by the previous model are more highly weighted (this is intuitively similar to adding an extra copy of them to the dataset).
A new model is trained on the weighted dataset. The model suffers a greater penalty for making the same kinds of errors as the previous model(s), and so tends to make different errors.
Steps 2-4 are repeated for some time.
An ensemble of models is output, rather than a single model.

To classify new points, the most common score from all ensemble members is used.

I'm less familiar with NLP work using this approach, but it seems like the same basic idea could be used fruitfully here: the notion is to train many models that make independent errors, and then trust their collective decisions.


How will morality questions be settled in the domain of self-driving cars? How will morality questions be settled in the domain of self-driving cars?
For example

If a dog is crossing the road, I'd expect the car to try to avoid it. But what if this leads to .00001% more risk for the driver? What is the 'risk cut-off'?

What if a cockroach is crossing the road? Will the car have a list of animals okay to run over?

What if a kid is crossing the street and avoiding it would kill the driver?

These questions seem to not really have an answer, yet self-driving cars are almost ready. What are they doing about all of this?

I don't think these questions will need to be answered.

A self driving car will almost certainly avoid a situation like the ones described well before a human would have and hence would not have to choose.

For example it would slow down as soon as it sees a child close to the road. It will identify and react to the fact that the child starts moving towards the road and act before the situation requires the "drastic" scenarios that we invent.

If for some reason the car has to choose, it can also make the impact/avoidance have the highest chance of not killing the occupant of the vehicle considering how well designed cars are these days and that the speeds involved shouldn't be anything wild like 200mph.

Not to mention a network of cars along with street cameras and sensors would act/work together to resolve as a swarm intelligence so the car ahead or the traffic camera can warn/tell other cars that something they cannot see is a potential hazard. I can go on and on...

In my opinion, the bottom line is a self driving car will not road rage, drive at dangerous speeds in a residential area, get tired and fall asleep, text and drive or drink and drive, etc... I cannot wait.


PPO agent for vehicle control does not learn to stop at traffic lights I have built a custom RL environment with gym, which simulates the RL vehicle and potential vehicles in front of the RL vehicle as well as traffic lights (including their state; red, yellow, green). I trained a PPO agent from stable_baselines3 without considering setting of hyperparameters and the agent learned to follow the vehicle in front of it without crashing. However it does not learn to stop at red lights after extensive training.
I tried training it without surrounding vehicles to get more interactions of the RL vehicle with traffic lights and this helped the agent to learn stopping a red light. However when I then continue training of the agent in a new environment with surrounding traffic, the agent again un-learns stopping at red lights.
I am still a novice with RL and do not understand as to why this happens and what I can do here. Should I set hyperparameters? Or try a different model? Or should I exchange the default policy of the PPO model?

The way I dealt with it was by giving a (very) strong negative reward when committing the mistake (here going under a red light) and the agent should learn to do not do this mistake anymore.
It is often better to change actions, rewards or environment rather than acting on hyperparameters in my opinion, but I may be mistaken.


Feasibility of sorting a basket of clothes in the real world I'm wondering how feasible it is to create a machine that can separate clothing from a basket.

At the most basic level it would distinguish between tops, pants, button downs and socks

Programmatically, I'd image this would require training a neural network to recognize these items, but in real time it becomes exponentially difficult to do this in a small space at a fast rate:

pick up an item
lay it in such a way that is recognizable
deduce whether it is a top, button down, etc.
sort it accordingly

If this sounds ridiculous please let me know...

If it is possible :

would this be based on some sort of computer vision?
or only a well trained neural network?

Any insight is much appreciated!

Peter Abbeel does work in deep learning for robotics, and one of the projects they've tackled is manipulating clothes. Here's a video from 2011 of their robot folding laundry (one piece at a time).

There are also companies attempting to market this; seven dreamers makes the Laundroid and Foldimate claims that it will start taking pre-orders in 2017.


Why is tf.abs non-differentiable in Tensorflow? I understand why tf.abs is non-differentiable in principle (discontinuity at 0) but the same applies to tf.nn.relu yet, in case of this function gradient is simply set to 0 at 0. Why the same logic is not applied to tf.abs? Whenever I tried to use it in my custom loss implementation TF was throwing errors about missing gradients.

By convention, the $\mathrm{ReLU}$ activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for tf.nn.relu. As you've found, of course, it's not true in general that we treat the gradient of the absolute value function as zero in the same situation; it makes sense for it to be an explicit choice to use this trick, because it might not be what the code author intends in general.
In a way this is compatible with the Python philosophy that explicit is better than implicit. If you mean to use $\mathrm{ReLU}$, it's probably best to use tf.nn.relu if it is suitable for your use case.
[1] Vinod Nair and Geoffrey Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. ICML'10 (2010). URL.


What are the fundamental differences between VAE and GAN for image generation? Starting from my own understanding, and scoped to the purpose of image generation, I'm well aware of the major architectural differences:

A GAN's generator samples from a relatively low dimensional random variable and produces an image. Then the discriminator takes that image and predicts whether the image belongs to a target distribution or not. Once trained, I can generate a variety of images just by sampling the initial random variable and forwarding through the generator.

A VAE's encoder takes an image from a target distribution and compresses it into a low dimensional latent space. Then the decoder's job is to take that latent space representation and reproduce the original image. Once the network is trained, I can generate latent space representations of various images, and interpolate between these before forwarding through the decoder which produces new images.

What I'm more interested is the consequences of said architectural differences. Why would I choose one approach over the other? And why? (for example, if GANs typically produce better quality images, any ideas why that is so? is it true in all cases or just some?)

GANs generally produce better photo-realistic images but can be difficult to work with.  Conversely, VAEs are easier to train but don’t usually give the best results.
I recommend picking VAEs if you don’t have a lot of time to experiment with GANs and photorealism isn’t paramount.
There are exceptions such as Google’s VQ-VAE 2 which can compete with GANs for image quality and realism.  There is also VAE-GAN and VQ-VAE-GAN.
As a note, GANs and VAEs are not specifically for images and can be used for other data types/structures.


Sutton and Barto 2nd Edition Exercise 13.1 I'm attempting exercise 13.1 in the Sutton and Barto textbook. It asks for an optimal probability for selecting action right in the short corridor scenario (see first 6 lines of the image below for the scenario).
Exercise 13.1: Use your knowledge of the gridworld and its dynamics to determine an
exact symbolic expression for the optimal probability of selecting the right action in
Example 13.1.
My attempt:
Letting $p$ denote the probability of choosing right, I understand that using the Bellman equations, we can solve for the value of $s_1, s_2, s_3$ where the states are numbered from left to right in terms of $p$. We have $v(s_1) = \frac{2-p}{p-1}$, $v(s_2) = \frac{1}{(p-1)p}$, $v(s_3) = -\frac{p+1}{p}$.
I can see how we can find the max of each of these functions to get the best optimal policy, given the state we're currently in.
However, how do you find the optimal policy generally (irrespective of starting state)? I found solutions here, which magically arrives at
$\frac{p^2-2p+2}{p(1-p)}$. Can someone explain this part?
https://github.com/brynhayder/reinforcement_learning_an_introduction/blob/master/exercises/exercises.pdf

There are problems with both the approach and the expressions that you have. I don't want to just give the correct solution, though, that's an exercise for you to go through and learn from your own experience trying to accomplish it. Instead, let me illustrate that your expressions for $v(s_i)$ are wrong. To do that we'll just do a Monte-Carlo estimate for a range of values of $p$ and compare to your expression.
Here's the Python code that runs a single payout starting from state state and following the policy with probability p of choosing "right". It returns the collected reward:
import numpy as np

def playout(state, p):
   reward = 0
   while state != 0:
       action_right = np.random.rand() < p
       move_right = action_right if state != 2 else (not action_right)
       state = state - 1 if move_right else state + 1
       state = 3 if state > 3 else state
       reward -= 1
   return reward

Then we make a simulation code that runs the playout multiple times, collects the reward counts, and returns the average reward (so, essentially estimates $v^\pi(s_i)$):
from collections import defaultdict

def simulate(state, p, n):
    rewards = defaultdict(lambda : 0, {})
    for _ in range(n):
        rewards[playout(state,p)] += 1
    results = np.array([[a,b] for a,b in rewards.items()]).T
    reward , nplayouts = results[0] , results[1]
    value = (reward * nplayouts).sum() / nplayouts.sum()
    return value

Finally, I make a grid in p and run each playout 10000 times:
p = np.linspace(0,1,51)[1:-1]
v3 = [simulate(3,p,10000) for p in p]
v2 = [simulate(2,p,10000) for p in p]
v1 = [simulate(1,p,10000) for p in p]

I've plotted the resulting value estimates for each state. Together with your expression for them (blue curves). And the correct expression (red curve) that I've obtained by actually writing down and solving the equations:

As you can see, the expressions you've presented are all too far off from the results that the simulation returns. More than that - the asymptotic behavior of your solutions at $p\to0$ and $p\to1$ doesn't make much sense.
I've obtained the expressions for the red curves above by solving the system for $v(s_i)$ on my own account - without relying on weird and wrong solutions that I googled on the internet. Which I'd recommend you do as well.
Finally, the question of the exercise is to find an optimal $p$ for a policy that starts at $s_3$ - not "irrespective of starting state" as you've thought is should be. Unlike your expression the correct expression for  $v(s_3)$ has a maximum, which can be found analytically and it is
$$\max_p v(s_3) = -6-4\sqrt2 \simeq -11.6$$
$$ \text{at}\quad p = ??? \simeq 0.59$$


Applications of AI for creatives and artists I have just watched a few videos on Tedtalks talking about how AI benefits creatives and artists but none of the videos I watched provided further resources for reference.

So far, I have only came across mentioning of the creative usage of AI but would like to know how creatives and artists can apply AI in their work process. Like at least a tutorial guide on how it works.

Are there any recommendations on communities, tutorials, guides, platforms and real world AI applications that are meant for creatives and artists?

The machine learning (~AI) is all about code and data. If you planning to create something really unique, you'll probably need to, well, work with code and data.

You'll probably should look at github and try to get inspiration from opensourced projects. If you found some AI-related artwork, try to google it's name with 'github'

Another approach would be to look at medium, it's usually have some post/tutorial.

The art is very broad topic and it's hard to cover it all. I try to provide examples of different resources that could help you to start.

Google Research made some examples, where you run code in colab without installing anything locally. It helps you to try basic stuff.
There is tensorflow.js, that allows to launch code at browser. You can look at demos with code
There is intro course for visual artist on kadenze.
Here it is some more links on different courses.
DCGAN - very popular image generation library.
TouchDesigner, very popular tool among visual artists, provides python interface. For example, here is usage of aforementioned DCGAN with it

Also, there is creative section on NIPS(one of the biggest ML conferences) http://nips4creativity.com/. Some of the work provides detailed explanation, like this one

I know, starting to code could be intimidating, but many github repos provides some description about how to launch them and you don't even have to know how they work exactly to use them. Also, it would help to have a person with programming experience, who could help you to deal with code.


Are there any meaningful books entirely written by an artificial intelligence? Are there any meaningful books entirely written by an artificial intelligence? I mean something with meaning, unlike random words or empty books.
Something that can be charactersed as fiction literature.
If yes, then I think it is also interesting to know if any of those books is available for sale. Is there a specific name for such books? Like "robot books"?

James Ryan has done a lot of 'archaeological' work on this; you can find references to his work on his website.
Story generation has been a dream for a long time (in computing terms), and various genres have been explored, with not that much success. There have been episodes of a Western written by a computer (and actually filmed and acted out by human actors, see summary here), and various books, but the technology is nowhere near good enough to produce something worthwhile reading or watching without heavy editing.
So far it's only good for curiosity value.
There is NaNoGenMo, where since 2013 people work on programs generating novels. But most of them — again — are more interesting for curiosity. They either take an existing work and modify it procedurally, or generate random templated text (eg travel reports through a ficticious, auto-generated landscape). I don't think anyone has met the target of creating a 50k novel by computer yet.
Modern approaches with deep learning generators can produce reams of more or less well-formed text, but that doesn't make a compelling story. It's the meaning that is currently the problem, and as AI researchers keep finding out, it's hard.


How to understand mapping function of kernel? For a kernel function, we have two conditions one is that it should be symmetric which is easy to understand intuitively because dot products are symmetric as well and our kernel should also follow this. The other condition is given below
There exists a map $φ:R^d→H$ called kernel feature map into some high dimensional feature space H such that $∀x,x'$ in $R^d:k(x,x') = <φ(x),φ(x')>$
I understand that this means that there should exist a feature map that will project the data from low dimension to any high dimension $D$ and kernel function will take the dot product in that space.
For example, the Euclidean distance is given as
$d(x,y)=∑_i(x_i−y_i)^2=<x,x>+<y,y>−2<x,y>$
If I look this in terms of second condition how do we know that doesn't exist any feature map for euclidean distance? What exactly are we looking in feature maps mathematically?

A kernel function $f : \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ is a valid support vector kernel if it is a Mercer kernel. Mercer's condition essentially ensures that the Gram matrix of the kernel is positive semi-definite. Interestingly, this ensures that the SVM objective is convex.
The Euclidean distance function does not satisfy Mercer's condition since it's Gram matrix is not necessary positive semi-definite. Thus, it is not a valid kernel.


Confusion between function learned and the underlying distribution Let us assume that I am working on a dataset of black and white dog images.
Each image is of size $28 \times 28$.
Now, I can say that I have a sample space $S$ of all possible images. And $p_{data}$ is the probability distribution for dog images. It is easy to understand that all other images get a probability value of zero. And it is obvious that $n(S)= 2^{28 \times 28}$.
Now, I am going to design a generative model that sample from $S$ using $p_{data}$ rather than random sampling.
My generative model is a neural network that takes random noise (say, of length 100) and generates an image of the size $28 \times 28$. My function is learning a function $f$, which is totally different from the function $p_{data}$. It is because of the reason that  $f$ is from $R^{100}$ to $S$ and $p_{data}$ is from $S$ to $[0,1]$.
In the literature, I often read the phrases that our generative model learned $p_{data}$ or our goal is to get $p_{data}$, etc.,  but in fact, they are trying to learn $f$, which just obeys $p_{data}$ while giving its output.
Am I going wrong anywhere or the usage in literature is somewhat random?

You're right! The generative model $f$ is not the same as the probability density  (p.d.f.) function $p_{data}$. The kind of phrases you've referred to are to be interpreted informally. You learn $f$ with the hope that sampling a latent vector $z$
from some known distribution (from which it is easy to sample), results in $f(z)$ that has the probability density function $p_{data}$. However, merely learning $f$ does not give you the power to estimate what $p_{data}(x)$ is for some image $x$. Learning $f$ only gives you the power to sample according to $p_{data}(\cdot)$ (if you've learned an accurate such $f$).


Is tabular Q-learning considered interpretable? I am working on a research project in a domain where other related works have always resorted to deep Q-learning. The motivation of my research stems from the fact that the domain has an inherent structure to it, and should not require resorting to deep Q-learning. Based on my hypothesis, I managed to create a tabular Q-learning based algorithm which uses limited domain knowledge to perform on-par/outperform the deep Q-learning based approaches.

Given that model interpretability is a subjective and sometimes vague topic, I was wondering if my algorithm should be considered interpretable. The way I understand it, the lack of interpretability in deep-learning-based models stems from the stochastic gradient descent step. However, in case of tabular Q-learning, every chosen action can always be traced back to a finite set of action-value pairs, which in turn are a deterministic function of inputs of the algorithm, although over multiple training episodes.

I believe in using deep-learning-based approaches conservatively only when absolutely required. However, I am not sure how to justify this in my paper without wading into the debated topic of model interpretability. I would greatly appreciate any suggestions/opinions regarding this.

There is not a widely accepted definition of explainable AI (XAI). However, as a rule of thumb (my rule of thumb), if you can't explain it easily to a layperson (or even an expert), then the model or algorithm is not (very) interpretable. There are other concepts related to XAI, such as accountability (who is responsible for what?), transparency and fairness.

For example, the final decision of (trained) decision tree can easily be explained to (almost) any person, so a (trained) decision tree is a relatively interpretable model. See the chapter 4.4. Decision Tree of the book Interpretable Machine Learning: A Guide for Making Black Box Models Explainable.

An artificial neural network (ANN) is usually considered not very interpretable because, unless you attempt to understand which parts of the network contribute to the output of the ANN (for example, with the technique layer-wise relevance propagation), then you cannot immediately or easily understand the output or decision of the ANN, given that an ANN involves many non-linear functions, which produce unintuitive outcomes. In other words, it is more difficult to attribute the contributions of each unit of an ANN to the output of the same ANN than to explain e.g. the decision of a decision tree.

In the context of deep reinforcement learning (DRL), the ANN is used to approximate the value or policy functions. This approximation is, in the first place, the main reason behind the low interpretability of deep RL models.

Q-learning is an algorithm, so it is not a model, like an ANN. Q-learning is used to learn a state-action value function, denoted with $Q: S \times A \rightarrow \mathbb{R}$, which can then be used to derive another function, the policy, which can then be used to take actions. In a way, Q-learning is similar to gradient descent, because both are machine learning (or optimization) algorithms. The $Q$ function is a model of the environment, given that, for each state, it represents the expected amount of reward that can be obtained, so, in a certain way, the learned $Q$ function represents a prediction of reward.

Is the learned tabular $Q$ function interpretable? Yes, it is relatively interpretable, but how much? What kind of interpretation do you really need? It depends on the context and people that need the interpretation or explanation. A reinforcement learning researcher will usually be satisfied with the usual explanation of the inner workings of $Q$-learning, Markov decision processes, etc., because the usual RL researcher is not concerned with the really important problems that involve the life of people and other beings. However, for example, in the context of healthcare, doctors might not just be interested in the explanation "expected maximum future reward", but they might also be interested in the environment, the credit assignment problem, the meaning and effectiveness of the reward function with respect to the actual problem that needs to be solved, in a probabilistic interpretation of the results (rather than just a mere action that needs to be taken), possible alternative good actions, etc.

Recently, there have been some attempts to make RL and, in particular, deep RL more interpretable and explainable. In the paper Programmatically Interpretable Reinforcement Learning (2019), Verma et al. propose a more interpretable (than deep RL) RL framework that is based on the idea of learning policies that are represented in a human-readable language. In the paper InfoRL: Interpretable Reinforcement Learning using Information Maximization (2019), the authors focus on learning multiple ways of solving the same task and they claim that their approach provides more interpretability. In the paper Toward Interpretable Deep Reinforcement Learning with Linear Model U-Trees (2018), the authors also claim that their approach facilitates understanding the network's learned knowledge by analyzing feature influence, extracting rules, and highlighting the super-pixels in image inputs.

To conclude, deep RL should not necessarily be avoided: it depends on the context (e.g., it is usually perfectly fine to use deep RL to solve video games). However, in cases where liability is an issue, then deep RL should also be explainable or more explainable alternatives should also be taken into account.


Executing trained image classification model for video I've been working with vanilla feed forward neural networks and have been researching the convolutional neural network literature. Thus far I've have not encountered how often the model is executed in order to classify objects. For example if a camera is capturing video at a rate of 15 frames per second is the classification model being trained / executed iteratively in order to maintain non time delayed classifications ?

It is possible to deploy models that process video ranging from frame-by-frame analysis up to live streaming, depending on the use case. The reason for deployment will influence video quality and pre-processing as well as the complexity of model that can be used.

As video can result in a lot of data piling up very rapidly, compromises might be made between quality of image resolution and colour, frames might be skipped, processing could be done offline, or it might just cost a ton of money to pay for the compute power! Without going into much detail, different algorithms can be used for the different tasks of detection, recognition/classification, and tracking, and the input to each can be processed in different ways.

This tutorial does a great job of showing how OpenCV can be used for object detection from a laptop webcam stream and a pre-trained CNN, it explicitly mentions dropping frames to improve performance.

There's also this post with a good overview of object detection, recognition and tracking. Depending on the use case in the context of each algorithm described, you can determine how important it is to retain and process all frames or not - for instance, see the section on background subtraction; you might not want to risk something flying through the frame without detecting it, let alone classifying it.


Is it expected that adding an additional hidden layer to my 3-layer ANN reduces accuracy significantly? I've been using several resources to implement my own artificial neural network package in C++.

Among some of the resources I've been using are

https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/

https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/

https://cs.stanford.edu/people/karpathy/convnetjs/intro.html,

as well as several others.

My code manages to replicate the results in the first two resources exactly. However, these are fairly simple networks in terms of depth. Hence the following (detailed) question:

For my implementation, I've been working with the MNIST Database of handwritten digits (http://yann.lecun.com/exdb/mnist/).

Using the ANN package I wrote, I have created a simple ANN with 784 input neurons, one hidden layer with 16 neurons, as well as an output layer with ten neurons. I have implemented ReLU on the hidden layer and the ouput layer, as well as a softmax on the output layer to get probabilities.The weights and biases are each individiually initialized to random values in the range [-1,1]

So the network is 784x16x10.

My backpropagation incorporates weight gradient and bias gradient logic.

With this configuration, I repeatedly get about a 90% hit rate with a total average cost of ~0.07 on the MNIST training set comprising 60,000 digits, and a slightly higher hit rate of ~92.5% on the test set comprising 10,000 digits.

For my first implementation of an ANN, I am pretty happy with that. However, my next thought was:

"If I add another hidden layer, I should get even better results...?".

So I created another artificial network with the same configuration, except for the addition of another hidden layer of 16 neurons, which I also run through a reLU. So this network is 784x16x16x10.

On this ANN, I get significantly worse results. The hit rate on the training set repeatedly comes out at ~45% with a total average error of ~0.35, and on the test set I also only get about 45%.

This leads me to either one or both of the following conclusions:

A) My implementation of the ANN in C++ is somehow faulty. If so, my bet would be it is somewhere in the backpropagation, as I am not 100% certain my weight gradient and bias gradient calculation is correct for any layers before the last hidden layer.

B) This is an expected effect. Something about adding another layer makes the ANN not suitable for this (digit classification) kind of problem.

Of course, A, B, or A and B could be true.

Could someone with more experience than me give me some input, especially on whether B) is true or not?

If B) is not true, then I know I have to look at my code again.

You probably got the back propagation wrong. I have done a test on the accuracy on adding an extra layer and the accuracy went up from 94% to 96% for me. See this for details:

https://colab.research.google.com/drive/17kAJ2KJ36grG9sz-KW10fZCQW9i2Tf2c

To run the notebook click Open in playground and run the code. There is a commented line which add 1 extra layer. The syntax should be easy to understand even though it is in python.

For back propagation, you can try to see this python implementation of multi layer perceptron backpropagation.

https://github.com/enggen/Deep-Learning-Coursera/blob/master/Neural%20Networks%20and%20Deep%20Learning/Building%20your%20Deep%20Neural%20Network%20-%20Step%20by%20Step.ipynb

A network will not usually decrease it's accuracy by almost a half in normal scenario when you add an extra layer, though it is possible to have the network decrease accuracy when you add an extra layer due to overfitting. Though if this happen the performance drop won't be that dramatic.

Hope I can help you.


What would happen if we sampled only one tuple from the experience replay? The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right?
What would happen if we calculate our loss using just one experience instead of a mini-batch of experiences?

The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right?

Yes that is a major benefit of using a replay buffer.
A secondary benefit is the ability to use the same sample more than once. This can lead to beter sample efficiency, although that is not guaranteed.

What would happen if we calculate our loss using just one experience instead of a mini-batch of experiences?

The algorithm is still valid, but the gradient estimate for the update step would be based on a single record of [state, action, reward, next state]. This would be a high variance update process, with many steps in wrong directions, but in expectation over many steps you should still see a correct gradient. You would probably need to compensate for the high variance per sample by reducing the learning rate.
In addition, assuming the standard approach of collecting one time step then making one update to DQN neural network, each piece of experience would only be used once on average before being discarded.
These two effects will likely combine such that the learning process would not be very sample efficient.
The size of the minibatch is one of many hyperparameters you can change in DQN. It might be the case for some problems that choosing a low minibatch size is helpful, provided other adjustments (such as a lower learning rate) are made along with it. If you are not sure, you mostly have to try and see.
In my experience on a small range of problems, a moderate size of minibatch - ranging from 10 to 100 - has worked the best in terms of end results of high scoring agents. However, I have not spent a long time trying to make low batch sizes work.


How do you implement NEAT by taking into account the loops? I'm working on my own implementation of NEAT algorithm based on the original 2002 paper called "Efficient Reinforcement Learning through Evolving Neural Network Topologies" (by Kenneth O. Stanley and Risto Miikkulainen). The way the algorithm is designed it may generate loops in connection of hidden layer. Which obviously will cause difficulties in calculating the output.

I have searched and came across two types of approaches. One set like this example claim that the value should be calculated like a time series usually seen in RNNs and the circular nodes should use "old" values as their "current" output. But, this seems wrong since the training data is not always ordered and the previous value has nothing to do with current one.

A second group like this example claim that the structure should be pruned with some method to avoid loops and cycles. This approach apart from being really expensive to do, is also against the core idea of the algorithm. Deleting connections like this may cause later structural changes.

I my self have so far tried setting the unknown forward values as 0 and this hides the connection (as whatever weight it has will have no effect on the result) but have failed also for two reasons. One is my networks get big quickly destroying the "smallest network required" idea and also not good results.

What is the correct approach?

NEAT does not employ a feed forward concept and also it does not take any special action to avoid loops.
Network is evaluated in non-recursive model. The only non-deterministic loop, the evaluation has is the loop for activating all the outputs. Pseudo code is something like this,
Until all the outputs are active
    for all non-sensor nodes
        activate node
        sum the input
    for all non-sensor and active nodes
        calculate the output

NOTE1: You can use a defensive mechanism (like a counter) to avoid a infinite loop
NOTE2: When summing the input, outputs of nodes which were at least evaluated/calculated once are considered, otherwise their outputs are assumed to be zero.
This is the note from the author of NEAT about identifying loops,

Note that checking networks for loops in general in not necessary and therefore I stopped writing this function
bool Network::integrity()


Huge variations in epoch count for highest generalized accuracy in CNN I have written my own basic convolutional neural network in Java as a learning exercise. I am using it to analyze the MIT CBCL face database image set. They are a set of 19x19 pixel greyscale images.

Network specifications are:

Single Convolution Layer with 1 filter:
Filter Size: 4x4.
Stride Size: 1

Single Pooling Layer
2x2 Max Pooling

3 layer MLP(input, 1 hidden and output)
input = 64 neurons
hidden = 15 neurons
output = 2 neurons
learning rate = 0.1

Now I am getting reasonable accuracy(92.85%), but my issue is that it is being achieved at very different points in the epoch count across network runs:

Epochs  Training Accuracy   Test Accuracy   Validation Accuracy

Run 1   415 93.13   92.44   93.35
Run 2   515 92.44   93.18   92.84
Run 3   327 93.83   92.05   92.38

I am using the Java random class with the same seed for every run to initialize the kernel, the MLP weights and break the input data into 3 sets.(training is being done using the 33-33-33 method)

I am a loss as to what is causing this variation in epoch count to achieve the highest point in validation accuracy. Can anybody explain this?

Fixed. Was an issue with the random generator. In my class for the Neuron layer where I initialize the weights I get new doubles from the generator for each of the initial weight values, but I found a bug where I was re-initializing the random generator, which was of course causing different values.


How can certain information about the goal be given to the RL learning algorithm? Policy learning refers to mapping an agent state onto an action to maximize reward. A linear policy, such as the one used in the Augmented Random Search paper, refers to learning a linear mapping between state and reward.
When the entire state changes at each time-step, for example in the Continuous Mountain Car OpenAI Gym, the position and speed of the car changes at each time-step.
However, assume we also wanted to communicate the constant position of one or more goals. By "constant", I mean does not change within a training episode, but may change between episodes. For example, if there was a goal on the left and right of the Mountain Car.
Are there examples of how this constant/static information be communicated from the environment other than appending the location of the two goals to the state vector? Can static/constant state be differentiated from state which changes with each action?

I see three main ways to do this, which one makes more sense will depend on your application.

One is to append that information to the state/observations like you mentioned. While this information is static for a particular episode, it will be different across episodes and the policy should learn to condition the actions it chooses based on this information.

Another would be to leave goal information out entirely and force the agent to learn a policy that works when the goal is unknown. This will likely be more difficult to learn and you may end up with a policy that moves to the average of all goals or explores and tries them all.

A third option and probably the most natural is to have some context cue that the agent can observe (e.g. often in lab experiments with rats, a cue is placed on a wall that lets the rat know which way to go to get a reward). This is similar to the first method, except that the cue has to be observed rather than given directly. For the Mountain Car example, this could be an extra signal that the agent only sees in a particular location, such as the bottom of the valley or when it moves close to a particular side.


If neurons performed the operation of an entire layer, would that make the neural network more effective? (I have a very primitive understanding of neural networks, so please forgive the lack of technicality here.)
I am used to seeing a neuron in a neural network as something that-

Takes the inputs and multiplies them by their weights,
then sums them up,
and after that it applies the activation function to the sum.

Now, what if it was "smarter"? Say, a single neuron could do the function of an entire layer in a network, could that make the network more effective? This comes from an article I was reading at Quanta, where the author says:

Later, Mel and several colleagues looked more closely at how the cell might be managing multiple inputs within its individual dendrites. What they found surprised them: The dendrites generated local spikes, had their own nonlinear input-output curves and had their own activation thresholds, distinct from those of the neuron as a whole. The dendrites themselves could act as AND gates, or as a host of other computing devices.

...realised that this meant that they could conceive of a single neuron as a two-layer network. The dendrites would serve as nonlinear computing subunits, collecting inputs and spitting out intermediate outputs. Those signals would then get combined in the cell body, which would determine how the neuron as a whole would respond.

My thoughts: I know that Backpropagation is used to "teach" the network in the normal case, and the fact that neurons are simply activation buttons is somehow related to that. So, if neurons were to be more complicated, it would reduce efficiency. However, I am not sure of this: why would complex individual components make the network less effective?

Say, a single neuron could do the function of an entire layer in a network, could that make the network more effective?

That depends what you mean by "more effective". In terms of number of neurons to achieve the same result, then you should need fewer units. In terms of being able to calculate an end result for any specific problem, then no, because you can generally solve a problem using simpler units by adding more of them.
If this could somehow be done using less resources, it might reduce overall costs. In a biological system, there are possibly overheads per cell in order to maintain it on top of the costs for calculation, so it may be better to do more than the simplest calculation in each cell. Further to that, there may be an optimal amount of processing that each cell could do (this is all conjecture on my part).
In an artifical neural network, the calculations are the only thing being considered, there is no separate overhead per neuron.
There are neural network architectures with complex "sub-units". Probably the most well known are the recurrent neural network designs for LSTM and gated recurrent units (GRU), plus "skip connections" in residual neural networks. For efficiency these are normally processed in groups per layer with matrix processing functions, but you can also view them as per-neuron complexities.

I am not sure of this- why would complex individual components make the network less effective?

If the complexity was not used, or not really needed, in some of the units, then it would be wasted capacity. In a biological system, this might correspond to maintaining cells larger than they needed to be. In an artificial system, it would mean using memory and CPU to calculate interim values that were not needed for the task at hand.


How can we prove that an autoassociator network will continue to perform if we zero the diagonal elements of a weight matrix? How can we prove that an auto-associator network will continue to perform if we zero the diagonal elements of a weight matrix that has been determined by the Hebb rule? In other words, suppose that the weight matrix is determined from $W = PP^T- QI$, where $Q$ is the number of prototype vectors.

I have been given a hint: show that the prototype vectors continue to be eigenvectors of the new weight matrix.

This is a question from Neural Network Design (2nd Edition) book by
Martin T. Hagan, Howard B. Demuth, Mark H. Beale, Orlando De Jesus .

Resource : E7.5 p 224-225

As we have an autoassociative network prototype vectors are both input and output vectors. So , we have that :
\begin{equation}
    T = P
\end{equation}

\begin{equation}
    W = PP^T - QI = TP^T - QI = \sum_{q=1}^Q p_q p_q^T - QI
\end{equation}

Applying a prototype vector as input :

\begin{equation}
    \alpha = W \cdot p_k = \sum_{q=1}^Q p_k p_q p_q^T - QIp_k
\end{equation}

Because they are orthogonal we have :
\begin{equation}
    \alpha = p_k(p_k^T\cdot p_k) - Q \cdot I \cdot p_k = p_k (p_k^T\cdot p_k - Q \cdot I ) = p_k (\textbf{R - Q $\cdot$ I})
\end{equation}

where $R-Q\cdot I$ is the length of vectors

So since ,

\begin{equation}
    W \cdot p_k= (R - Q \cdot I) \cdot p _k
\end{equation}
prototype vectors
continue to be eigenvectors of the new weight matrix .

It is often the case that for auto-associative nets, the diagonal weights (those which connect an input component to the corresponding output component) are set to 0. There are papers that say this helps learning. Setting these weights to zero may improve the net's
ability to generalize or may increase the biological
plausibility of the net. In addition, it is necessary if we use iterations (iterative nets) or the delta rule is used


Is feature engineer an important step for a deep learning approach? I'd like to ask you if feature engineering is an important step for a deep learning approach.
By feature engineering I mean some advanced preprocessing steps, such as looking at histogram distributions and try to make it look like a normal distribution or, in the case of time series, make it stationary first (not filling missing values or normalizing the data).
I feel like with enough regularization, the deep learning models don't need feature engineering compared to some machine learning models (SVMs, random forests, etc.), but I'm not sure.

From what I believe, feature engineering is important, it's a part of the job of ML network designer.
Network designing involves

Feature engineering: What should be in the input to the network, as processed from similar or totally different data
Deciding network shape, layer shapes, types of neurons in layers, etc.
Feature engineering again (but labels), in the output, what should the output be, either regression values or classes

And possibly also tasks rather simple as mentioned in the question: filling missing values, normalising data, create pre-feeding normalisation steps in code, etc.


Is the Markov property assumed in the forward algorithm? I'm majoring in pure linguistics (not computational), and I don't have any basic knowledge regarding computational science or mathematics. But I happen to take "Automatic Speech Recognition" course in my graduate school and struggling with it.

I have a question regarding getting the formula for a component of the forward algorithm.

$$
\alpha_t(j) = \sum_{i=1}^{N} P(q_{t-1} = i, q_t=j, o_1^{t-1}, o^t|\lambda)
$$

When $q$ is a hidden state, $o$ is a given observation, and $\lambda$ contains transition probability, emission probability and the start/end state.

Is the Markov assumption (the current state is only dependent upon the one right before it) assumed here? I thought so, because it contains $q_{t-1}=i$ and not $q_{t-2}=k$ or $q_{t-3}=l$.

In general, to formally state that the Markov property holds, you need to have $P(
x_t \mid x_{t-1:1}) = P(x_t \mid x_{t-1})$.

So, you cannot conclude only from $P(q_{t-1} = i, q_t=j, o_1^{t-1}, o^t|\lambda)$ that the Markov property holds, because  $P(q_{t-1} = i, q_t=j, o_1^{t-1}, o^t|\lambda)$ is the joint probability of $q_{t-1} = i, q_t=j, o_1^{t-1}$ and $o^t$ given $\lambda$.

Nonetheless, the theory of hidden Markov model often assumes that a few properties (including the Markov property) hold

the Markov property: $$P(q_{t+1} \mid q_{t}) = P(q_{t+1} \mid q_{t:1}),$$ where $q_t$ is the hidden state at time step $t$ and $q_{t:1} = q_t, q_{t-1}, \dots, q_1$.
the stationarity property: $$P(q_{t_1 + 1} \mid q_{t_1}) = P(q_{t_2 + 1} \mid q_{t_2}),$$ for any time step $t_1$ and $t_2$. In other words, the state transition probabilities are independent of the actual time at which the transitions takes place.
the output independence property: $$P(o^{T:1} \mid q_{T:1}) = \prod_{t=1}^T P(o^t \mid q_t , \lambda)$$
where $o^{T:1} = o^T, o^{T-1}, \dots, o^1$. In other words, this is the assumption that the output at time step $t$, $o^t$, is independent of the outputs at previous time steps.


If the goal of training of a GAN is to have $P_g=P_{data}$, shouldn't this produce the exact same images? Referring to the blog, Image Completion with Deep Learning in TensorFlow, it clearly says that we would want a generator $g$ whose modeled distribution fits our dataset $data$, in other words, $P_{data}=P_g$.

But, as described earlier in the blog, the space $P_{data}$ is in is a higher-dimensional space, where a dimension represents a particular pixel in an image, making it a $64*64*3$ dimensional space (in this case). I have a few questions regarding this

Since each pixel here will have an intensity value, will the pdf try to encapsulate a unique pdf for each pixel?
If we sample the most likely pixel value for each pixel, considering the distributions need not be the same for each pixel, is it not quite likely that the most probabilistic generated image is just noise apart from things like a common background or so?
If $P_g$ is trying to replicate $P_{data}$ only, does that mean a GAN only tries to learn lower level features that are common in the training set? Are GANs clueless about what its doing?

Answers:

No, generally GANs model the joint distribution of pixel space. The space is so entangled that it would be very difficult to model a singular pixels PDF.
This question doesn't make sense given the first question's answer. You do NOT sample singular pixels, you sample the joint.
$P_g$ is trying to model $P_{data}$, so that's your goal, so yes it will find features in the training set. Ideally, though it can extrapolate usage of the features to generate new images. But, yes depending on your setup you can end up with overfitting or mode-collapse where you really only generate a small subset of images that trick the discriminator. So I would say, no, the GAN doesn't know what its doing (since its just a minimax optimization of a loss you setup), so your goal is to tell it what to do in a way to achieve your best result.


How to fight with unstability in self play? I'm working on a neural network that plays some board games like reversi or tic-tac-toe (zero-sum games, two players). I'm trying to have one network topology for all the games - I specifically don't want to set any limit for the number of available actions, thus I'm using only a state value network.
I use a convolutional network - some residual blocks inspired by the Alpha Zero, then global pooling and a linear layer. The network outputs one value between 0 and 1 for a given game state - it's value.
The agent, for each possible action, chooses the one that results in a state with the highest value, it uses the epsilon greedy policy.
After each game I record the states and the results and create a replay memory. Then, in order to train the network, I sample from the replay memory and update the network (if the player that made a move that resulted in the current state won the game, the state's target value is 1, otherwise it's 0).
The problem is that after some training, the model plays quite well as one of the players, but loses as the other one (it plays worse than the random agent). At first, I thought it was a bug in the training code, but after further investigation it seems very unlikely. It successfully trains to play vs a random agent as both players, the problem arises when I'm using only self play.
I think I've found some solution to that - initially I train the model against a random player (half of the games as the first player, half as the second one), then when the model has some idea what moves are better or worse, it starts training against itself. I achieved pretty good results with that approach - in tic-tac-toe, after 10k games, I have 98.5% win rate against the random player as the starting player (around 1% draws), 95% as the second one (again around 3% draws) - it finds a nearly optimal strategy. It seems to work also in reversi and breakthrough (80%+ wins against random player after the 10k games as both players). It's not perfect, but it's also not that bad, especially with only 10k games played.
I believe that, when training with self play from the beginning, one of the players gains a significant advantage and repeats the strategy in every game, while the other one struggles with finding a counter. In the end, the states corresponding to the losing player are usually set to 0, thus the model learns that whenever there is the losing player's turn it should return a 0. I'm not sure how to deal with that issue, are there any specific approaches? I also tried to set the epsilon (in eps-greedy) initially to some large value like 0.5 (50% chance for a random move) and gradually decrease it during the training, but it doesn't really help.

The AlphaZero paper mentions an "evaluation" step that seems to deal with the  the problem similar to yours:

... we evaluate each new neural network checkpoint against the current best network $f_{\theta_*}$ before using it for data generation ... Each evaluation consists of 400 games ... If the new player wins by a margin of > 55% (to avoid selecting on noise alone) then it becomes the best player $\alpha_{\theta_*}$ , and is subsequently used for self-play generation, and also becomes the baseline for subsequent comparisons

In the AlphaStar they've use a whole league of agents that was constantly played against each other.


How can supervised learning be viewed as a conditional probability of the labels given the inputs? In the literature and textbooks, one often sees supervised learning expressed as a conditional probability, e.g.,
$$\rho(\vec{y}|\vec{x},\vec{\theta})$$
where $\vec{\theta}$ denotes a learned set of network parameters, $\vec{x}$ is an arbitrary input, and $\vec{y}$ is an arbitrary output. If we assume we have already learned $\vec{\theta}$, then, in words, $\rho(\vec{y}|\vec{x},\vec{\theta})$ is the probability that the network will output an arbitrary $\vec{y}$ given an arbitrary input $\vec{x}$.
I am having a hard time reconciling how, after learning $\vec{\theta}$, there is still a probabilistic aspect to it. Post training, a network is, in general, a deterministic function, not a probability. For any specific input $\vec{x}$, a trained network will always produce the same output.
Any insight would be appreciated.

This formulation/interpretation can indeed be confusing (or even misleading), as the output of a neural network is usually deterministic (i.e. given the same input $x$, the output is always the same, so there is no sampling), and there isn't really a probability distribution that models any uncertainty associated with the parameters of the network or the input.
People often use this notation to indicate that, in the case of classification, there is a categorical distribution over the labels given the inputs, but this can be misleading, as the softmax (the function often used to model this categorical distribution) only squashes its inputs and doesn't really model any uncertainty associated with the input or the parameter of the neural network, although the elements of the resulting vector add up to 1. In other words, in traditional deep learning, only a point estimate for each parameter of the network is learned and no uncertainty is properly modeled.
Nevertheless, certain supervised learning problems have a formal probabilistic interpretation. For example, the minimization of the mean squared error function is equivalent to the maximization of a log probability, assuming your probability distribution is a Gaussian with a mean equal to the output of your model. In this probabilistic interpretation, you typically attempt to learn a probability (e.g. of the labels in the training dataset) and not a probability distribution. Watch Lecture 9.5 — The Bayesian interpretation of weight decay (Neural Networks for Machine Learning) by G. Hinton or read the paper Bayesian Learning via Stochastic Dynamics or Bayesian Training of Backpropagation Networks by the Hybrid Monte Carlo Method by R. Neal for more details.
Moreover, there are Bayesian neural networks (BNNs), which actually maintain a probability distribution over each parameter of the neural network that models the uncertainty associated with the value of this parameter. During the forward pass of this BNN, the specific parameters are actually sampled from the corresponding probability distributions. The actual learnable parameters of a BNN are the parameters of these distributions. For example, if you decide to have a Gaussian distribution over each parameter of the neural network, then you will learn the mean and variance of these Gaussians.


Why can't I reproduce the experiments in the original paper that introduced the Firefly Algorithm? I have been trying to reproduce the experiments done in the original: Firefly Algorithm for multimodal optimization (2010) by Xin-She Yang, but so far unsuccessfully. For the moment being, I'm okay if anyone points me in the right direction.
I wrote the algorithm as specified in the paper in C++ programming language (I also downloaded several other implementations from the internet for comparison purposes) and used the very same parameters as specified in the paper (a random steep of 0.2, an initial light intensity of 1.0 and a light decay coefficient of 1.0, a population size of 40). I used the two bright update equations given and for De Jung test function (as for example) a number of dimensions of 256 in a search domain in [-5.12, 5.12] as referred to in common optimization literature and in the paper.
In the paper, the algorithm converges very quickly, as can be expected, since this is a very simple test function. However, neither my implementation nor any code I have downloaded converges with that parameters.
My final questions are:

Am I doing something wrong with the experimental methodology or am I using the wrong parameter settings (may be something different than the original paper)?

Do anyone knows where can I find a code sample of the Firefly Algorithm that I can use to reproduce the experiments of the mentioned paper?

Please, notice that there may be a lot of variations of this algorithm that can produce better results, but right now I'm only interested in reproducing the experiments of the so-called paper.

I wrote some python code to reproduce this paper's purported results. My code very efficiently optimizes simple smooth functions like bowls, but does not come close to reproducing the paper's claimed results on more complex functions, including with the parameters the authors report. I think that, since both @Jairo and I were unable to reproduce the results from the information in the paper, independently, it is likely that something is wrong with the paper.

It may be possible to reproduce the paper's claimed behaviors using a library like py_swarm, however the paper is using a Firefly Algorithm (already a rarer form of a PSO), and is using a home-brew variant of that algorithm, so my guess is that rolling your own is required, as we both did.

Additionally, I find the paper's claims fairly unlikely. For example, in section 4.1, they claim to converge on the global minimum of a function with many local minima in just 10 iterations. It seems to me like most of the time the fireflies will quickly converge on the bottom of one of the gullies in this function, and get stuck there. This is also what I observe in my replications. I suspect the authors may have cherry-picked their results from the best runs without reporting this, or have omitted some key detail from the paper.

Here is my replication code, in case someone else wants to try to reproduce this:

from math import sin, pi, exp, sqrt
from random import random
from copy import deepcopy

def michalewiz_objective(x):
    result = 0
    for i, x_i in enumerate(x):
        result -= sin(x_i)*(sin(i*(x_i**2)/pi))**20
    return -result

def bowl_objective(x):
    return -sum([x_i**2 for x_i in x])

pop_size = 40
max_generations = 10
alpha = 0.2
gamma = 1
beta_0 = 1
d = 2
I = michalewiz_objective
#I = bowl_objective

def move(firefly, other_firefly):
    radius = sqrt(sum([(firefly[i] - other_firefly[i])**2 for i in range(0, len(firefly))]))
    for i, value in enumerate(firefly):
        firefly[i] += beta_0*exp(-gamma*radius**2)*(other_firefly[i] - firefly[i])
        firefly[i] += alpha * (random() - 0.5)

# Using 4*random() to match Figure 3's apparent spread.
fireflies = [[4*random() for i in range(0, d)] for j in range(0, pop_size)]

for generation in range(0, max_generations):
    new_fireflies = [deepcopy(firefly) for firefly in fireflies]
    for index, firefly in enumerate(fireflies):
        for other_firefly in fireflies:
            if I(other_firefly) > I(firefly):
                move(new_fireflies[index], other_firefly)
    fireflies = new_fireflies
    best = max([I(f) for f in fireflies])
    mean = [sum([f[0] for f in fireflies]), sum([f[1] for f in fireflies])]
    print(best)
    print(mean)


How to use pre-trained BERT to extract the vectors from sentences? I'm trying to extract the vectors from the sentences. Spent soo much time searching for the pre-trained BERT models but found nothing.

Is it possible to get the vectors using pre-trained BERT from the data?

Did you mean:

  How do you use a pre-trained BERT model in a feature-based setting
  to get pre-trained word contextual embeddings?

Here is the BERT paper. I highly recommend you read it.

Firstly, by sentences, we mean a sequence of word embedding representations of the words (or tokens) in the sentence. Word embeddings are the vectors that you mentioned, and so a (usually fixed) sequence of such vectors represent the sentence input. (We don't need the input to always be divided to individual sentences)

There are mainly two ways to train the modern neural network models for sequence modeling tasks such as language modelling, machine translation and question answering:

1. The pretrain - finetune based approach (also called Transfer Learning)

In this approach, a model is first pre-trained on some auxiliary dataset whose domain usually overlaps with the domain of the dataset corresponding to the downstream task, that is, the actual task for which we need to build the model.

In the case of Natural Language Processing (NLP) tasks, such as machine translation and question answering, the pre-training that is done is usually unsupervised (no labels in the dataset) where our training objective is to maximize the log likelihood of the sentences. You can read more about Maximum Likelihood estimation here.

Its almost always a good idea to pre-train models (whether it is for NLP tasks or for computer vision tasks etc). The more we pre-train a model, the closer its parameters are to the optimal ones, and the work needed in the fine-tuning phase is minimized. This is why we use large scale datasets for pre-training.

Later, during the fine-tuning phase, we fine tune the parameters of the model to suit a downstream task, such as a task of translation between two specific languages of text. Since the parameters of the model are already at a position in the error/loss space that is already sort of good enough for the model (because we already pre-trained it with text in, say, 10 languages), the fine-tuning can, in a sense, tailor the model to suit the task a hand.

2. The feature based approach

In this approach, we take an already pre-trained model (any model, e.g. a transformer based neural net such as BERT, which has been pre-trained as described in the previous approach) and then we extract the activations from one or more layers of this pre-trained model.

This is done by simply inputting the word embedding sequence corresponding to a sentence to the pre-trained model and then extracting the activations from one or more layers (one or more of the last layers, since the features associated to the activations in these layers are far more complex and include more context) of this pre-trained model.

These activations (also called contextual embeddings) are used as input (similar to the word embeddings described earlier) to another model, such as an LSTM or even another BERT. That is, fixed features are extracted from the pre-trained model.

Usually the size of these layers are the same as the size of the input layer, i.e. they are also a sequence, but a sequence of far more complex features instead of word embeddings. If the word embedding vector of the input to this model id $D$ and the length of the sentence which is input to the model is $L$, then each of the hidden activation layers have the same length $L$, but the hidden size of these vectors is $H >> D$, i.e. far higher to incorporate more complex features.

Even though the dimensionality of the hidden layers is far higher, these layers are equal in length to the input layer, and so we have a one to one correspondence between the input tokens in the sentence and each of these hidden layer activations.

The bonus in using these hidden representations is that we get to use representations of the input sentence that also incorporate CONTEXT, dependency between the words of the sentence, an important feature that is to be modeled by all models used for NLP tasks. Models such as BERT, are based on Transformer models that use an attention mechanism in order to model these contextual features.

This is a superb article describing to the last detail the attention mechanism and the Transformer model based on which models like BERT operate. For more about contextual embeddings I recommend taking a look at this article. The figure used here is from this article.

Models such as BERT make use of one half of these Transformer models, called the encoder, since the Transformer is a seq2seq or encoder-decoder model. Here is the link to the Transformer paper.

Here is a great blog on extracting contextual word embeddings from BERT using Tensorflow and Keras.

I'll also provide a link to a Kaggle Python Notebook on using Pipelines functionality from the HuggingFace community repo on github that also is used for feature extraction (contextual embeddings).

Please let me know if you have any questions, happy to help!


How does high entropy targets relate to less variance of the gradient between training cases? I've been trying to understand the Distilling the Knowledge in a Neural Network paper by Hinton et al. But I cannot fully understand this:

When the soft targets have high entropy, they provide much more information per training case than hard targets and much less variance in the gradient between training cases [...]

The information part is very clear, but how does high entropy correlate to less variance between training cases?

Since it is a trained network already, when you run an example through it, the gradient will not have a very high variance.
The gradient varies a lot when you are training a network from the scratch but then it stops varying much since it understands the pattern.


What is the difference between machine learning and deep learning? Can someone explain to me the difference between machine learning and deep learning? Is it possible to learn deep learning without knowing machine learning?

Deep learning is a specific variety of a specific type of machine learning. So it's possible to learn about deep learning without learning all of machine learning, but it requires learning some machine learning (because it is some machine learning).

Machine learning refers to any technique that focuses on teaching the machine how it can learn statistical parameters from a large amount of training data. One particular type of machine learning is artificial neural networks, which learn a network of nonlinear transformations that can approximate very complicated functions of wide arrays of input variables. Recent advances in artificial neural networks have to do with how to train deep neural networks, which have more layers than normal and also special structure to deal with the challenges of learning more layers.


In NEAT, is it a good idea to give the same ID to node genes created from the same connection gene? Do I have to prevent nodes created from the same connection gene to have different IDs/innovation number? In this example, the node 6 is created from the connection going from node 3 to node 4:

In the case where that specific node was already globally created, is it useful to give it the same ID for crossover? Because the goal of NEAT is to do meaningful crossover by doing historical marking. The paper from Kenneth O. Stanley says at page 108:

  [...] by keeping a list of the innovations that occurred in the current generation, it
  is possible to ensure that when the same structure arises more than once through independent mutations in the same generation, each identical mutation is assigned the
  same innovation number.

Why don't we do that for node genes too?

In NEAT, the innovation of a node does not affect the evolution directly. Only the connection genes and their innovation will matter. So you can simply have whole numbers as IDs under each Genome / Network.
--EDIT-- (Complete reasoning)
In the original paper, it is clearly stated that the nodes from the better genome is taken during crossover and then only the connections are cross-over'ed (in some method) and hence the innovation numbers for the connections. NEAT is connection-centric and does not care much about evolving nodes.
Adding to that from basic neural networks theory, the nodes will never matter in a neural network because all the calculation and the learning happens in the connections. Think about a regular feed-forward network. You only care about the weight matrix which is a property of the connections instead of the nodes though both are present. Similarly in a NEAT generated network, the nodes will not matter, as all the learning takes place in the way these nodes are connected and the weights of the network.
Further, the nodes list can easily be derived from the connection list and hence marking the connections, are enough.


What evolutionary algorithms are there that model epigenetics? What evolutionary algorithms are there that model or incorporate some notion of epigenetics? What are the pros/cons of those approaches? Are there vast insufficiencies or wide-open questions about their usefulness?

Over the last few years, evolutionary computation research has shown increasing interest in including some aspect of epigenetics. For example:

A 2008 paper by Tanev and Yuta
Work from Lee Spector's genetic programming group
A recent paper by Ricalde and Banzhaf


Is it possible for visual systems in AI to have Inattentional Blindness? Inattentional Blindness is common in humans (see: https://en.wikipedia.org/wiki/Inattentional_blindness ). Could this also be common with machines built with artificial vision?

Presumably what happens to people in the famous Invisible Gorilla experiment, is that an incongruous object is simply filtered out of human perception.

If we wish to interpret this mechanistically, we could hypothesize that a 'gorilla object' is simply not presented by low levels of perception to our higher level pattern recognizers because the lower levels are not biased towards the construction of 'gorilla-like' features in such a context.

The recent Tesla fatality (arising from a failure to distinguish between the sky and a high-sided white truck) could conceivably be considered to be an example of this.

See this AI SE question.


Could error surface shape be useful to detect which local minima is better for generalization? The following plot shows error function output based on system weights.
Two equal local minima are shown in green pointers. Note that the red dots are not related to the question.

Does the right one generalize better compared to the left one?

My assumption is that for the right minimum if the weights change, the overall error increases less compared to the left minimum point. Would this somehow mean the system does better generalization if the right one is chosen as the optimum minimum?

In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations:

what the author of the question provides us with is the Loss Function Shape so I'll try to use the full information here to compare the 2 minima
looking at the LF steepness we observe the Left LM is in a steeper region than the Right LM: how can this be interpreted?
I'd interpret the LF Steepness as a Measure of Parametrization Stability: in fact perturbating the Parametrization slightly has a bigger effect on Left LM observed performance rather than the Right LM one
when the NN is running "in production" typically the parametrization is fixed (in most typical application, weights are not changed out of the training phase) so you should not be concerned about parametrization stability,
however I'm one of those believing the Flat Minima - Schmidhuber 1997 idea that Local Minima Flatness is connected to generalization property

However it is important to observe this is still a very open and interesting question as in Sharp Minima Can Generalize For Deep Nets Dinh 2017 et al. demonstrated
it is not just about flatness, because reparametrization of the NN model, though preserving minima locations, change its shape so basically sharp minima might be transformed into flat ones without affecting network performance