Use Machine Learning/Artificial Intelligence to predict next number (n+1) in a given sequence of random increasing integers The AI must predict the next number in a given sequence of incremental integers (with no obvious pattern) using Python but so far I don't get the intended result!
I tried changing the learning rate and iterations but so far no luck!

Example sequence: [1, 3, 7, 8, 21, 49, 76, 224]

Expected result: 467

Result found : 2,795.5

Cost: 504579.43

This is what I've done so far:

import numpy as np

# Init sequence
data =\
    [
        [0, 1.0], [1, 3.0], [2, 7.0], [3, 8.0],
        [4, 21.0], [5, 49.0], [6, 76.0], [7, 224.0]
    ]

X = np.matrix(data)[:, 0]
y = np.matrix(data)[:, 1]

def J(X, y, theta):
    theta = np.matrix(theta).T
    m = len(y)
    predictions = X * theta
    sqError = np.power((predictions-y), [2])
    return 1/(2*m) * sum(sqError)

dataX = np.matrix(data)[:, 0:1]
X = np.ones((len(dataX), 2))
X[:, 1:] = dataX

# gradient descent function
def gradient(X, y, alpha, theta, iters):
    J_history = np.zeros(iters)
    m = len(y)
    theta = np.matrix(theta).T
    for i in range(iters):
        h0 = X * theta
        delta = (1 / m) * (X.T * h0 - X.T * y)
        theta = theta - alpha * delta
        J_history[i] = J(X, y, theta.T)
     return J_history, theta
print('\n'+40*'=')

# Theta initialization
theta = np.matrix([np.random.random(), np.random.random()])

# Learning rate
alpha = 0.02

# Iterations
iters = 1000000

print('\n== Model summary ==\nLearning rate: {}\nIterations: {}\nInitial
theta: {}\nInitial J: {:.2f}\n'
  .format(alpha, iters, theta, J(X, y, theta).item()))
print('Training model... ')

# Train model and find optimal Theta value
J_history, theta_min = gradient(X, y, alpha, theta, iters)
print('Done, Model is trained')
print('\nModelled prediction function is:\ny = {:.2f} * x + {:.2f}'
  .format(theta_min[1].item(), theta_min[0].item()))
print('Cost is: {:.2f}'.format(J(X, y, theta_min.T).item()))

# Calculate the predicted profit
def predict(pop):
    return [1, pop] * theta_min

# Now
p = len(data)
print('\n'+40*'=')
print('Initial sequence was:\n', *np.array(data)[:, 1])
print('\nNext numbers should be: {:,.1f}'
  .format(predict(p).item()))

UPDATE Another method I tried but still giving wrong results

import numpy as np
from sklearn import datasets, linear_model

# Define the problem
problem = [1, 3, 7, 8, 21, 49, 76, 224]

# create x and y for the problem

x = []
y = []

for (xi, yi) in enumerate(problem):
    x.append([xi])
    y.append(yi)

x = np.array(x)
y = np.array(y)
# Create linear regression object
regr = linear_model.LinearRegression()
regr.fit(x, y)

# create the testing set
x_test = [[i] for i in range(len(x), 3 + len(x))]

# The coefficients
print('Coefficients: \n', regr.coef_)
# The mean squared error
print("Mean squared error: %.2f" % np.mean((regr.predict(x) - y) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % regr.score(x, y))

# Do predictions
y_predicted = regr.predict(x_test)

print("Next few numbers in the series are")
for pred in y_predicted:
    print(pred)

I think your code works fine for what is meant to be doing - fitting a linear regression model. The problem here is that you are using a linear model. Linear model does not have an adequate approximation capacity, it will only be able to fit data that is described by a linear function. Here, you gave a random sequence of numbers, that is very difficult for linear model to approximate. I would advise you to try 2 things:

1) Try something simpler first. Instead of doing a random sequence of numbers do a linear sequence of numbers for example a function like $y = 2x$ or maybe affine function like $y = 2x + 5$. So you would have a sequence like:

$2, 4, 6, 8 ...$ or $7, 9, 11, 13, ...$

If you manage to get that working try a nonlinear function like $x^2$ for example.

2) Instead of using a linear model try a nonlinear model. For example a polynomial regression model. Especially powerful function approximators are neural networks. In theory, a neural network with single hidden layer can approximate an arbitrary continuous function under some conditions
(Universal approximation theorem) , so you could try to see how would a neural network solve the problem, there are several open source neural network libraries that you could try.


In the case of invalid actions, which output probability matrix should we use in back-propagation? As discussed in this thread, you can handle invalid moves in reinforcement learning by re-setting the probabilities of all illegal moves to zero and renormalising the output vector.
In back-propagation, which probability matrix should we use? The raw output probabilities, or the post-processed vector?

I am also quite new to this field, but I think you should use the normalized outputs for the backpropagation. In general, you would want to backpropagate all the calculations you did in the forward pass, so why would you want to exclude the step of the normalization in your backward pass? This would essentially make the renormalization to have no effect (different loss values but no different model weight update).
For example, in policy gradients, you backpropagate through the log propability of the selected action. In the forward pass, the sampling of the probability (that determines which action is selected) is not affected by the renormalization (you might just get different loss values in the end in your loss function). But, compared to this, in the backward pass, you need the actual value of the log probability to calculate the gradient that updates the model weights.
So (I think) the normalization is mostly just done for the backpropagation to get "renormalized" gradients. And no unbalanced gradients between states with more/less allowed actions.


What are some academic AI podcast out there? I am looking for AI podcasts that are purely academic-oriented that I can use for learning purposes. Thanks for any resource pointers.

The AI podcasts I am aware of are (not sure how many of these can be considered academic):

The AI Podcast
Linear Digressions
O'Reilly Bots Podcast
A16z Podcast

Also a few others options (that I listen to) include:

Learning Machines 101
Machine Learning Guide
Machine Learning - Software Engineering Daily


Is there any Python library available for manifold learning using diffusion map? I would like to use unsupervised learning with a technique called diffusion map based manifold learning in Python. The original paper on the diffusion map is An Introduction to Diffusion Maps.

I have already checkout pydiffmap library, but it is super slow with my use case, and I am yet to see someone endorse the correctness of the implementation. I am looking for an alternate implementation in Python or R.

PS: It is also not available in sklearn, as far as I know. The list of other methods in sklearn is here.

The first place I would have directed you would be Sklearn and pydiffmap. I found this paper specifically about the problem you are doing using python the reference a package called  megaman  It seems like an active  Github . I suggest not just looking at manifold learning papers but leaning towards a search toward non linear embedding or non linear dimensionality reduction as well. The paper you give doesn't reference local linear embedding but seems to be similar conceptually.

I found this   kernel pca document  which is very straight forward and I think will help you going forward as well. I believe the RBF Kernel PCA part may be of use to you at least as a way of benchmarking how well your diffusion map is functioning.


One hot encoding vs dummy variables best practices for explainable AI (XAI) When creating artificial columns for your categorical variables there are two mainstream methods you could use:
Disclaimer: For this example, I use the following definitions of dummy variables and one-hot-encoding. I'm aware both methods can be used to either return n or n-1 columns.
Dummy variables: each category is converted to it's own column and the value 0 or 1 indicates if that category is present for each record
one-hot-encoding: similar to dummy variables, but one column is dropped, as its value can be derived from the other columns. This is to prevent multicollinearity and the dummy variable trap.
As an arbitrary example, let's take people's favorite color: pink, blue and green. For a person who's favorite color is pink, the dummy and one-hot-encoded data would look as follows:
dummy variables

person_id
favorite_color_pink
favorite_color_blue
favorite_color_green

xyz
1
0
0

one-hot-encoded variables

person_id
favorite_color_blue
favorite_color_green

xyz
0
0

From a statistics point of view, I would use the one-hot encoded columns to build my model. In addition, I can infer the favorite color is pink, because I encoded the variables.
However, when I'm applying XAI to explain the prediction to someone else and they see the favorite color wasn't blue or green. I'm not so sure they will infer the favorite color was pink unless it's explicitly stated. So using dummy variables might serve explainability better, but brings other risks..
Are there any best practices on this?

Personally I would chose one hot encoding as it is statistically correct/model friendly. Moreover, you can always provide additional help/tools to aid explainability. Lastly even if you add the nth column, you still need some idea about the working of model(and the boundaries it created while training) to interpret the result.


Neural network to control movement and "home in" on a target I'm a software developer who keeps trying (and failing) to get my head around AI and neural networks. There is one area that sparked my interest recently - simulating a mouse "homing in" on a piece of cheese by following the smell. Based on the rule that moving closer to the cheese = stronger smell = good, then it feels like it should be quite a simple problem to solve - in theory at least!

My thought process was to start by placing the mouse and cheese in random positions on the screen. I would then move the mouse one step in a random direction and measure its distance to the cheese, and if it's closer than before (stronger smell) then that's good. This is where I come unstuck on the theory - this "feedback" somehow needs to modify the mechanism used to move the mouse, gradually refining it until the mouse is able to head straight towards the cheese.
Once "trained", I should be able to reposition the cheese and expect the mouse to travel to it more quickly. Note I'm also keeping things simple by not having obstacles for the mouse to negotiate around.

How on earth would this be implemented with a NN? I understand the basic concepts, but I find that things unravel once I start looking at real code! The examples I've seen typically start by training the NN from a data set, but this doesn't seem to apply here as it feels like the only training available is "on the fly" as the mouse moves around (i.e. closer = good, further away = bad). I'm assuming the brain has some kind of "reward mechanism" triggered by a stronger smell of cheese.

Am I barking up the wrong tree - either with my thought process, or NN not being a good fit for this problem? This isn't homework btw, just something that I've been puzzling over in the back of my mind.

I would go with a NEAT algorithm for this one. Basically, you "breed" the right network that does the best every generation. Then Rinse and repeat. Doesn't require a training datbase. Bots compete and evolve as they go.

For example, this demonstration shows how Neataptic.js teaches a bunch of bots to follow a target:
https://wagenaartje.github.io/neataptic/articles/targetseeking/

Here is more complex problem - a game of Agar.io:
https://wagenaartje.github.io/neataptic/articles/agario/


What are the reasons behind slow YOLO training? I'm testing out YOLOv3 using the 'darknet' binary, and custom config. It trains rather slow.

My testing out is only with 1 image, 1 class, and using YOLOv3-tiny instead of YOLOv3 full, but the training of yolov3-tiny isn't fast as expected for 1 class/1 image.

The accuracy reached near 100% after like 3000 or 4000 batches, in similarly 3 to 4 hours.

Why is it slow with just 1 class/1 image?

I think you underestimate the size of YOLO. This is the size of one segment of yolo tiny according to the darknet .cfg file:

Convolutional Neural Network structure:
416x416x3                Input image
416x416x16               Convolutional layer: 3x3x16, stride = 1, padding = 1
208x208x16               Max pooling layer: 2x2, stride = 2
208x208x32               Convolutional layer: 3x3x32, stride = 1, padding = 1
104x104x32               Max pooling layer: 2x2, stride = 2
104x104x64               Convolutional layer: 3x3x64, stride = 1, padding = 1
52x52x64                 Max pooling layer: 2x2, stride = 2
52x52x128                Convolutional layer: 3x3x128, stride = 1, padding = 1
26x26x128                Max pooling layer: 2x2, stride = 2
26x26x256                Convolutional layer: 3x3x256, stride = 1, padding = 1
13x13x256                Max pooling layer: 2x2, stride = 2
13x13x512                Convolutional layer: 3x3x512, stride = 1, padding = 1
12x12x512                Max pooling layer: 2x2, stride = 1
12x12x1024               Convolutional layer: 3x3x1024, stride = 1, padding = 1

.cfg file found here: https://github.com/pjreddie/darknet/blob/master/cfg/yolov3-tiny.cfg

EDIT: These networks generally aren't specifically designed to train fast, they're designed to run fast at test time, where it matters


How to calculate the distance between the camera and an object using Computer Vision? I want to create a Deep Learning model that measures the distance between the camera and certain objects in an image. Is it possible? Please, let me know some resources related to this task.

In general, calculation of distance between camera and object is impossible if you don't have further scene dependent information.
To my knowledge you have 3 options:
Stereo Vision
If you have 2 cameras looking at the same scene from a different point of view you can calculate the distance with classical Computer Vision algorithms. This is called stereo vision, or also multiview geometry.
Stereo Vision is the reason why humans can infer the distance to objects around them (because we have 2 eyes).
Structure from Motion
You move your camera and therefore change your viewpoint and can essentially do stereo mapping over time. Structure from Motion
Scene Understanding
Why is it then still possible for a one-eyed person to infer depth to some extent? Because humans have lots of scene dependent understanding. If you see a rubber duck that takes half of your field of view, you know it's pretty close because you know a rubber duck is not big. If you don't know the size of rubber ducks it is impossible to know whether you see a big rubber duck that is far away or a small rubber duck that is really close.
This is where Deep Learning based models come into play. A recent overview over monocular
depth estimation can be found in Zhao2020


Why does the hill climbing algorithm only produce a local maximum? Apparently, the hill climbing algorithm just produces a local maximum, and not necessarily a global optimum. It's stuck on a local maximum. Why does hill climbing algorithm only produce a local maximum?

This part of your sentence is not always true "and not reached to final goal/solution". If you have just one maximum at all and it is finite, hill climbing (HL) can reach to it and it is a global maximum too (for example, if the function is a parabola).

To answer your question is back to the stop criteria of HL. It will be stopped when reaching to a maximum point. The maximum point means a point which the objective function has a greater value in that point than on its $\epsilon$-neighborhood of that point. Therefore, if objective function has some local maxima, depends on the initial point of HL, it might be trapped in one of the local maxima as their definition is satisfied in the explained definition.

In sum, depends on the initial point of HL, it might not reach to the global maxima and trapped in local maxima if there exists.


Which kind of prioritized experience replay should I use? The Prioritized Experience Replay paper gives two different ways of sampling from the replay buffer. One, called "proportional prioritization", assigns each transition a priority proportional to its TD-error.
$$p_i = |\delta_i|+\epsilon$$

The other, called "rank-based prioritization", assigns each transition a priority inversely proportional to its rank.
$$p_i = 1/\text{rank}(i)$$
where $\text{rank}(i)$ is the rank of transition $i$ when the replay buffer is sorted according to $|\delta_i|$.

The paper goes on to show that the two methods give similar performance for certain problems.
Are there times when I should choose one sampling method over the other?

The authors of that paper hypothesized that rank-based prioritization would be more robust to outliers. They suggested that rank-based sampling would be preferred for this reason. However, as they noted later, the fact that DQN clips rewards anyways weakens this argument.

If you're going to use someone else's ready-made code for your prioritized experience replay, then you'll probably end up using proportional sampling. Every implementation I've been able to find, including OpenAI's baselines implementation, uses proportional sampling. If you were going to write your own, proportional sampling might be preferred for being simpler to implement (so less error-prone).

Comparing these two sampling methods is complicated by the fact that rank-based sampling involves a hyperparameter that determines how often you sort your replay buffer. The authors of the original PER paper only sorted every $n$ timesteps, giving the nice amortized time of $O(\log n)$, where $n$ is the size of the replay buffer. Sampling takes constant time, so sampling a minibatch of size $k$ takes $O(k)$ time.

Proportional sampling doesn't involve sorting, but it does need to maintain a sum tree structure, which takes $O(\log n)$ time each time we add to the buffer. Sampling also takes $O(\log n)$ time, so sampling a minibatch of size $k$ takes $O(k\log n)$ time.

If we only sort our replay buffer every $n$ timesteps, then rank-based sampling is faster. However, because the buffer is almost always only approximately sorted, the distribution we sample from is only a rough estimate of the distribution we wanted. It's not clear that this estimation would be accurate enough to be performant when the replay buffer is scaled up in size past the $n=10^6$ transitions used in the paper. I haven't seen a study that compares performance for different frequencies of sorting and different buffer sizes.

So, rank-based sampling might be faster, but it also might not work as well. Adjustment of the sorting frequency hyperparameter might be necessary. The simpler and surer approach would be to use proportional sampling with clipped TD-errors.


DQN exploration strategy for large grid-world environment My task involves a large grid-world type of environment (grid size may be $30\times30$, $50\times50$, $100\times100$, at the largest $200\times200$). Each element in this grid either contains a 0 or a 1, which are randomly initialized in each episode.  My goal is to train an agent, which starts in a random position on the grid, and navigate to every cell with the value 1, and set it to 0. (Note that in general, the grid is mostly 0s, with sparse 1s).

I am trying to train a DQN model with 5 actions to accomplish this task:

Move up
Move right
Move down
Move left
Clear (sets current element to 0)

The "state" that I give the model is the current grid ($N\times N$ tensor). I provide the agent's current location through the concatenation of a flattened one-hot ($1\times(N^2)$) tensor to the output of my convolutional feature vector (before the FC layers).

However, I find that the epsilon-greedy exploration policy does not lead to sufficient exploration. Also, early in the training (when the model is essentially choosing random actions anyway), the pseudo-random action combinations end up "canceling out", and my agent does not move far enough away from the starting location to discover that there is a cell with value 1 in a different quadrant of the grid, for example. I am getting a converging policy on a $5\times5$ grid w/ a non-convolutional MLP model, so I think that my implementation is sound.

How I might encourage exploration that will not always "cancel out" to only explore a very local region to my starting location?
Is this approach a good way to accomplish this task (assuming I want to use RL)?
I would think that attempting to work with a "continuous" action space (model outputs 2 values: vertical and horizontal indices of grid cells that contain 1s) would be more difficult to achieve convergence. Is it wise to always try to use discrete action spaces?

What does your Q-network look like?

If you're using Convolutional Neural Networks (CNNs), I would advise against using pooling layers since the spatial translation data matters in this application (i.e. a cell with '1' top right of the agent is not the same as a cell with '1' below).

  1) How I might encourage exploration that will not always "cancel out" to only explore a very local region to my starting location?

This is referred to as exploration vs exploitation dilemma.

One way to add tackle this is by adding extra logic to the way you're choosing random actions. To elaborate, the chosen random action shouldn't be the same as the opposite of the previous action: if the agent went LEFT, the next random shouldn't be RIGHT as that is not exploring.

Another method is to use different action selection strategies such as Boltzmann Policy rather than epsilon-greedy. Click here for a quick crash course on the mainstream strategies by Arthur Juliani.

  2) Is this approach a good way to accomplish this task (assuming I want to use RL)?

RL is well suited for this application. Try different parameters for your network. I'd start with a small network and add more layers/neurons as necessary if it does not converge. However, keep in mind that it might take a while (hours or days) to converge so don't get hasty.

  3) I would think that attempting to work with a "continuous" action space (model outputs indexes of "1" elements) would be more difficult to achieve convergence. Is it wise to always try to use discrete action spaces?

I'm not sure I fully understand your question. Mind elaborating? What do you mean by the indexes of "1" elements that the model outputs?


Learning policy where action involves discrete and continuous parameters Typically it seems like reinforcement learning involves learning over either a discrete or a continuous action space. An example might be choosing from a set of pre-defined game actions in Gym Retro or learning the right engine force to apply in Continuous Mountain Car; some popular approaches for these problems are deep Q-learning for the former and actor-critic methods for the latter.

What about in the case where a single action involves picking both a discrete and a continuous parameter? For example, when choosing the type (discrete), pixel grid location (discrete), and angular orientation (continuous) of a shape from a given set to place on a grid and optimize for some reward.
Is there a well-established approach for learning a policy to make both types of decisions at once?

There is a recent paper: Continuous-Discrete Reinforcement Learning for
Hybrid Control in Robotics published by DeepMind that aims to solve this problem, as stated in the abstract:

  Many real-world control problems involve both discrete decision variables – such as the choice of control modes, gear switching or digital outputs – as
  well as continuous decision variables – such as velocity setpoints, control gains or
  analogue outputs. However, when defining the corresponding optimal control or reinforcement learning problem, it is commonly approximated with fully continuous
  or fully discrete action spaces. These simplifications aim at tailoring the problem
  to a particular algorithm or solver which may only support one type of action
  space. Alternatively, expert heuristics are used to remove discrete actions from
  an otherwise continuous space. In contrast, we propose to treat hybrid problems
  in their ‘native’ form by solving them with hybrid reinforcement learning, which
  optimizes for discrete and continuous actions simultaneously.

The idea is that they use a hybrid policy that uses a Gaussian distribution for the continuous decision variables and a categorical distribution for the discrete decision variables. Then, they extend the Maximum a Posteriori Policy Optimisation (MPO) algorithm (also by DeepMind) to allow it to handle hybrid policies.

Here is a video showing how they used the resulting hybrid MPO policy in a robotics task, where in addition to the continuous actions, the robot can choose a discrete action which is the control mode to be used (coarse vs. fine).


What could cause a big fluctuation of the loss in the last epochs of training an AlexNet? I am training an AlexNet neural network, with about 12000 images which 80% is for training, 10% is for validation and another 10% is for testing.
I have a problem in my plots. There is a big fluctuation in epoch 47,
how can I have a smooth plot? What is the problem?

I tried to increase my validation data because the fluctuation was for validation loss. but nothing changed.
I decreased learning rate, but it stucks in a local optimum.

I solved my problem with data augmentation and decreasing learning rate value and more epochs.


What are examples of problems where neural networks have achieved human-level or higher performance? What are examples of problems where neural networks have been used and have achieved human-level or higher performance?
Each answer can contain one or more examples. Please, provide links to research papers or reliable articles that validate your claims.

Here is an initial list of AI systems that used neural networks and have achieved human-level or superhuman performance. All of these systems are reinforcement learning systems that play videogames.

AlphaGo and AlphaGo Zero (an improved version of AlphaGo that does not use human knowledge but learns by playing against itself) have achieved superhuman performance in the game of go and, in the case of AlphaZero (a generalized version of AlphaGo Zero), also in the games of chess and shogi.
DQN has achieved human-level or superhuman performance in many Atari games
DeepStack has achieved human-level performance in poker
AlphaStar defeated a top professional player in the real-time strategy game StarCraft 2
OpenAI Five defeated world champions in the game of Dota


Do I need to know in advance all possible number of states in Q-Learning? In Q-learning, is it mandatory to know all possible states that can the agent may end up in?
I have a network with 4 source nodes, 3 sink nodes, and 4 main links. The initial state is the status network where the sink nodes have its resources at its maximum. In a random manner, I generate service requests from the source nodes to the sink nodes. These service requests are generated at random timesteps, which means that, from state to state, the network status may stay the same.
When a service request is launched, the resources from the sink nodes change, and the network status changes.
The aim of the agent is to balance the network by associating each service request to a sink node along with the path.
I know that in MDP you are supposed to have a finite set of states, my question is: if that finite set of states is supposed to be all possible states that can happen, or is just a number that you consider enough to optimize the Q-table?

When you start off learning about Q-learning, you start with a simple example that has a few states. For each of the states, you try to estimate what the 'value' is of that state. Because there are so few states, it is possible to store these values in a table (it is also useful for the intuitiveness of the explanation).
However, if you start trying to solve more 'real-life' problems, the number of states can be insanely huge. The essence however will stay the same. You are trying to estimate what you want to do next, based on an estimation of how good each state is that you can end up in. However, now the values are 'most of the times' not stored in a table anymore, as the approach will often come down to using an ANN to estimate the value function.
Answer to your question: You are going to run into a lot of problems when training some model with Q-learning if your table is not able to store the values of possible states that it can come across. In practice, most implementations do not use a Q-table and just use an ANN, which alleviates the problem of having to 'define' how many states your problem consists of.


What kind of neural network architecture do I use to classify images into one hundred thousand classes? I have an image dataset where objects may belong to one of the hundred thousand classes.

What kind of neural network architecture should I use in order to achieve this?

Classification tasks with a large number of classes are usually handled with hierarchical softmax to reduce the complexity of the final layer. This is useful, for example, in applications such as word embedding where you have hundreds of thousands of classes (words), like in your case.


How can AI researchers avoid "overfitting" to commonly-used benchmarks as a community? In fields such as Machine Learning, we typically (somewhat informally) say that we are overfitting if improve our performance on a training set at the cost of reduced performance on a test set / the true population from which data is sampled.

More generally, in AI research, we often end up testing performance of newly proposed algorithms / ideas on the same benchmarks over and over again. For example:

For over a decade, researchers kept trying thousands of ideas on the game of Go.
The ImageNet dataset has been used for huge amounts of different publications
The Arcade Learning Environment (Atari games) has been used for thousands of Reinforcement Learning papers, having become especially popular since the DQN paper in 2015.

Of course, there are very good reasons for this phenomenon where the same benchmarks keep getting used:

Reduced likelihood of researchers "creating" a benchmark themselves for which their proposed algorithm "happens" to perform well
Easy comparison of results to other publications (previous as well as future publications) if they're all consistently evaluated in the same manner.

However, there is also a risk that the research community as a whole is in some sense "overfitting" to these commonly-used benchmarks. If thousands of researchers are generating new ideas for new algorithms, and evaluate them all on these same benchmarks, and there is a large bias towards primarily submitting/accepting publications that perform well on these benchmarks, the research output that gets published does not necessarily describe the algorithms that perform well across all interesting problems in the world; there may be a bias towards the set of commonly-used benchmarks.

Question: to what extent is what I described above a problem, and in what ways could it be reduced, mitigated or avoided?

Great question Dennis!

This is a perennial topic at AI conferences, and sometimes even in special issues of journals. The most recent one I recall was Moving Beyond the Turing Test in 2015, which ended up leading to a collection of articles in AI magazine later that year.

Usually these discussions cover a number of themes:

"Existing benchmarks suck". This is usually the topic that opens discussion. In the 2015/2016 discussion, which focused on the Turing Test as a benchmark specifically, criticisms ranged from "it doesn't incentivize AI research on the right things", to claims that it was poorly defined, too hard, too easy, or not realistic.
General concensus that we need new benchmarks.
Suggestions of benchmarks based on various current research directions. In the latest discussion this included answering standardized tests for human students (well defined success, clear format, requires linking and understanding many ares), playing video games (well defined success, requires visual/auditory processing, planning, coping with uncertainty), and switching focus to robotics competitions.

I remember attending very similar discussions at machine learning conferences in the late 2000's, but I'm not sure anything was published out of it.

Despite these discussions, AI researchers seem to incorporate the new benchmarks, rather than displacing the older ones entirely. The Turing Test is still going strong for instance. I think there are a few reasons for this.

First, benchmarks are useful, particularly to provide context for research. Machine learning is a good example. If the author sets up an experiment on totally new data, then even if they apply a competing method, I have to trust that they did so faithfully, including things like optimizing the parameters as much as with their own methods. Very often they do not do this (it requires some expertise with competing methods), which inflates the reported advantage of their own techniques. If they also run their algorithm on a benchmark, then I can easily compare it to the benchmark performances reported by other authors, for their own methods. This makes it easier to spot a technique that's not really effective.

Second, even if new benchmarks or new problems are more useful, nobody knows about them! Beating the current record for performance on ImageNet can slingshot someone's career in a way that top performance on a new problem simply cannot.

Third, benchmarks tend to be things that AI researchers think can actually be accomplished with current tools (whether or not they are correct!). Usually iterative improvement on them is fairly easy (e.g. extend an existing technique). In a "publish-or-perish" world, I'd rather publish a small improvement on an existing benchmark than attempt a risker problem, at least pre-tenure.

So, I guess my view is fixing the dependence on benchmarks involves fixing the things that make people want to use them:

Have some standard way to compare techniques, but require researchers to also apply new techniques to a real world problem.
Remove the career and prestige rewards for working on benchmark problems, perhaps by explicitly tagging them as artificial.
Remove the incentives for publishing often.


Is doing a master's in natural language processing combined with machine learning worth it? Is doing a master's in natural language processing combined with machine learning worth it? What are the opportunities that can be opened by having such a master's?

The combination of NLP and ML is an interesting one. The most interesting work is not the mainstream systems employing feed forward, convolution kernel, and LSTM networks with word-to-vector input normalization or straight audio samples as input data.

The kinds that are most interesting are the kinds that more closely mimic the systems in which natural language developed, which are those of the language networks in the brain, including auditory, visual, and motor pathways and the kinds of cognitive abilities that support conversations beyond chat with no underlying comprehension or belief.

That some balanced rise and fall of beliefs and semantic associations are part of real conversation has been known for a few decades. The AI research directions along these lines include semantic nets, recursive artificial networks (an advanced type of RNN just entering the scene), and fuzzy belief models. Some of the current perspectives centered on letter, word, and grammar are obsolete from a linguistics point of view. Contemporary linguistics do not assume words are the elementary objects and formal grammar rules describe legal combinations of them. Smaller elements such as word roots, prefixes, suffixes, conjugations, phonetics, adjacency, and the dynamic aspects of language are more accurate modeling elements of real talk and writing.

Looking up some of these terms in an academic article search engine can pull up some of this research over the last two or three decades so that a historical background is clearer. Many of those articles have proposed systems, mathematics to describe the perspective, and sometimes algorithms and results of testing the modality. After spending a few days looking around, it will be easier to pick a master's thesis or a group of them to run past those at a university.

Applications for deeper forms of natural language capability beyond idiot talk chatbots is so profoundly needed, there will be little need to investigate the career options for most of these lines of research, provided the feasibility of developing components that fit into products and services is high for that line of research.


Does the recent advent of a Go playing computer represent Artificial Intelligence? I read that in the spring of 2016 a computer Go program was finally able to beat a professional human for the first time.

Now that this milestone has been reached, does that represent a significant advance in artificial intelligence techniques or was it just a matter of even more processing power being applied to the problem?

What are some of the methods used to program the successful Go-playing program?

Are those methods considered to be artificial intelligence?

There are at least two questions in your question:

  What are some of the methods used to program the successful go playing program?

and

  Are those methods considered to be artificial intelligence?

The first question is deep and technical, the second broad and philosophical.

The methods have been described in: Mastering the Game of Go with Deep Neural Networks and Tree Search.

The problem of Go or perfect information games in general is that:

  exhaustive search is infeasible.

So the methods will concentrate on shrinking the search space in an efficient way.

Methods and structures described in the paper include:

learning from expert human players in a supervised fashion
learning by playing against itself (reinforcement learning)
Monte-Carlo tree search (MCTS) combined with policy and value networks

The second question has no definite answer, as you will have at least two angles on AI: strong and weak.

  All real-world systems labeled "artificial intelligence" of any sort are weak AI at most.

So, yes, it is artificial intelligence, but it is non-sentient.


Will an AI ever understand its own functionality? Assuming I have a quite advanced AI with consciousness which can understand the basics of electronics and software structures.

Will it ever be able to understand that its consciousness is just some bits in memory and threads in an operating system?

This is a great question, elements of which I have also been pondering on, though we are very far from being able to actually wrestle with it algorithmically.  This question raises all kinds of metaphysical questions (Kant himself showed that pure reason is not sufficient for all questions, but I'm going to avoid that rabbit hole and focus on the mechanics of your question.)

Consciousness: This is distinct from self-awareness, and fundamentally, may be said to require only awareness of something.

  Consciousness, most scientists argue, is not a universal property of all matter in the universe. Rather, consciousness is restricted to a subset of animals with relatively complex brains. The more scientists study animal behavior and brain anatomy, however, the more universal consciousness seems to be. A brain as complex as the human brain is definitely not necessary for consciousness.
  Source: Scientific American "Does Self-Awareness Require a Complex Brain?"

Thus, an automata that receives input may be said to be consciousness, with the caveat that this idea is probably still considered radical.  The key is distinguishing mere "consciousness" from much more complex concepts such as self-awareness.

Self-Awareness: the holy grail.  This is the idea that a set of elements, such as a human organism, is aware of itself.

But this is sticky, because automata that use Machine Learning are "aware" of themselves in that the may modify their "thought" process and even their "physical" structure.

But ML systems are certainly not self-aware in the human sense.  A question might be, is this simply a function of these systems not being full Algorithmic General Intelligences, or is there more to it?  If there is more to it, is it strictly a metaphysical question, or can an answer be derived through purely rational means?  Even if the latter were the case, there is still the problem of subjectivity, as in: "Is the automata truly self-aware or is it just mimicking self-awareness?" which brings us back to the metaphysical question of "Is there a difference?".

However,

If there were a full Algorithmic General Intelligence that had consciousness equatable with human consciousness, that was aware, and even able to work with the basic components of it's corpus*, it would certainly be able to grasp that it's consciousness is a function of the "bits and bytes", just as a human is aware we are soft machines, and that our consciousness is a function of our bodies and minds.

I intentionally use corpus because it relates both to text (which may be code or even a string of bits in its most reduced form, per the concept of a Turing Machine) and also has an anatomical meaning, as in the body of an organism.  Corpus comes from the Latin and the extension of its meaning to include matter-as-information is modern.


Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network? There are proofs for the universal approximation theorem with just 1 hidden layer.
The proof goes like this:

Create a "bump" function using 2 neurons.

Create (infinitely) many of these step functions with different angles in order to create a tower-like shape.

Decrease the step/radius to a very small value in order to approximate a cylinder. This is what I'm not convinced of

Using these cylinders one can approximate any shape. (At this point it's basically just a packing problem like this.

In this video, minute 42, the lecturer says

In the limit that's going to be a perfect cylinder. If the cylinder is small enough. It's gonna be a perfect cylinder. Right ? I have control over the radius.

Here are the slides.

Here is a pdf version from another university, so you do not have to watch the video.
Why am I not convinced?
I created a program to plot this, and even if I decrease the radius by orders of magnitude it still has the same shape.
Let's start with a simple tower of radius 0.1:

Now let's decrease the radius to 0.01:

Now, you might think that it gets close to a cylinder, but it just looks like it is approximating a perfect cylinder, because of the zoomed out effect.
Let's zoom in:

Let's decrease the radius to 0.0000001.

Still not a perfect cylinder. In fact, the "quality" of the cylinder is the same.
Python code to reproduce (requires NumPy and matplotlib): https://pastebin.com/CMXFXvNj.
So my questions are:
Q1 Is it true that we can get a perfect cylinder solely by decreasing the radius of the tower to 0 ?
Q2 If this true, why is there no difference when I plot it with different radii(0.1, vs 1e-7) ?
Both towers have the same shape
Clarification: What do I mean with: same shape ? Let's say we calculate the volume of an actual cylinder(Vc) with the same raius and height as our tower and divide it by the volume of the tower(Vt) .
Vc = Volume Cylinder
Vt = Volume Tower
ratio(r) = Vc/Vt
What this documents/lectures claim that is the ratio of these 2 volumes depends on the radius but in my view it's just constant.
So what they are saying is that: lim r -> 0 for ratio(r) = 1
But my experiments show that: lim r -> 0 for ratio(r) = const and don't depend on the radius at all.
Q3 Preface
An objection i got multiple times once from Dutta and once from D.W is that just decreasing the radious and plotting it isn't mathematical rigorous.
So let's assume in the limit of r=0 it's really a perfect cylinder.
One possible explanation for this would be that the limit is a special case and one can't approximate towards it
But if that is true this would imply that there is no use for it since it's impossible to have a radius of exactly zero. It would only be useful if we could get gradually closer to a perfect cylinder by decreasing the radius.
Q3 So why should we even care about this then ?
Further Clarifications
The original universal approximation theorem proof for single hidden layer neural networks was done by G. Cybenko. Then I think people tried to make some visual explations for it. I am NOT questioning the paper ! But i am questioning the visual explanation given in the linked lecutre/pdf (made by other people)

The more I think about it the more convinced I am that the visual explanation from the linked lecture is wrong. But the good news is there are still some ways to get close to the cylinder but not before the activation of the last neuron but instead afterwards. I haven't done it with simgoid. But I tried with ReLu instead for now.
We can cut the tower at the very top (thanks to ReLu and a bias). The closer to the top we cut it the more it will be like a cylinder.
We can control the height of the tower with the weights.
First in 2d:

Unfortunately the closer we put this towers together the more they will start two influence each other.

But we can counter that with a negative tower between them.

Now in 3d:

This Answer is work in progress I will update it when i find out something new.


Maxpooling in inception? Maxpooling is performed as one of the steps in inception which yields same output dimension as that of the input.
Can anyone explain how this max pooling is performed?

My bad, didn't look into the block diagram first

maxpooling filter/kernel is 3x3 = fxf

Formula for padding to get the same dimension as the input

p = (f - 1)/2

here f=3 ,so padding to be performed before maxpooling is (3-1)/2 = 1


Is there a simple way of classifying images of size differing from the input of existing image classifiers? Most image classifiers like Inception-v3 accept images of about size 299 x 299 x 3 as input. In this particular case, I cannot resize the image and lose resolution. Is there an easy solution of dealing with this rather than retraining the model? (Particularly in tensorflow)

My guess:
If you add a few CNN layers before the input of the given model and train only those layers while keeping the given model's parameters frozen, you might get better result.
Essentially these few extra layers would "transform" your input image into the appropriate shape, but with more accuracy since its trained and not hard coded.


ANNs with multiple activation outputs Interested to know if there was any use or interest in activation functions with more than one output value to the next column instead of  single firing.

I'm interested to know if this would have any use or would be almost identical to a single value firing.

I think this is an interesting approach, but it's hard to tell if it can make better or faster learning networks.

If I understand correctly, this could be looked at as using a function returning a tuple as activation function. The number of elements in the tuple must be equal to the number of neurons in the next layer.

For example you could use an activation function like f(x) = (ReLU(x), sigmoid(x)) if the next layer contains two neurons.

I think the biggest problem is to determine why one value should be passed to exactly THAT neuron and another value to another neuron - why not opposite? This must either be determined randomly or by doing some advanced analysis in beforehand.

Anyway, I think it's an interesting idea which should be tested out, but it's hard to say whether it will give better or worse results before trying it out.


What does the Fourier transformed image mean? I have been trying to figure out what the Fourier transformed image represents. I am aware of Fourier transformation in general, but I can't explain myself the image it forms after transformation.

In the given image, what does the outlined white sort of lines mean?

Have a look at this explanation of the two dimensional Fourier Transform applied to an image.

The way I read it is that it somewhat mirrors the diagonal structure of the 'X', as that is where most of the 'image energy' is, so that are the lines you highlighted. Because the image is fairly uniform in colour, most of the components are low-frequency (ie close to the centre of the 2DFT image).

Disclaimer: I'm not really that familiar with a 2D version either, as I mainly have experience with using FFTs in speech processing.


Does adding a constant to all rewards change the set of optimal policies in episodic tasks? I'm taking a Coursera course on Reinforcement learning. There was a question there that wasn't addressed in the learning material: Does adding a constant to all rewards change the set of optimal policies in episodic tasks?

The answer is Yes - Adding a constant to the reward signal can make longer episodes more or less advantageous (depending on whether the constant is positive or negative).

Can anyone explain why is this so? And why it doesn't change in the case of continuous (non episodic) tasks? I don't see why adding a constant matters - as an optimal policy would still want to get the maximum reward...

Can anyone give an example of this?

Generally we can write for $R_c$ the total reward with added constant $c$ of a policy as
$$
R_c = \sum_{i=0}^K (r_i + c) \gamma^i = \sum_{i=0}^K r_i \gamma^i + \sum_{i=0}^K c \gamma^i
$$
So if we have two policies with the same total reward (without added constant)
$$
\sum_{i=0}^{K_1} r_i^1 \gamma^i = \sum_{i=0}^{K_2} r_i^2 \gamma^i
$$
but with different lengths $K_1 \neq K_2$ the total reward with added constant will be different, because the second term in $R_c$ ( $\sum_{i=0}^K c \gamma^i$ ) will be different.

As an example: Consider two optimal policies, both generating the same cumulative reward of 10, but the first policy visits 4 states, before it reaches a terminal state, while the second visits only two states. The rewards can be written as:
$$
10 + 0 + 0 + 0 = 10
$$
and
$$
0 + 10 = 10
$$
But when we add 100 to every reward:
$$
110 + 100 + 100 + 100 = 410
$$
and
$$
100 + 110 = 210
$$
Thus, now the first one is better.

In the continious case, the episodes always have length $K = \infty$. Therefore, they always have the same length, and adding a constant doesnt change anything, because the second term in $R_c$ stays the same.


What are the aspects that most impact on the inference time for neural networks in embedded systems? I work with neural networks for real-time image processing on embedded softwares and I tested different architectures (Googlenet, Mobilenet, Resnet, custom networks...) and different hardware solutions (boards, processors, AI accelerators...). I noticed that the performance of the system, in terms of inference time, does not depend only on the processor but also on other factors.

For example, I have two boards from different manifacturers, B1 (with a cheap processor) and B2 (with a better processor), and two neural networks, N1 (very light with regular convolutions and fully connected layers) and N2 (very large, with inception modules and many layers). The inference time for N1 is better on B1, while for N2 it is better on N2. Moreover, it happens that, as the software is executed, the inference time changes over time.

So my question is: in an embedded system, what are the aspects that impact on the inference time, and how? I am interested not only in the hardware features but also in the neural network architecture (convolutional filter size, types of layers and so on).

You can expect that the inference time will strongly depend on particular hardware and software present on your platform. First, GPU equipped devices (eg NVidia TX) will outperform non-GPU equipped devices (eg. Intel Movidius). Second, software support (eg. cudnn, TensorRT) will make dramatic further impact.

For instance, we have measured the inference time of two convolutional models. The model A requires 250% more floating point operations than the model B. Yet, the two models take roughly the same time to evaluate on our device, since the layers of model A are better optimized in software. Conclusion: algorithmic complexity and practical execution time on a particular computing platform are not bound to be proportional any more.


What are the biggest barriers to get RL in production? I am studying the state of the art of Reinforcement Learning, and my point is that we see so many applications in the real world using Supervised and Unsupervised learning algorithms in production, but I don't see the same thing with Reinforcement Learning algorithms.
What are the biggest barriers to get RL in production?

There is a relatively recent paper that tackles this issue: Challenges of real-world reinforcement learning (2019) by Gabriel Dulac-Arnold et al., which presents all the challenges that need to be addressed to productionize RL to real world problems, the current approaches/solutions to solve the challenges, and metrics to evaluate them. I will only list them (based on the notes I had taken a few weeks ago). You should read the paper for more details. In any case, for people that are familiar with RL, they will be quite obvious.

Batch off-line and off-policy training

One current solution is importance sampling

Learning on the real system from limited samples (sample inefficiency)

Solutions: MAML, use expert demonstrations to bootstrap the agent, model-based approaches

High dimensional continuous state and action spaces

Solutions: AE-DQN, DRRN

Satisfying safety constraints

Solutions: constrained MDP, safe exploration strategies, etc.

Partial observability and non-stationarity

Solutions to partial observability: incorporate history in the observation, recurrent neural networks, etc.
Solutions to non-stationarity: domain randomization or system identification

Unspecified and multi-objective reward functions

Solutions: CVaR, Distributional DQN

Explainability
Real-time inference
System delays (see also this and this answers)

There's also a more recent and related paper An empirical investigation of the challenges of real-world reinforcement learning (2020) by Gabriel Dulac-Arnold et al, and here you have the associated code with the experiments.
However, note that RL (in particular, bandits) is already being used to solve at least one real-world problem [1, 2]. See also this answer.


Can recovering a reward function using IRL lead to better policies compared to reward shaping? I am working on a research project about the different reward functions being used in the RL domain. I have read up on Inverse Reinforcement Learning (IRL) and Reward Shaping (RS). I would like to clarify some doubts that I have with the 2 concepts.
In the case of IRL, the goal is to find a reward function based on the policy that experts take. I have read that recovering the reward function that experts were trying to optimize, and then finding an optimal policy from those expert demonstrations has a possibility of resulting in a better policy (e.g. apprenticeship learning). Why does it lead to a better policy?

Inverse Reinforcement Learning (IRL) is a technique that attempts to recover the reward function that the expert is implicitly maximising based on expert demonstrations. When solving reinforcement learning problems, the agent maximises a reward function specified by the designer, and in the process of reward maximisation, accomplishes some task that it had set out to do. However, reward functions for certain tasks are sometimes difficult to specify by hand. For example the task of driving takes into consideration many different factors such as the distance of the car in front of him, the road conditions and whether or not the person needs to get to his destination quickly. A reward function can be hand specified based on these features. However, when there exists trade offs between these different features, it is difficult to know how to specify the different desiderata of these tradeoffs.
Instead of specifying the trade offs manually, it would be easier to recover a reward function from expert demonstrations using IRL. Such a reward function can lead to better generalisations to unseen states as long as the features of driving do not change.
In the case where reward shaping fails to learn a task (such as driving), it would be better to have someone demonstrate a task and learn a reward function from these demonstrations. Solving the MDP with the learnt reward function will thus yield a policy that should resemble the demonstrated behaviour. The reward function learnt should also generalise to unseen states and the agent acting in unseen states should be able to perform actions that an expert would take when he is placed in the same conditions, assuming that the unseen states come from the same distribution as the training states.
While Reward Shaping might be able to perform the same task as well, IRL might be able to do better, based on some performance metric that will differ from problem to problem.


How important is it that the generator of a generative adversarial network doesn't take in information about input classes? I'm building a generative adversarial network that generates images based on an input image. From the literature I've read on GANs, it seems that the generator takes in a random variable and uses it to generate an image.

If I were to have the generator receive an input image, would it no longer be a GAN? Would the discriminator be extraneous?

If you're building a straight "vanilla" generative adversarial network, it's best to understand the network as a statistical engine:  You are training the generator on samples of a statistical distribution.  (And you're training the discriminator to distinguish between "ground truth" images, and images from that generator.)

Once you replace the input noise with another image... well.  Strictly speaking, it is probably still a generative adversarial network, if you're still doing everything else the same.  It is still a generator and a discriminator, acting in an adversarial fashion.

But you've radically altered the input distribution, so there is a good chance that you're no longer accomplishing what you want to accomplish unless you're being very careful and clever.

That said, there are GAN variants which do take images rather than noise as inputs.  See the wonderful paper on CycleGANs by Zhu, et al, along with a substantial body of followup literature. And note that CycleGANs use not one, but two discriminators, so even here the discriminator is necessary.


Is policy learning and online system identification the same? In some newer robotics literature, the term system identification is used in a certain meaning. The idea is not to use a fixed model, but to create the model on the fly. So it is equal to a model-free system identification. Perhaps a short remark for all, who doesn't know what the idea is. System identification means, to create a prediction model, better known as a forward numerical simulation. The model takes the input and calculates the outcome. It's not exactly the same like a physics engine, but both are operating with a model in the loop which is generating the output in realtime.

But what is policy learning? Somewhere, I've read that policy learning is equal to online system identification. Is that correct? And if yes, then it doesn't make much sense, because reinforcement learning has the goal to learn a policy. A policy is something which controls the robot. But if the aim is to do system identification, than the policy is equal to the prediction model. Perhaps somebody can lower the confusion about the different terms ...

Example Q-learning is a good example for reinforcement learning. The idea is to construct a q-table and this table controls the robot movements. But, if online-system-identification is equal to policy learning and this is equal to q-learning, then the q-table doesn't contains the servo signals for the robot, but it provides only the prediction of the system. That means, the q-table is equal to a box2d physics engine which can say, what x/y coordinates the robot will have. This kind of interpretation doesn't make much sense. Or does it make sense and the definition of a policy is quite different?

From the book Reinforcement Learning, An Introduction (R. Sutton, A. Barto):

  The term system identification is used in adaptive control for what we
  call model-learning (e.g., Goodwin and Sin, 1984; Ljung and S
  ̈oderstrom, 1983; Young, 1984).

Model-learning refers to the act of learning the model (environment). Reinforcement Learning can be divided into two types:

Model-based - first we build a model of an environment and then do the control.
Model-free - we do not try to model the behaviour of the environment.

Policy learning is the act of learning optimal policy. You can do it in two ways:

On-policy learning - learn about the policy $π$ by sampling from the same policy.
Off-policy learning - learn about the policy $π$ from the experience sampled from some other policy (e.g. watching different agent playing a game).


Why does TD Learning require Markovian domains? One of my friends and I were discussing the differences between Dynamic Programming, Monte-Carlo, and Temporal Difference (TD) Learning as policy evaluation methods - and we agreed on the fact that Dynamic Programming requires the Markov assumption while Monte-Carlo policy evaluation does not.
However, he also pointed out that Temporal Difference Learning cannot handle non-Markovian domains, i.e. it depends on the Markov assumption. Why is it so?
The way I understand it, the TD learning update is, in essence, the same as the Monte-Carlo update, except for the fact that the return instead of being calculated using the entire trajectory, is bootstrapped from the previous estimate of the value function, i.e. we can update the value as soon as we encounter a $(s,a,r,s')$ tuple, we don't have to wait for the episode (if finite) to terminate.
Where is the Markov assumption being used here, i.e the future is independent of the past given the present?
Thank you!

The Markov assumption is used when deriving the Bellman equation for state values:
$$v(s) = \sum_a \pi(a|s)\sum_{r,s'} p(r,s'|s,a)(r + \gamma v(s'))$$
One requirement for this equation to hold is that $p(r,s'|s,a)$ is consistent. The current state $s$ is a key argument of that function. There is no adjustment for history of previous states, actions or rewards. This is the same as requiring the Markov trait for state, i.e. that $s$ holds all information necessary to predict outcome probabilities of the next step.
The one step TD target that is sampled in basic TD learning is simply the inner part of this:
$$G_{t:t+1} = R_{t+1} + \gamma \hat{v}(S_{t+1})$$
which when sampled is equal to $v(s)$ in expectation *, when $S_t = s$. That is, when you measure a single instance of the TD target and use it to update a value function, you implicitly assume that the values or $r_{t+1}$ and $s_{t+1}$ that you observed occur with probabilities determined by $\pi(a|s)$ and $p(r,s'|s,a)$ as shown by the Bellman equation.
So the theory behind TD learning uses the Markov assumption, otherwise the sampled TD targets would be incorrect.
In practice you can get away with slightly non-Markov environments - most measurements of state for machinery are approximations that ignore details at some level, for instance, and TD learning can solve optimal control in many robotics environments. However, Monte Carlo methods are more robust against state representations that are not fully Markov.

* Technically this sample is biased because $\hat{v}(S_{t+1})$ is not correct when learning starts. The bias reduces over time and multiple updates. So the expected value during learning is approximately the same as the true value as shown by the Bellman equation.


How to handle rectangular images in convolutional neural networks? Almost all the convolutional neural network architecture I have come across have a square input size of an image, like $32 \times 32$, $64 \times 64$ or $128 \times 128$. Ideally, we might not have a square image for all kinds of scenarios. For example, we could have an image of size $384 \times 256$

My question is how to we handle such images during

training,
development, and
testing

of a neural network?

Do we force the image to resize to the input of the neural network or just crop the image to the required input size?

I think the squared image is more a choice for simplicity.

There are two types of convolutional neural networks

Traditional CNNs: CNNs that have fully connected layers at the end, and
fully convolutional networks (FCNs): they are only made of convolutional layers (and subsampling and upsampling layers), so they do not contain fully connected layers

With traditional CNNs, the inputs always need to have the same shape, because you flatten the last convolutional layer, with a fixed size. As the flatten layer has a fixed size, the feature map shape from the layer before has to be the same shape, and so the inputs (images) also have to.

However, in FCN, you don't flatten the last convolutional layer, so you don't need a fixed feature map shape, and so you don't need an input with a fixed size.

In both cases, you don't need a squared image. You just have to be careful in the case you use CNN with a fully connected layer, to have the right shape for the flatten layer.

For instance, if you have an input size $320 \times 160$, and you have 3 pooling layers, so your output in the last convolutional layer is $40 \times 20 \times c$ (with $c$ the number of filters/channels), then you just need the flatten layer to have $40*20*c$ neurons.

If you create a new network, just design it to handle a rectangle image.

If you want to use an already pre-trained one, I think the better choice is to resize the image.

If the information in the cropped parts is important, maybe your prediction can be wrong (it depends if the object of interest is in the parts of the image that is cropped). Actually, in Yolo (an object recognition network), images are resized if they don't fit the input requirements. See figure 1 of the YOLO paper. It's because you don't need a high resolution to detect an object (for example, the CIFAR dataset has images of shape $32 \times 32$, but the network can still predict the correct label). So, I think that resizing your image may not affect the prediction much (unless the new size is very different from the original)


What does top N accuracy mean? Places205-VGG, a CNN trained model for 205 scene categories of Places Database with  2.5 million images Places205 dataset has top1 accuracy = 58.9% and top5 accuracy = 87.7%.

What does top1 and top5 (and, in general, top $N$) accuracy mean in the context of deep learning?

It is explained in this CrossValidated post.

Top1 accuracy means the best guess (class with highest probability) is the correct result 58.9% of the time, while top5 accuracy means the correct result is in the top 5 best guesses (5 classes with highest probabilities) 87.7% of the time.


What is the difference between backpropagation and predictive coding? Reading the high-level descriptions of backpropagation and predictive coding, they don't sound so drastically different. What is the key difference between these techniques?

I am currently reading the following paper if that helps ground the explanation:

Predictive Coding-based Deep Dynamic Neural Network for Visuomotor
Learning

I would say that these concepts are quite different, even though they might have a few things in common (or might be vaguely related).

Back-propagation is an algorithm used (in machine learning) to compute the gradient of a function with respect to its parameters. This gradient is then used by algorithms, like gradient descent, to update the parameters of the model (e.g. a neural network).

Roughly, predictive coding is a general (neuroscience) theory of how the brain builds an internal model of the external world, of how it continuously predicts the sensory inputs (from the world) using its current model of the external world, and of how it updates this internal model once the sensory inputs are actually received.

You could think of the output of an ML model, while it is being trained (using gradient descent with back-propagation), as a prediction associated with the inputs. However, note that the output of the model is often not a prediction of the actual input, but e.g. a label (which is, nonetheless, associated with the input). Furthermore, it is the model that produces an output and not the back-propagation algorithm (even though the back-propagation algorithm is often used to train such models, e.g. neural networks). We could think of back-propagation as the way of updating these predictions associated with the inputs, but, anyway, it is well known (and accepted) that our brain does not perform back-propagation, but we learn in an associative fashion (Hebbian learning).


Changes in flow detection neural network? Do you have any advice, what architecture of neural network is the best for following task?

Let input be some (complex function), the neural network gains a flow of its values, so I guess there will be some kind of RNN or CNN?

The output is classifier like is the function same or not

If the neural network thinks, that the input is still the same function, the output is 0.
If the input function changes, the output will be 1.

The input function is of course not one value or simple math function (what will be trivial) but may be really sophisticated. So the neural network learns abstraction about same and different over any complex flow ?

How would you approach to that task ?

A network is able to fit to a certain function over several iteration while training. Now you want the model to be able to detect a change in a list of inputs from the function. This is not possible without first training the model on some data.

Say you want to use a simple function

# Sample function
f(x) = x

Say you create inputs for function using sets of 3 integers for x.

# Data
f1 = [[1, 2, 3],
      [4, 5, 6],
      ...      ]

# Some random values
f2 = [[1, 4, 19],
      [16, 35, 36],
      ...      ]

Now if you use data f1 labelled as 0 and data f2 labelled as 1, in the best case the ANN will only learn to differentiate between data from function f1 and some other data.

To detect change, first the model has to fit to a certain function, this requires it to be trained over a number of epochs. Then it will be able to detect if the values don't match the function, but such a model will only be able to differentiate a single function.


Google's Principles of Artificial Intelligence Earlier this month, Google released a set of principles governing their AI development initiatives.  The stated principles are:

  Objectives for AI Applications:

  Be socially beneficial.
  Avoid creating or reinforcing unfair bias.
  Be built and tested for safety.
  Be accountable to people.
  Incorporate privacy design principles.
  Uphold high standards of scientific excellence.
  Be made available for uses that accord with these principles.

  AI Applications not to be Pursued:

  Technologies that cause or are likely to cause overall harm. Where there is a material risk of harm, we will proceed only where we believe that the benefits substantially outweigh the risks, and will incorporate appropriate safety constraints.
  Weapons or other technologies whose principal purpose or implementation is to cause or directly facilitate injury to people.
  Technologies that gather or use information for surveillance violating internationally accepted norms.
  Technologies whose purpose contravenes widely accepted principles of international law and human rights.
  SOURCE: Artificial Intelligence at Google: Our Principles

My questions are:

Are this guidelines sufficient?
Are there any "I, Robot" conflicts
How much does this matter if other corporations and state agencies don't hew to similar guidelines?

Artificial Intelligence at Google — Our Principles

Objectives for AI Applications

Be socially beneficial.
Avoid creating or reinforcing unfair bias.
Be built and tested for safety.
Be accountable to people.
Incorporate privacy design principles.
Uphold high standards of scientific excellence.
Be made available for uses that accord with these principles.

AI Applications We Will Not Pursue

Technologies that cause or are likely to cause overall harm. Where there is a material risk of harm, we will proceed only where we believe that the benefits substantially outweigh the risks, and will incorporate appropriate safety constraints.
Weapons or other technologies whose principal purpose or implementation is to cause or directly facilitate injury to people.
Technologies that gather or use information for surveillance violating internationally accepted norms.
Technologies whose purpose contravenes widely accepted principles of international law and human rights.

Questions

Are these guidelines sufficient?

They are not sufficiently defined to be as sufficient as a policy, not even in the paragraphs following each, but it is presented as a vision comprised of principles, which should not be expected to be as defined as policy.  Here is at least one caveat for each objective.

Half the world has would have been too poor to communicate globally had Linux Torvalds not somewhat antisocially insist on certain things.
Distributing money equally among homeless heroin addicts and entrepreneurs is considered fair by many, yet it is merely equal and perhaps monumentally unfair.
National defense may be considered for safety, but someone who survived Hiroshima or a Syrian refuge may find the definition tragic.
Google cannot be accountable to people without revealing its company confidential ranking, at which point, it would be exploited and they'd have to change it.
Incorporating privacy design principles is far from not storing data so that it cannot be subpoenaed.
The concepts of the singularity, solar panels sustainability, the climate crisis, the ample availability of fossil fuels within U.S borders, and cell communications technology are all considered scientific excellence by the general public.  That these lack the primary signs of rigor in mathematics, statistics, economics, and engineering doesn't occur to anyone, so upon what basis will Google judge?  Will they seek corroboration between theoretical models and previous ones along with empirical validation for every fact?
Being made available, if they are to be a soluble corporation, will depend on the financial condition of the customer, which is in conflict with fairness (#2) and no I, Robot precedence rules are established.

That segues into the next question, skipping the non-pursuit items, which could be similarly treated.

Are there any "I, Robot" conflicts?

The Three Laws of Asimov worked well with the robotic character R. Daneel Olivaw in the Foundation sci fi novel series.  Daneel simulated human emotion, transcended human selfishness, and maintained the three laws.  In the screenplay adaptation of his I, Robot, the three laws didn't work out as well.  VIKI decided that human freedom was in conflict with law two.

Because the eleven dos and don'ts are not codified in precise policies, there are 10! (3,628,800) potential conflicts.  There is at least one that is already clear, mentioned above.

How much does this matter if other corporations and state agencies don't hew to similar guidelines?

In a world where information may have become more powerful than money, perhaps it matters quite a bit when an information giant takes a position.

It's Wikipedia that is most questionable in that through their web service language evolution may have been democratized to a self-defeating extreme.  One can create a word definition without peer review that gains more public authority than the definition in rigorously prepared and cross-correlated encyclopedias and dictionaries.  Even in this artificial intelligence forum, where the education level is high, people define tags with Wikipedia links.  I am tempted to endeavor to remove them all from here. :)


Is gradient descent scale invariant or not? I know we should scale the input and output (assuming regression task) before we feed it to the neural network. Then the gradient descent will give the better minima much faster. But I have subtle confusion whether gradient descent with feature scale and without feature scale gives the same result or just gradient descent is not scale-invariant.

What are you hoping to get out of the answer for this question?
Feature scaling is a method you CAN(but don't have to) use, so that your algorithm performs faster and reaches better general accuracy.
I would say that on a simple regression task, where the feature value ranges do not vary a lot, the output would probably be almost the same, but as soon as you introduce data that has one feature of range 1-10, and other range of 10000-100000, that is where you would notice that you NEED to standardize/normalize your features in order to reach optimal results.
That's why it's almost a general rule to just scale your data, so your algorithm can generalize better, and you don't have to worry about your algorithm giving higher importance to a feature with higher values, instead of the one with lower values(just an example).


How is a validation set used to tune the hyperparameters in a non-biased way, if the new models depends on the values of these? I've built a neural network from the scratch, choosing arbitrary numbers for the hyperparameters: learning rate, number of hidden layers and neurons for these, number of epochs and size of mini batches. Now that I've been able to build something potentially useful (~93% of accuracy with test data, unseen by the model before), I want to focus on hyperparameter tuning.

The conceptual difference between training and validation sets is clear and makes a lot of sense. It's obvious that the model is biased towards the training set, so it wouldn't make sense to use it to tune the hyperparameters, nor for evaluating its performance.

But, how can I use the validation set for this, if changing any of the parameters enforces me to rebuild a new model again? The final prediction depends on the values of X number of MxN matrices (weights) and X number of N vectors (biases), whose values depend on the learning rate, batch size and number of epochs; and whose dimensions depends on the number and size of hidden layers. If I change any of these, I'd need to rebuild my model again. So I'd be using this validation set for training different models, ending up as in the first step: fitting a model from the scratch.

To sum up: I fall in a recursive problem in which I need to fine tune the hyperparameters of my model with unseen data, but changing any of these hyperparameters implies rebuilding the model.

This is a standard ML problem: changing hyper-parameters changes the performance of the whole model. Ideally, you'd be cross-validating hyper-parameter choices, not merely comparing on a static validation set. That being said you need to be careful with hyper-parameter optimization because you could overfit these to the peculiarities of your validation set ; cross-validation helps to some extent but really what helps is having a test set that you hardly ever test against. Ideally never before you've chosen your HPs with (cross-) validation. And test-set performance will indicate how much your  HP-optimization procedure was biased.

I'm afraid training from scratch is your only solution. This does not however mean that you have to train until the end and many hyper-parameter optimization techniques out there will help you stop training early enough so you don't waste computational resources on HPs which are not worth it. A good starting point would be this blog post by Criteo's Aloïs Bissuel: Hyper-parameter optimization algorithms: a short review


What is "Pattern Theory"? I came across Grenander's work "Probabilities on Algebraic Structures" recently and found that much of Grenander's work focused on what he called "Pattern Theory." He's written many texts on the matter and from what I've seen, they seem like an attempt to unify some mathematical underpinnings of pattern representation. However, I'm not sure what this really means in practice, nor how it relates to results we already have in learning theory. The mathematical aspect of the work is really quite intriguing, but I am skeptical as to its practicality.
Are there any applications of Grenander's pattern theory? Either for getting a better theoretical understanding of certain methods of pattern recognition or for directly implementing algorithms?
Some links to what I'm referring to: Wikipedia entry, book, first book of 3 set volume, and research group, including fields medalist David Mumford.

Pattern-matching is a classical sub-problem of AI systems. Read about the Rete algorithm, or pattern matching in Prolog. Or about Unification. Or about the Warren Abstract Machine. Or about topological data analysis like in GUDHI.
A typical example is the CLIPSrules expert system shell. Or the match expression construct of Ocaml or in Haskell.
In some ways, pattern matching is related to compilers: the GCC compiler (or the Stalin one, or Bigloo, or SBCL) is somehow doing pattern matching on ASTs. So parser generators (e.g. GNU bison, or Iburg) are related to pattern matching.
Patterns also have a different meaning in computer vision.
A classical paper is of course Derivation of a Pattern-Matching Compiler.
A more interesting approach is described in Jacques Pitrat's last book:
Artificial Beings: the conscience of a conscious machine ISBN-13: 978-1848211018
You could also look into Pitrat's blog here.
PS. Jacques Pitrat passed away in oct. 2019.


What are the main AI technologies behind the Watson platform? What AI techniques does IBM use for its Watson platform (specifically, its natural language processing part)?

As a person who works with people who work on Watson, perhaps I can give some insight.

The name Watson is casually thrown around a lot whilst many people aren't aware of its evolution into a larger suite of systems and services. We now have Chef Watson, Watson Health, and many other developing projects along the "cognitive" route. Watson is really an amalgamation and varied application of the different cognitive computing routes IBM is pursuing.

So, there are many forms of NLP that Watson conducts and has conducted, developed by different teams to fit different processes, interconnected in different ways. Additionally, much (probably all) of it is proprietary/classified since, as one would imagine, ongoing research is constantly being conducted and added to Watson. This is likely your largest obstacle. The precise workings of the NLP of the Jeopardy flavor of Watson are probably themselves still classified (I can't find anything in the time I've just spent looking myself)

There are, thus, many answers to this question; many outdated, and others not always applicable. The full answer is very complicated and, by the time you find out what the answer is today, it's probably already been advanced. The researches I know are always working on new, cutting-edge algorithms and processes for text classification and the related NLP topics. See, for example, this article http://www.aaai.org/Magazine/Watson/watson.php, where it is stated

  For the Jeopardy Challenge, we use more than 100 different techniques for analyzing natural language, identifying sources, finding and generating hypotheses, finding and scoring evidence, and merging and ranking hypotheses.

and

  Watson's metalearner uses multiple trained models to handle different question classes as, for instance, certain scores that may be crucial to identifying the correct answer for a factoid question may not be as useful on puzzle questions

Most of these techniques have likely already been modified, adapted, or dropped altogether for other methods.

To point you to more information, take a look at the links below.

General

The DeepQA Research Team, specifically, have a look at their publications
IBM Research AI

Articles

Building Watson: An Overview of the DeepQA Project (2010) written by the IBM Watson Research Team, led by David Ferucci, and published in AI Magazine Fall, 2010.

Videos

How does IBM Watson work? (2018), by IBM Watson, which states that Watson uses deep learning and transfer learning

Other

Natural language processing: an introduction (2011), a good but technical paper on natural language understanding (that acknowledges Watson)


Do all filters of the same convolutional layer need to have the same dimensions and stride? In Convolutional Neural Networks, do all filters of the same convolutional layer need to have the same dimensions and stride?
If they don't, then it would seem the channel produced by each filter would have different sizes. Or is there some way to get around that?

It seems that a similar question has been raised here: https://stackoverflow.com/questions/57438922/different-size-filters-in-the-same-layer-with-tensorflow-2-0
Like answered in the link above, you could combine severall Conv2D ops with different kernel sizes on the same input. You would have to adapt each output with padding, or cropping, so that you could concatenate all of them.
Hope this helps!


What is the difference between model and data distributions? Is there any difference between the model distribution and data distribution, or are they the same?

Yes. In Machine Learning we consider that the samples in your training set are sampled from an underlying distribution called the data generating distribution.

Generative models classify the samples by trying to learn the distribution of the data. In most cases, either the model is incapable of doing so, or the training samples aren't enough to properly describe the data-generating distribution, so the model learns an approximation of this. This is what you call the model's distribution.

You can find more info about these concepts in a more detailed answer I wrote. If you're familiar with GANs, you can also read this post, to see where these two concepts come into play when training the two networks.


How does one know that a problem is "model-free" in reinforcement learning? Consider this slide from a Stanford lecture on reinforcement learning. It states that a model is

the agent's representation of how the world changes in response to the agent's action.

I've been experimenting with Q-learning for simple problems such as OpenAI's FrozenLake and Mountain Car, which both are amenable to the Q-learning framework (the latter upon discretization). I consider the topologies of the lake and the mountain to be the "worlds" (aka. environments) in the two cases, respectively.
Q-learning is said to be "model-free". Given the two examples above, is it because neither the lake's topology nor that of the mountain are changed by the actions taken?

Q-learning is said to be "model-free". Given the two examples above, is it because neither the lake's topology nor that of the mountain are changed by the actions taken?

No. That's not why Q-learning is model-free. Q-learning assumes that the underlying environment (FrozenLake or MountainCar, for example) can be modelled as a Markov decision process (MDP), which is a mathematical model that describes problems where decisions/actions can be taken and the outcomes of those decisions are at least partially stochastic (or random). More precisely, an MDP is composed of

A set of actions $A$ (that the RL agent can take); for example, up and down, in some grid world
A set of states $S$ (where the RL agent can be);
A transition function $p(s_{t+1} = s' \mid s_{t} = s , a_t = a)$ (aka the model), which represents the probability of going to state $s'$ at time step $t+1$, given that at time step $t$ the RL agent is in the state $s$ and takes action $a$.
A reward function $r(s, a, s')$ (sometimes also denoted as $r(s)$ or $r(s, s')$, although these can have different semantics); the reward function gives the reward (or reinforcement) to the RL agent when it takes an action in a certain state and moves to another state; the reward function can also be included in the transition function, i.e., often you will also see $p(s_{t+1} = s', r_{t+1} = r \mid s_{t} = s , a_t = a)$, and this is the model: this is what we mean by model in reinforcement learning, it's this $p$ (which is a probability distribution)!

A model-free algorithm is any algorithm that does not use or estimate this $p$. Q-learning, if you look at its pseudocode, does not make use of this model. Q-learning estimates the value function $q(s, a)$ by interacting with the environment (taking actions and receiving rewards), but, meanwhile, it does not know or keep track of the dynamics (i.e. $p$) of the environment, and that's why it's model-free.
And, no, the value function is not what we mean by "model" in reinforcement learning. The value function is, as the name suggests, a function.

How does one know that a problem is "model-free" in reinforcement learning?

A problem is not model-free or model-based. An algorithm is model-free or model-based. Again, a model-free algorithm does not use or estimate $p$, a model-based one uses (and/or estimates) it.

Given the two examples above, is it because neither the lake's topology nor that of the mountain are changed by the actions taken?

No. As stated in the other answer, you could apply the model-based algorithm Dyna-Q to these environments.


Reasoning behind performance improvement with hopfield networks In the paper Hopfield networks is all you need, the authors mention that their modern Hopfield network layers are a good replacement for pooling, GRU, LSTM, and attention layers, and tend to outperform them in various tasks.
I understand that they show that the layers can store an exponential amount of vectors, but that should still be worse than attention layers that can focus parts of an arbitrary length input sequence.
Also, in their paper, they briefly allude to Neural Turing Machine and related memory augmentation architectures, but do not comment on the comparison between them.
Has someone studied how these layers help improve the performance over pooling and attention layers, and is there any comparison between replacing layers with Hopfield layers vs augmenting networks with external memory like Neural Turing Machines?
Edit 29 Jan 2020
I believe my intuition that attention mechanism should outperform hopfield layers was wrong, as I was comparing the hopfield layer that uses an input vector for query $R (\approx Q)$ and stored patterns $Y$ for both Key $K$ and Values $V$. In this case my assumption was that hopfield layer would be limited by its storage capacity while attention mechanism does not have such constraints.
However the authors do mention that the input $Y$ may be modified to ingest two extra input vectors for Key and Value. I believe in this case it would perform hopfield network mapping instead of attention and I do not know how the 2 compare.

Will try to formulate my understanding of the ideas in this paper, mention my own concerns that I see are relevant to your question, and see if we can identify any confusions along the way that might clarify the issue
On eq(6) of the relevant blog post, they identify the weight matrix of a discrete, binary Hopfield network as
$$ \boldsymbol{W} = \sum_i^N x_i x_i^T $$
with N raw stored entries, that are retrieved by iterating the initial guess $\xi$ with the following update rule
$$ \xi_{t+1} = \text{sgn}( \boldsymbol{W} \xi_{t} - b ) $$
Now to the paper in question, the update rule for the generalization they propose for continuous states that would be used is (eq 22 in OP):
$$ \xi_{t+1} = \boldsymbol{X} \text{softmax}( \beta \boldsymbol{X}^T \xi_{t} ) $$
Where $\boldsymbol{X} = (x_0, x_1, \dots , x_N ) $
The first substantial difference I see is that, while on the case of binary entries, all the weights of the network are encoded in the matrix $\boldsymbol{W}$, hence the storage is constant regardless of how many actual patterns are stored in it. In contrast, in this continuous case generalized rule, the $\boldsymbol{X}$ matrix seems to grow linearly with the size of entries, in fact it keeps all the stored entries directly. By their update rule, it doesn't seem to be a way around storing and keeping around the entire entries, and the update rule seems to only find a "best fit" among the entries, using the scalar dot product of the attention mechanism. I still think I might be missing something important here


Trying to make snake learn different directions I'm really new to neural networks. I'm trying to make a neural network with genetics algorithms which will make a snake learn to look for the food and avoid hitting his tail.

The thing is that I think that I've done it, but as there's no walls the snake learns to go one direction only without making a 180 turn [GIF HERE].

I've tried to incentivate mutations that make them turn by decreasing the score of the snakes that always takes the same directions, but it don't works. I've only made them dumber, needing more breeds to reach another "smart" linear snake.

I've made a network with 5 inputs:

Food position relative to my position and direction (2 inputs. x and y)
Nearest wall (my tail) if I turn left
Nearest wall (my tail) if I do not turn
Nearest wall (my tail) if I turn right

The output are 3, being the first one turning left, the second do not turn and the third going right. I make the snake go the highest one of the 3 outputs.

I've added 1 hidden layer of 8 neurons (inputs + outputs premise).

The way I calculate the score is:

Each step, 1 point.
Each food eaten, 10 points.
If the snake lasts too much time without eating food, dies.
If hits his tail, dies.

Then I save each time the direction this snake has gone (up, right, down, left) and increment them by one. When the snake dies, I weight the final score by the difference between the lowest and highest values. If the difference is high, they receive a big penalty (down to 0.25 of their score). This way if a snake is pretty much linear gets a high penalty and if a snake does a cool pattern, gets a low penalty.

Also, I keep a record of each time a direction change happens compared to the last direction change, so if a snake keeps going in circles don't gets a high score because of "cool pattern method" for going all 4 directions.

With all this, I don't understand why my best snakes are always the linear ones :-/

I spawn 20 snakes and get the best 4 of each generation when everyone dies.

For generations I use neataptics.js and for neural networks I use synaptics.js. I have a Fiddle here: http://jsfiddle.net/Llorx/gunsct5r/

At line 10 you can see the network definition. At 211 you can see the snake "view" (food position and walls. Where it gets the inputs) and at 164 you can see the score weight calculation depending on steps taken that I mentioned before.

All inputs are normalized from 0 to 1.

I'm sure that I'm doing, not one, but a lot of things pretty bad, as I'm a newbie on this, but some light on this will be really cool.

I observed the scoring system you used and compared it to the field size and starving rate. The worms either died or did not try enough hard with the prize of 10 of eating, so I added the amount to 1000.

This did still not introduce the 180 degrees turn yet. I had to adjust these settings on Neat:

  mutationRate: 0.7,
  mutationAmount: 6,

which introduced  where the worms optimize their path as diagonal and when forced to change 90 direction, actually a tiny 180 happens on screen.

On same mutationRate and mutationAmount, but 10 points I once observed a horizontal 180, when the food appeared back and diagonal movement was not invented. Also I observed with this last setup a motion of constant 180s flow where consecutive vertical or horizontal 180s formed a two pixel thick line on screen.

These observations do not 100% do the same as your animation, but do instead introduce various ways that are quite near, or can be in certain conditions an initiative to such phenomenom.

White worm introduces two pixel line, red 90 degree turn with miniature 180.

EDIT

I changed the nice curve ratio like this:

     var ratio = 0.75 + ((mindir / maxdir) * 0.25* difstepsratio);

and around generation 200 there lived one worm to be able to move to all directions and make 180 if needed.

Unfortunately it died before I found screen shot buttons from my phone and (s)he happened to become second in the race and thus was left as a unicorn who existed only once.

Still remains mystery whether that mutation could survive and become the main species some day.

Further remarks:

I noticed that when mutation takes place in snake code the same pattern is replicated many times. If snake gets idea of consecutive turn lefts or turn rights when it is long, a suicide with tail is quite often taking place.

I tried to taggle this with a higher score on snake collision than starvation and around generation #180 the dna of suicidal 360 loop was born and remained some generations.

However, this generation did not evolve to the desired intelligent 180 worm I previously had seen. Instead, the suicide was so rewarding that it killed that branch quite soon away.