JavaScript Client Side GAN implementation Recently I came across this website which is a year old: https://affinelayer.com/pixsrv/

On Desktop we can draw and download the trained model in browser and see the corresponding image generated on the right side.

Wondering how it works and with some investigation I have the following 2 questions:

Why are we only downloading one file per “showcase”? Because GAN should need 1 model for converting the image and 1 model for verifying the image from what I have read.

Why are the models in pict extension? Which framework did the author use to create the pict files? In linked repositories I have found TensorFlow, PyTorch etc which none of them produces pict files...

Thanks in advance.  I am very curious why such JavaScript client side demo is lacking on the internet... or I just haven’t found the correct keyword to find them out.

While training a GAN, 2 models are used. A generator and a discriminator. This training process usually takes hours (or days) to complete. This is an offline process and is not happening in the browser.
The pict file is the pre-trained model that has been imported to deeplearn.js for inference. The example you have linked to above accepts a sketch drawn in the Canvas and uses the (previously-created) model with deeplearn.js to generate the result.

The full process used by the Tensorflow port of pix2pix is described here.
https://github.com/affinelayer/pix2pix-tensorflow#getting-started


How to choose the first action in a Monte Carlo Tree Search? I'm working on reimplementing the MuZero paper. In the description of the MCTS (page 12), they indicate that a new node with associated state $s$ is to be initialized with $Q(s,a) = 0$, $N(s,a) = 0$ and $P(s,a) = p_a$. From this, I understand that the root node with state $s_0$ will have edges with zero visits each, zero value and policy evaluated on $s_0$ by the prediction network.
So far so good. Then they explain how actions are selected, according to the equation (also on page 12):

But for the very first action (from the root node) this will give a vector of zeros as argument to the argmax: $Q(s_0,a) = 0$ and $\sum_bN(s_0,b)=0$, so even though $P(s_0,a)$ is not zero, it will be multiplied by a zero weight.
Surely there is a mistake somewhere? Or is it that the very first action is uniformly random?

Yeah, it seems that you're right and based on the description of the paper it would indeed behave uniformly random at the very first iteration (or maybe just always deterministically pick whichever action happens to be the first one in the list). I can't find anything that would suggest otherwise in the paper, and also the pseudocode they put on arXiv suggests the same behaviour.
I can't really think of any reason why this would be better than playing according to $P(s, a)$ in the very first iteration though. Maybe it creates a tiny little bit more variety in the tree search results you get? I suppose all the stochasticity that is normally inherent to a vanilla MCTS is no longer present in MuZero (or AlphaZero), because they always run for exactly the same number of iterations, and don't have any sort of random rollouts anymore; this would at least introduce a tiny bit of variation.
On the other hand, I also can't imagine this choice really making a meaningful difference in practice, except maybe if you go down to extremely low iteration counts (like if you run fewer iterations than there are legal actions in the root node).


Can we use genetic algorithms to evolve datasets? Genetic algorithms are used to solve many optimization tasks.
If I have a dataset, can I evolve it with a genetic algorithm to create an evolved version of the same dataset?
We could consider each feature of the initial dataset as a chromosome (or individual), which is then combined with other chromosomes (features) to find more features. Is this possible? Has this been done?
I will like to edit the details with an example so that it is easier to understand.
Example: In practice cyber-security attacks evolve over time since it finds a new way to breach a system. The main draw-back of intrusion detection model is that it needs to be trained every time attack evolves. So I was hoping if genetic algorithm can be used on the present benchmarked datasets (like NSL-KDD) to come up with a futuristic type dataset maybe after X-number of generations.
And check if a model is able to classify that generated dataset as well.

This question raises a lot more questions. It seems like a solution looking for a problem, instead of the other way round.

How do you measure the fitness of a feature?
What would one of the "evolved datasets" mean? What does it represent?
What would your overal purpose be? If you just wish to generate simulated datasets, there are easier ways to do this, with more control over the various aspects of the resulting datasets.

If you want to compute a new set of features to "better" describe a given dataset, there are many approaches to this, such as PCA, ISOMAP, self-organizing maps, ... If this is the kind of thing you're thinking about, I would recommend starting there.


How are small scale features represented in an Inverse Graphics Network (autoencoder)? This post refers to Fig. 1 of a paper by Microsoft on their Deep Convolutional Inverse Graphics Network:

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/11/kwkt_nips2015.pdf

Having read the paper, I understand in general terms how the network functions. However, one detail has been bothering me: how does the network decoder (or "Renderer") generate small scale features in the correct location as defined by the graphics code? For example, when training the dataset on faces, one might train a single parameter in the graphics code to control the (x,y) location of a small freckle. Since this feature is small, it will be "rendered" by the last convolutional layer where the associated kernels are small. What I don't understand is how the information of the location of the freckle (in the graphics code) propagates through to the last layer, when there are many larger-scale unpooling + convolutional layers in-between.

Thanks for the help!

Simply said, there is no specific "meaning" to the features generated. They are simply features that are fitted through math and calculus, and nobody knows what they represent exactly, and will never knows. However we can run PCA (Principal Component Analysis) to see which feature is the most "important" of all, aka which feature affects the most in the output image. Then, you can try adjusting the value to manually see and guess what teh value do, but you will never know what exactly it does as it is an arbitrary feature, not specifically set by the network. One value may mean multiple things, or just things we human don't understand. See this amazing video about this for details:

https://youtu.be/4VAkrUNLKSo

This video explains what PCA does and also an example of the features generated by the network.

For small scales features, they may simply be ignored as they don't contribute much to the loss or accuracy, or maybe they are represented by a big dot or something else until the last few layers. With just 80 features one cannot fully represent a face with such details, and with the resolution the networks like these are trained on, small features like these probably won't be shown in the image.


Are there OpenAI Gym continuing environments (other than inverted pendulum) and baselines? I would like to use OpenAI Gym to solve a continuing environment, that is, a problem with a single, never-ending episode (please note I don't mean a continuous environment with continuous state and actions).

The only continuing environment I found in their repository was the classic inverted pendulum problem, and I found no baseline methods (algorithms) that don't require episodic environments.

So I have two questions:

are there any continuing environments other than the inverted pendulum one?
is there an OpenAI Gym baseline method that I can use to solve the inverted pendulum problem as well as other continuing environments?

I'm not sure why you need a continuing environment, but actually you can make most (if not all) OpenAI Gym environments continuing. When you perform a step, you receive information about the next state, the reward, a termination signal and a dictionary with additional information. Simply ignore the termination signal if you want the episodes to continue indefinitely. In some cases you will need to modify a variable of the environments called $\texttt {_max_episode_steps}$, which may force the simulation to stop or to restart.

About your second question, check a resource called Spinning Up, from OpenAI too. They explain several methods and their implementation.


How to set the target for the actor in A2C? I did a simple Actor-Critic implementation in Keras using 2 networks where the critic learns the Q-Values of every action, and the actor predicts probabilities for choosing each action. In training, the target probabilities for the actor was a one-hot vector with 1.0 in the maximum Q-Value prediction position and 0.0 in all the rest, and simply used fit method on the actor model with mean squared error loss function.

However, I'm not sure what to set as the target when switching to A2C. In all the guides I saw it's mentioned that the critic now learns one value per state, not one value per action in the action space.

This change makes it unclear on how to set the target vector for the actor. The guides/SE questions I went over did not explain this point and simply said that we can calculate the advantage value using the value function (here, here and here) for the current and next state, which is fine, except we can only do that for the specific action taken and not for every action in the action-space because we don't the value for every next state for every action.

In other words, we only know A(s,a) for our memorized a, and we know nothing about the advantage of other actions.

One of my guesses was that you still calculate the Q-Values, because after all, the value function is defined by the Q-Values. The value function is the sum over every action a of Q(s,a)*p(a). So does the critic need to learn the Q-Values and sum their multiplications with the probabilities generated by the policy network (actor), and calculate the advantages of every action?

It's even more confusing because in one of the guides they said that the critic actually learns the advantage values, and not the value function (like all the other guides said), which is strange because you need to use the critic to predict the value function of the state and the next state. Also, the advantage function is per-action and in the implementations I see the critic has one output neuron.

I think that what's being done in the examples I saw was to train the actor to fit a one-hot vector for the selected action (not the best action by the critic), but modify the loss-function value using the advantage value (possibly to influence the gradient). Is that the case?

In short, my last sentence was the correct answer. The "target" is a one-hot with the selected action, but there's a trick.

A2C Loss Function

A very crucial part of A2C implementation that I missed is the custom loss function that takes into account the advantage. The loss function multiplies the advantage with the negative log of the current probability to select the action that was selected.

The trick is that if the advantage is negative, the loss function will switch sign, so the gradients will be applied to the opposite direction.

In one dimension it's easier to understand. Let's say my target prediction is 1 and my actual prediction is 0.6. A simple loss would be defined as target - prediction, or in this case 0.4 and future predictions will be closer to one. If my prediction was 1.4, then the loss would be -0.4. A negative loss would mean predicting a lower result in the future, and a positive result would mean predicting a higher result in the future.

If the sign of the loss function is switched, the prediction will actually move away from 1.

The same thing happens when you multiply the advantage in the loss function. A negative advantage would mean that this action is worse than the value of the state so we need to avoid it, and a positive advantage means that the action is encouraged.

In Keras (Tensorflow 2.0):

Here's the custom loss function:

def custom_actor_loss(y_true, y_prediction, advantage):
    prediction = K.clip(y_prediction, 1e-8, 1 - 1e-8)
    log_probabilities = y_true * K.log(prediction)

    return K.sum(-log_probabilities*advantage)

The values are clipped because log of 0 is undefined.

And the rest of the network building:

input_layer = Input(shape=self._state_size, name='state_in')
advantage = Input(shape=[1], name='advantage')
target_prediction = Input(shape=self._actions_num, name='target')

inner_layer = Dense(units=layer_size, activation='relu')(input_layer)
actor_out = Dense(units=self._actions_num, activation='softmax', name='actor_out')(inner_layer)

self._actor = Model([input_layer, target_prediction, advantage], actor_out, name='actor')
self._actor.add_loss(custom_actor_loss(actor_out, target_prediction, advantage))
self._actor.compile(optimizer=Adam(learning_rate=actor_learning_rate))

And in the training loop (where future_rewards_prediction and critic_prediction are the outputs from the critic for the current and next state (except for the terminal state where the future_rewards_prediction is set to 0):

# Train actor
target_probabilities = np.zeros([1, self._actions_num])
target_probabilities[0][memory[step_idx].action] = 1.0
advantage = memory[step_idx].reward + future_rewards_prediction * self._future_discount - critic_prediction
self._actor.fit((memory[step_idx].state, target_probabilities, advantage), verbose=0)

*Notice how I don't really specify a y in my fit call. This is because of an issue I had when trying to implement a custom loss function in Keras which was solved by this answer.


Is this a correct visual representation of a recurrent neural network (RNN)? This is a picture of a recurrent neural network (RNN) found on a udemy course (Deep Learning A-Z). The axis at the bottom is "time".

In a time series problem, each yellow row from left to right would represent a sequence of a feature. In this picture, then, there are 6 sequences from 6 different features that are being fed to the network.

I am wondering if the arrows in this picture are completely accurate in an RNN. Shouldn't every yellow node also connect to every other blue node along its depth dimension? By depth dimension here I mean the third dimensional axis of the input tensor.

For example, the yellow node at the bottom left of this picture, which is closest to the viewer, should have an arrow pointing to all the blue nodes in the array of blue nodes that is at the very left, and not just to the blue node directly above it.

I will answer this question, leaving it open to challenge by anyone more knowledgeable.

The equations to update each layer of an RNN are

$y_t = \sigma(W_x x_t + b_x )$

and

$h_t = \sigma(W_hx_t + b_h)$

Where $h_t$ is the hidden layer (in blue in the picture), and $y_t$ is the output layer (red in picture). This equations says that every single component in the hidden layer vector, i.e. every unit in the hidden layer, is a function of the linear combination of the $x_t$ vector, which is the first yellow row along the depth axis. In other words, all the first yellow input nodes on the bottom left in the picture.

Thus, technically this picture is not correct, all the yellow nodes should have points to all the blue nodes. Also, by similar reasoning, all the blue nodes in the subsequent steps of the hidden layer should be connected to all the blue nodes of the previous layer.

Of course, that would make for a much uglier/harder picture to make, so I don't blame the authors, although this has given me a few hours of confused research, which I guess still meets their educational goal.


How is transfer learning used to mitigate catastrophic forgetting in neural networks? How can transfer learning be used to mitigate catastrophic forgetting. Could someone elaborate on this?

Transfer learning is a field where you apply knowledge from a source onto a target. This is a vague notion and there is an abundance of literature pertaining to it. Given your question I will work under the assumption that you are referring weight/architecture sharing between model (in other words training a model on one dataset and using it as a featurizer for another dataset)

Now any learning system without lossless memory will have remnants of catastrophic forgetting. So let's think about how we would implement this transfer and what effects can be derived from this.

One implementation involves transferring a component and only training additional layers.
Another is retraining the entire system but at a lower learning rate?

In setting 1, we can make the claim catastrophic forgetting is minimized by the fact that there is an unbiased featurizer that cant forget based on a sampling regime, though this does not mean additional layers which are still being trained can still faulter in this error mode.

In setting 2, we can make the claim catasrophic forgetting can be reduced compared to a normal end-to-end no-transfer training because the unbiased featurizers difference can be analytically bounded by its initial transferred featurization (complexity class is based on both the function and the number of steps -- so longer you train, the more likely it can forget)

These reasons are talking about mitigating and not erasing the concept of catastrophic forgetting, that is because as I mentioned above any learning system without lossless memory will have remnants of catastrophic forgetting, so making the generalized claim about transfer learning may not always fit the bill.


How does training for DQN work if messing up in the environment in costly? Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works.
Is it that we are exploring the environment at the same time that we are training the Q network? If so, is there not a way to train the Q network before actually going out into the real world? And then, aren't there millions of possible states in the real world? So, how does RL or I guess the neural network generalize so that it can function during rush hour, empty roads, etc.

In general, you need to actively explore the environment to gather data to train your Q network. However, especially in your self-driving car example, you might be looking for Batch RL. In Batch RL you start with a given, fixed dataset of transitions (state, action, reward, next state) and you learn a policy (or Q function) based on the dataset without exploring. The Batch-Constrained Q-learning algorithm (BCQ) is a good example of this.
With regards to acquiring the data necessary for Batch RL, that's another story. I suppose one could collect data through sensors in a car, and label datapoints with rewards according to some reward function.


Periodic Pattern in Validation Loss Curve I am currently trying to solve a regression problem using neural networks. I want to detect movement patterns in images over time (video) and output a continuous value.
During the training process I noticed a strange behaviour for the validation loss curve and I was wondering if anyone has noticed this kind of periodic pattern on some of their own work. What might cause this?

The model looks like the following:

- TimeDistributed(Conv2D(32, (3,3)))
- TimeDistributed(Conv2D(16, (3,3)))
- TimeDistributed(Flatten())
- GRU(64, stateful=True)
- Dropout(0.5)
- Dense(64, activation='relu')
- Dense(1)

I trained the model using the mean squared error as the loss function, a batch size of 1 and the AdamOptimizer with an initial learning rate of 10^(-6). Obviously, the loss curve for the training data is not very good, but I am currently just wondering about the pattern of the val_loss. The plots below represent the loss of 65 epochs.

Thanks!

Edit:
The way I try to solve my task relies on a sliding window approach where I try to predict a continuous value for the next second based on the last 20 seconds (400 frames) of the time-series input data. But I don't think this information is needed to solve my initial question since the periodic patterns appear over several epochs (one "peak" for about every 15 epochs) which is strange. Although the stateful-version of the GRU is used (btw: using TensorFlow and Keras), the internal state of the GRU is reset after every epoch to maintain a clean start. The stateful keyword is used to indicate a dependency between batches.

your model is giving a high loss for start of the video frame , later its loss is getting decreased as it runs through several frames of the "same" video, but again after a new video occurs its loss again peaks to certain point.


Classifying non-labeled data with high dimensionality Disclaimer: I am a novice in the world of machine learning, so please excuse my ignorance.

My dataset consists of things like age, days since last visit, etc. This information is medical related. None of which is geometrical, just data pertaining to particular clients.

The goal is to classify my dataset into three labels. The dataset is not labeled, meaning I'm dealing with an unsupervised learning problem. My dataset consists of ~20,000 records, but this will linearly increase overtime. The data is nearly all floats, with some being strings that can easily be converted into a float. Using this cheat sheet for selecting a solution from the scikit site, a KMeans Cluster seems like potential solution, but I've been reading that having high dimensionality can render the KMeans Cluster unhelpful. I'm not married to a particular implementation either. I've currently got a KMeans Cluster implementation using TensorFlow in Python, but am open for alternatives.

My question is: what would be some solutions for me to further explore that might be more optimal for my particular situation?

I would recommend to have a look at Finding Groups in Data, which is a very readable introduction to clustering methods. It gives a good overview over a number of different algorithms, both agglomerative and hierarchical. As far as I remember, source code for the various algorithms is available on the web somewhere.

I am sure you will find a fitting algorithm for your problem in there.


What's the difference between poker with public cards and without them? Example: texas holdem poker vs texas holdem poker with the same rounds, just with no public cards dealt.

Would algorithms like CFR approximate Nash-equilibrium more easily?
Could AI that does not look at public cards achieve similar performance in normal texas holdem as AI that looks at public state tree?

It depends a little on what you mean by "the same rounds, just with no public cards dealt."

If you mean that each player will just be dealt 2 cards, and no public cards exist, then really we're playing a sort of "high card" game. The best hand is just a pair of aces, CFR will solve this quickly, because the number of possible game states is extremely small compared to a full poker game (especially if we exploit the symmetry of suits, since flushes aren't possible).

If you mean that each player will be dealt 5 cards, with several rounds of betting as before, CFR will probably do less well. The state space will be larger, since there are more cards in play (10 instead of 9). Betting may become more complex, and more complex betting expands the state space enormously.

If you mean that the cards are dealt as before, but the program simply will not look at the cards, then you've kept the state space of the game the same size, but radically reduced the number of information sets. Playing against an opponent who can look at the cards on the table, your program would be at an enormous disadvantage. For instance, imagine the cards on the table are "3 3 8 8 5", and that you have a pair of 2's in your hand. You would want to play very differently from if the cards on the table were "2 2 10 4 7", but an AI without access to the table cards would have to act the same in both situations.


Is Russia contributing to open research concerning friendly AI? I did a little bit of research on the most famous organizations and institutions that promote friendly AI in an open and collaborative way, like OpenAI, MIRI and others. From what I have gathered, most of them have strong ties to the US with some influences from Central Europe and South East Asia. But I didn't find a single instance where Russia participates in any of those groups.

Does Russia contribute to open research concerning friendly AI? Are there any open AI projects with strong ties to Russia?

One instance of interest to you could be the extremely popular OpenCV library.
https://en.wikipedia.org/wiki/OpenCV

Initiated by Intel in 1999. A majority of the contributors were optimization experts from Russia. It is BSD licensed. Development is now done by core team in Russia with corporate sponsors.

The about page of OpenCV currently states

  Along with well-established companies like Google, Yahoo, Microsoft, Intel, IBM, Sony, Honda, Toyota that employ the library, there are many startups such as Applied Minds, VideoSurf, and Zeitera, that make extensive use of OpenCV. OpenCV’s deployed uses span the range from stitching streetview images together, detecting intrusions in surveillance video in Israel, monitoring mine equipment in China, helping robots navigate and pick up objects at Willow Garage, detection of swimming pool drowning accidents in Europe, running interactive art in Spain and New York, checking runways for debris in Turkey, inspecting labels on products in factories around the world on to rapid face detection in Japan.

So, OpenCV has had machine learning, and now it has deep learning as well among its modules in its latest release, so I guess it qualifies for the friendly AI label.

Furthermore, there is a lot of technical research generated in Russia that is published in Russian. I've heard of interesting examples of very advanced aerospace research, collecting dust in Russian journals, that has 'inspired' research in the US. I am not aware of the original researchers were properly credited for their work, or indeed the extent of the veracity of these stories. Nevertheless, it would certainly be to everyone's benefit if language barriers did not impede collaboration between developers and researchers in Russia and the rest of the world.

I feel that the only real hurdle here is that someone has to go and create a portal/framework/institution that provides a platform for developers in Russia to share their work in AI and ease the language and any other non-technical issues that might be impeding this.


Why isn't the loss of my neural network reduced after 2500 iterations? I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly.
The loss function which I am using is
$L(\hat{y},y) = -ylog\hat{y}-(1-y)log(1-\hat{y})$
Can you please point out where I am going wrong the link to the notebook is
https://www.kaggle.com/sidcodegladiator/catnoncat-nn?

You may try to adjust the learning rate first. As the learning rate has a great effect on changing the weights and the bias value.
See if the results has changed after adjusting the learning rate.


Single-Shot Learning for Object Re-Identification I am looking for a way to re-identify/classify/recognize x real life objects (x < 50) with a camera. Each object should be presented to the AI only once for learning and there's always only one of these objects in the query image. New objects should be addable to the list of "known" objects. The objects are not necessarily part of ImageNet nor do I have a training dataset with various instances of these objects.
Example:

In the beginning I have no "known" objects. Now I present a
smartphone, a teddy bear and a pair of scissors to the system. It
should learn to re-identify these three objects if presented in the
future. The objects will be the exact same objects, i.e. not a different phone, but definitely in a different viewing angle, lighting etc.

My understanding is that I would have to place each object in an embedding space and do a simple nearest neighbor lookup in that space for the queries. Maybe just use a trained ResNet, cut off the classification and simply use the output vector for each object? Not sure what the best way would be.
Any advice or hint to the right direction would be highly appreciated.

I have put my initial idea to a test and used a small pretrained CNN (MobileNet) to compute features for reference images and stored the feature vectors in a "database". Query images go through the exact same network and the resulting feature vector is used for nearest neighbor retrieval in the DB.
from glob import glob

import torch
from PIL import Image
from numpy.linalg import norm
from torchvision import transforms
from torchvision.models import mobilenet_v2

model = mobilenet_v2(pretrained=True)
model.eval()

preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

# Generate DB
db = {}
dp_paths = glob('db/*.jpg')
for path in dp_paths:
    image = preprocess(Image.open(path)).unsqueeze(0)
    with torch.no_grad():
        output = model(image)
    db[output] = path

# Query
image = preprocess(Image.open('queries/box.jpg')).unsqueeze(0)
with torch.no_grad():
    query = model(image)

# Nearest Neighbor (poor man's version)
min_distance = float('inf')
candidate = None
for k, v in db.items():
    distance = norm(k.numpy() - query.numpy())
    if distance < min_distance:
        min_distance = distance
        candidate = v

print(candidate, min_distance)

At least with my 5 test reference images and several query images it worked without a single failed "classification". However, I am not sure if it will stand up to a larger test...


Why are the weights of the previous layers updated only considering the old values of the weights of the later layer, not the updated values? Why are the weights of a neural net updated only considering the old values of the later layer, not the already updated values?
I use this example to explain my problem. When applying the backpropagation chain rule, the weights  of the previous layer ($w_1, w_2, w_3, w_4$) are updated making use of the chain rule:
$$\frac{\partial E_{total}}{\partial w_1} = \frac{\partial E_{total}}{\partial out_{h1}} * \frac{\partial out_{h1}}{\partial net_{h1}}*\frac{\partial net_{h1}}{\partial w_1}$$
He then says:
$$\frac{\partial net_{o1}}{\partial out_{h1}}=w_5$$
Although he has already calculated the updated value for $w_5$, he uses the old value of $w_5$ to update $w_1$? Because the updated value of $w_1$ will have an impact on the outcome together with the updated value of $w_5$?

The basic idea of gradient descent is:

Calculate the gradient of some score with respect to parameters that you can control

Take a step in the direction of that gradient that improves the score (subract a multiple of gradient - for gradient descent - if you want to minimise some cost function)

The backpropagation using chain rule in neural network layers is part of the first part, calculating the gradient.
If you interleaved these steps during a single calculation, by taking an update step before propagating the gradient over all layers, you would not propagate the true gradient, but some interim value. There is no guarantee this value would reflect the true gradient of the affected layer when compared to training data and loss functions.
There are some manipulations of the gradient that are acceptable and improve convergence. For instance momentum, and various forms of weighting based on previous gradient calculations. However, I am not aware of any successful attempt to perform updates with partially-calculated gradient then attempt to continue the interrupted gradient calculations over the mixed updated and not-yet-updated parameters.
It is also possible to calculate gradients and then only update some of the possible parameters. In some cases this is useful, for instance it is a good choice when performing transfer learning, by only updating a few layers close to the output. This constrains the network to keep a lot of its trained parameters as-is, which reduces the chances of over-fitting to the smaller dataset that the network is learning from when transferring.


Can we use Q-learning update for policy evaluation (not control)? For policy evaluation purposes, can we use the Q-learning algorithm even though, technically, it is meant for control?
Maybe like this:

Have the policy to be evaluated as the behaviour policy.
Update the Q value conventionally (i.e. updating $Q(s,a)$ using the action $a'$ giving highest $Q(s',a')$ value)
The final $Q(s,a)$ values will reflect the values for the policy being evaluated.

Am I missing something here, given that I have not seen Q-learning being used anywhere for evaluation purposes?

For off-policy learning you must have two policies - a behaviour policy and a target policy. If the two policies are the same, then you end up with SARSA, not Q learning.
You cannot use Q learning directly for evaluating a fixed target policy, because it directly learns optimal value function as the target policy, regardless of the behaviour policy. Instead you must use another variant of off-policy learning that can evaluate an arbitrary target policy.
Your suggested algorithm is:

Have the policy to be evaluated as the behaviour policy.
Update the Q value conventionally (i.e. updating $Q(s,a)$ using the action $a'$ giving highest $Q(s',a')$ value)
The final $Q(s,a)$ values will reflect the values for the policy being evaluated.

This will not work for evaluating the behaviour policy. If the behaviour policy was stochastic and covered all possible state/action choices, then it will still be Q learning and converge on the optimal value function - maybe very slowly if the behaviour policy did not get to important states very often.
The "trick" to off-policy is that the environment interaction part uses the behaviour policy to collect data, and the update step uses the target policy to calculate estimated returns. In general for off-policy updates, there can be corrections required to re-weight the estimated returns. However, one nice thing about single-step TD methods is that there are no such additional corrections needed.
So this gives a way to do off-policy TD learning, using an approach called Expected SARSA. To use Expected SARSA, you will need to know the distribution of action choices i.e. know $\pi(a|s)$ for the target policy.
This is the variant of your description that will work to evaluate your target policy $\pi(a|s)$:

Have any stochastic policy that "covers" the target policy as the behaviour policy.
Update the Q value using Expected SARSA $Q(s,a) = Q(s,a) + \alpha(r + \gamma [\sum_{a'} \pi(a'|s')Q(s',a')]  - Q(s,a))$
The final $Q(s,a)$ values will reflect the values for the policy being evaluated.

Worth noting that Expected SARSA with a target policy of $\pi(s) = \text{argmax}_a Q(s,a)$ is exactly Q learning. Expected SARSA is a strict generalisation of Q learning that allows for learning the value function of any target policy. You may not see it used as much as Q learning, because the goal of learning an optimal value function is more common in practice.


How can I train a RL agent to play board games successfully without human play? How would you go about training an RL Tic Tac Toe (well, any board game, really) application to learn how to play successfully (win the game), without a human having to play against the RL?

Obviously, it would take a lot longer to train the AI if I have to sit and play "real" games against it. Is there a way for me to automate the training?

I guess creating "a human player" to train the AI who just selects random positions on the board won't help the AI to learn properly, as it won't be up against something that's not using a strategy to try to beat it.

The approach will vary depending on some features of the game:

How many players (two for tic tac toe and many classic games).
Whether it is a "perfect information" game (yes for chess and tic tac toe), or whether there is significant hidden information (true for most forms of poker)
Whether the game is zero sum. Any game with simple win/loss/draw results can be considered zero sum.
Branching complexity of the game. Tic tac toe has relatively low branching factor, at worst 9, and reduces by one each turn. The game of Go has branching factoraround 250.
Action complexity of the game. A card game like Magic the Gathering has huge complexity even if the number of possible actions at each turn is quite small.

The above traits and others make a large impact on the details of which algorithms and approaches will work best (or work at all). As that is very broad, I will not go into that, and I would suggest you ask separate questions about specific games if you take this further and try to implement some self-playing learning agents.

It is possible to outline a general approach that would work for many games:

1. Implement Game Rules

You need to have a programmable API for running the game, which allows for code representing a player (called an "agent") to receive current state of the game, to pass in a chosen action, and to return results of that action. The results should include whether any of the players has won or lost, and to update internal state of the game ready for next player.

2. Choose a Learning Agent Approach

There are several kinds of learning algorithms that are suitable for controlling agents and learning through experience. One popular choice would be Reinforcement Learning (RL), but also Genetic Algorithms (GA) can be used for example.

Key here is how different types of agent solve the issues of self-play:

GA does this very naturally. You create a population of agents, and have them play each other, selecting winners to create the next generation. The main issue with GA approaches is how well they scale with complexity - in general not as well as RL.
With RL you can have the current agent play best agent(s) you have created so far - which is most general approach. Or for some games you may be able to have it play both sides at once using the same predictive model - this works because predicting moves by the opposition is a significant part of game play.

2a. How self-play works in practice

Without going into lines of code, what you need for self-play is:

The game API for automation and scoring
One or more agents that can use the API
A system that takes the results of games between agents and feeds them back so that learning can occur:

In a GA, with tournament selection, this could simply be saving a reference - the id of the winner - into a list, until the list has grown large enough to be the parents for the next generation. Other selection methods are also possible, but the general approach is the same - the games are played to help populate the next generation of agents.
In RL, typically each game state is saved to memory along with next resulting state or the result win/draw/lose (converted into a reward value e.g. +1/0/-1). This memory is used to calculate and update estimates of future results from any given state (or in some variants used directly to decide best action to take from any given state). Over time, this link between current and next states provides a way for the agent to learn better early game moves that eventually lead to a win for each player.
An important "trick" in RL is to figure out a way for the model and update rules to reflect opposing goals of players. This is not a consideration in agents that solve optimal control in single agent environments. You either need to make the prediction algorithm predict future reward as seen from perspective of the current agent, or use a global scoring system and re-define one of the agents as trying to minimise the total reward - this latter approach works nicely for two player zero sum games, as both players can then direcly share the same estimator, just using it to different ends.

A lot of repetition in a loop. GA's unit of repetition is usually a generation - one complete assessment of all existing agents (although again there are variants). RL's unit of repetition is often smaller, individual episodes, and the learning part of the loop can be called on every single turn if desired. The basic iteration in all cases is:

Play anything from one move to multiple games, with automated agents taking all roles in the game and storing results.
Use results to update learned parameters for the agents.
Use the updated agents for the next stages of self-play learning.

3. Planning

A purely reactive learning agent can do well in simple games, but typically you also want the agent to look ahead to predict future results more directly. You can combine the outputs from a learning model like RL with a forward-planning approach to get the best of both worlds.

Forward search methods include minimax/negamax and Monte Carlo Tree Search. These can be combined with learning agents in multiple ways - for instance as well as in planning stages using the RL model, they can also be used to help train a RL model (this is how it is used in Alpha Zero).


What is the difference between DQN and AlphaGo Zero? I have already implemented a relatively simple DQN on Pacman.

Now I would like to clearly understand the difference between a DQN and the techniques used by AlphaGo zero/AlphaZero and I couldn't find a place where the features of both approaches are compared.

Also sometimes, when reading through blogs, I believe different terms might in fact be the same mathematical tool which adds to the difficulty of clearly understanding the differences. For example, variations of DQN e.g. Double DQN also uses two networks like alpha zero.

Has someone a good reference regarding this question ? Be it a book or an online ressource.

DQN and AlphaZero do not share much in terms of implementation.
However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, value, policy, then these are interchangeable between DQN and AlphaZero. When it comes to implementation, and what each part of the system is doing, then this is less interchangeable. For instance two networks you have read about in AlphaZero are the policy network and value network. Whilst double DQN alternates between two value networks.
Probably the best resource that summarises both DQN and AlphaZero, and explains how they extend the basic RL framework in different ways is Sutton & Barto's Reinforcement Learning: An Introduction (second edition) - Chapter 16 sections 5 and 6 cover the designs of DQN Atari, AlphaGo and AlphaZero in some depth.
In brief:
DQN Atari

Is model-free
Uses an action value estimator for $Q(s,a)$ values, based on a Convolutional Neural Network (CNN)
Uses experience replay and temporarily frozen target network to stabilise learning process
Uses a variety of tricks to simplify and standardise the state description and reward structure so that the exact same design and hyperparameters work across multiple games, demonstrating that it is a general learner.

AlphaZero

Is model based (although some of the learning is technically model-free, based on samples of play)
Uses a policy network (estimating $\pi(a|s)$) and a state value network (estimating $V(s)$), based on CNNs. In practice for efficiency the NN for these share many layers and parameters, so how many "networks" there are depends how you want to count them.

The earlier AlphaGo version had 4 separate networks, 3 variations of policy network - used during play at different stages of planning - and one value network.

Is designed around self-play
Uses Monte Carlo Tree Search (MCTS) as part of estimating returns - MCTS is a planning algorithm critical to AlphaZero's success, and there is no equivalent component in DQN


Is "emotion" ever used in AI? Is "emotion" ever used in AI?

Psychologists have a lot to say about emotion and it's functional utility for survival - but I've never seen any AI research that uses something resembling "emotion" inside an algorithm. (Yes, there's some work done on trying to classify human emotions, called "emotional intelligence", but that's extrememly different from /using/ emotions within an algorithm) For example, you could imagine that a robot might need fuel and be "very thirsty" - causing it to prioritize different tasks (seeking fuel). Emotions also sometimes don't just focus on objectives/priorities - but categorize how much certain classifications are "projected" into a particular emotions.
For example, maybe a robot that needs fuel might be very "afraid" of going towards cars because it's been hit in the past - while it might be "frustrated" at a container that doesn't open properly.
It seems very natural that these things are helpful for survival - and they are likely "hardcoded" in our genes (since some emotions - like sexual attraction - seem to be mostly unchangeable by "nurture") - so I would think they would have a lot of general utility in AI.

Current Simulation of Emotional Behavior

Emotion is used in AI in very limited ways in leading edge natural language systems. For instance, advanced natural language systems can calculate the probability that a particular segment of speech originates from an angry human. This recognition can be trained using labels from bio-monitors. However, the mental features of a human with soft skills tuned from years of experience with people is not nearly simulated in computers as of this writing.

We will not see computers becoming counselors (as once believed) or directors of movies or courtroom judges or customs officials any time soon. Nonetheless, the processes behind emotion are not entirely undiscovered, and there is definite interest in simulating them in computers. Much of that work is company confidential.

The emergence of emotional sophistication in computers likely to begin in the context of sexuality, primarily because flirtation is powerful and primordial emotional expression will probably be easier to simulate in natural language than higher emotional expressions such as love or chaotic ones like rage. Sexy AI will likely be exploited by what businesses might consider legitimate marketing activity.

It is also going to be exploited by the sex industry. The ethical and moral analysis of sexy AI beyond the scope of the question but will probably gain the attention of public media as it unfolds, and that has already begun on FaceBook, originating from third party attacks using fictitious identities.

The Science of Emotion

Emotion isn't a scientific quantity. From an AI perspective, emotion is a quality an individual might recognize through visual and audio queues, specifically through the natural language and affect of another individual. (Affect is a visual clue about a person's emotional and general mental state.)

An individual can also learn to recognize those clues in her or his self. They can be detected by replaying one's own speech as heard through the ear, by linguistic analysis of thoughts not spoken, or through the detection of muscle tension or vital signs. Those skilled in meditation can detect emotional predecessors closer to their causal centers in the brain and control them more directly before emotions even arise.

In the brain, emotion is not in a single geometric location. We cannot say, "That emotion of compassion comes from this group of neurons in Jerome's brain." We cannot say, "Sheri is angry at this 3D coordinate in her cerebral cortex." Emotions are also not strictly system wide either. An individual can be annoyed without going into rage, leaving most of the brain chemistry and electrical signaling unaffected.

Emotions are not entirely electrical and not entirely chemical. On the electric side, emotional states can occur simultaneously through separate circuit pathway connecting distinct and only distantly related regions of the brain. On the chemical side, there is the synaptic chemistry that is part of the electrical signal pathways. There are also many regional signaling systems using specialized pathways that are neither circulatory (blood) nor primary electrical (neuron) pathways. Serotonin is one of dozens of chemical signaling compounds that operate regionally in this way.

Emotions, being a largely social set of phenomena, should not be characterized as purely Darwinian. Although related to survival, emotional processing and communications impact mate selection and, more generally, social patterns within a community, including altruistic and collaborative activity.

Emotions don't always lead to survival. In some cases, emotional states may lead to death prior to reproduction. One could say that emotional balance and the ability to interact on emotional levels may improve odds of having offspring. Imbalance to the degree of any of hundreds of emotional extremes can lead to childlessness.

Emotional intelligence is different than using emotions within an algorithm, but not extremely so.

Discussion of emotional intelligence is one of many advancements in the concept of intelligence since the formation of one-dimensional conceptions of intelligence. Those nineteenth century conceptions, such as IQ and G-factor are poorly supported by genetic evidence and anthropological theory. Mathematically unproven and naive concepts like general intelligence rest on those one-dimensional concepts.

Emotional intelligence is a form of mental capability related to emotional balance. If a person's cognitive skills are honed with respect to their emotions and the assessment of the emotional states of others, then they have greater emotional intelligence than someone who cannot read the affect and linguistic clues of another and cannot integrate cognitive and emotional skill to balance of their own emotions.

Cybernetic Analysis

The interface between natural emotion and artificial emotion fits within the realm of cybernetics, the conceptual study of the interface between humans and machines. Such interaction is clearly related to both algorithms and topology, two important concepts in AI research and development.

Emotion has an algorithmic context because there is clearly some combination of neurons and chemistry that produce this algorithmic difference between a reactive person and one who has developed emotional intelligence.

    emotion[person] = recognize_emotion[person]
    if emotion[person] = anger
        be_in_responses(angry)

    emotion[person] = recognize_emotion[person]
    if emotion[person] = anger
        be_in_responses(extra_calm)

The former is reactive and the later exhibits emotional intelligence. The acquisition of the later skill may be cognitive and conscious or it may be intuitive and unconscious. In either case, the actual algorithms at a lower level may be entirely different than those shown above, yet the external behavior of the person as marshaled by the brain is essentially one of those shown.

The plural, algorithms, is used rather than the singular, algorithm, because it is unlikely that a single synchronous algorithm is involved. The brain is a massively parallel processor. Emotional processing is likely best expressed in artificial form as hundreds of thousands of algorithms operating in parallel and forming millions of balances within the system — multidimensional and highly parallel stasis.

This is why emotional recognition and emotional responses are not very sophisticated in computer systems as of this writing. The balances have much social nuance. It may be easier to simulate rational thought than emotional thought.

Desire as a Systemic Behavior

Hunger and thirst may sometimes be called feelings, but they are not strictly emotional. The detection of the need for air, energy, nutrients, and water may stimulate emotional states if the needs are unmet and other emotional states if met. A person may become frustrated and irritable when lacking something essential and confronted with another person's less important agenda. A robot may someday do the same. A person may become elated and generous when all such essentials have recently been made available in surplus. A robot may someday do the same. These relationships are expressed in the question this way.

  Emotions also sometimes don't just focus on objectives/priorities — but categorize how much certain classifications are "projected" into a particular emotions.

That statement in the question and its explanation is true in some respects. If a robot that needs fuel but is afraid of passing in front of a moving vehicle because it has been hit in the past can be seen in more than one way.

Probabilistic risk management based on past experiences
Fuzzy logic that produces control behavior
Feeling fear because of past experience

In AI design, these three would be handled in different ways.

Development of a function that ties visual and auditory information to a model of collision and produces a projected likelihood of injury based each of a number of paths to obtaining fuel
Rules that relates to travel risk with learned probabilities for each along with rules that relate to energy depletion risk with learned probabilities for each and a fuzzy logic rules engine
Artificial networks that have no audit trail but simulate reptilian emotional circuitry and simulate base instinct

Maintaining Scientific Perspective

Emotion is not hard coded into the brain circuitry or DNA. The reality is significantly more complex.

The DNA provides parameters to a genetic expression system that leads to protein synthesis that leads to brain structure and function that leads to the ability to learn emotional responses that lead to improved social behavior that may lead to higher probabilities of gene pool survival.

Applying digital system traditions to biological process can be counter productive, like anthropomorphic views of programs. Artificial networks don't actually learn; they converge. Nothing is hard coded into biology because the term code applied to DNA isn't anything like a page of Java or Python code.

It is true that some behavioral predispositions are strong forms of stasis within the course of a species. An organism will normally exhibit a strong desire to acquire resources from the biosphere, such as oxygen, proteins, nutrients, carbs, fats, and water. A robot might replace those with a voltage to use for a charge and lubricants for moving parts. An organism will normally exhibit a string desire to reproduce. A robot might be given a simulation of that recursive process and wish to build another like itself.

These are not hard coded in biology. They form a kind of stasis within a population. Some humans don't want children. Some are hospitalized for anorexia nervosa. Some commit suicide by asphyxiation. The statistical mean produces the behavior of the species, not a fixed behavior identical across individuals within the species.

Nature and Nurture

Nature and nurture are useful umbrella terms for general categories of causality in biology and may have equivalents in future robotic products, but they are broad generalities. There are no nature algorithms or nurture algorithms or algorithms that balance nature and nurture. That is where topology is of paramount conceptual importance.

Topology of Algorithmic Components

There is massive interaction between many systems operating independently in multiple dimensions. The visualization of such interactive structure would look more like the topology of all the web sites in a country than a machine learning block diagram. If somehow coded into one algorithm it is possible that all the silicon from all the sand on earth converted to random access memory (RAM) might be insufficient to hold the code expressing the algorithm. Perhaps not. Perhaps a simplicity underlies the interactive system design of life. Perhaps we'll someday know. Perhaps not.

The elegance in the design of life on earth is that multiple independent processes are tuned by billions of years of trial and error to inter-operate and support complex organic processes with billions of moving parts at a molecular level.

Veins of Interdisciplinary Research

Study of these are important for biology, for bioinformatics, for cognitive science, and for artificial intelligence. Emotional recognition and integration of emotional reaction and control into natural communications is part of this research and development.


How to test if my implementation of back propagation neural Network is correct I am working on an implementation of the back propagation algorithm. What I have implemented so far seems working but I can't be sure that the algorithm is well implemented, here is what I have noticed during training test of my network:

Specification of the implementation:

A data set containing almost 100000 raw containing (3 variable as input, the sinus of the sum of those three variables as expected output).
The network does have 7 layers, all the layers use the sigmoid activation function

When I run the back propagation training process:

The minimum of costs of the error is found at the fourth iteration (The minimum cost of error is 140, is it normal? I was expecting much less than that)
After the fourth iteration the costs of the error start increasing (I don't know if it is normal or not?)

Actually the implementation was correct,

The source of the problem that causes a big error and really slow learning was the architecture of the neural network it self, the ANN has 7 hidden layers which causes the vanishing gradient problem.

When I have decreased the ANN layers to 3 the cost of error was widely reduced besides of that the learning process was faster.

Another common solution is to use RELU or ELU or SELU in the hidden layers of the neural network instead of the sigmoid function


How to create Partially Connected NNs with prespecified connections using Tensorflow? I'd like to implement a partially connected neural network with ~3 to 4 hidden layers (a sparse deep neural network?) where I can specify which node connects to which node from the previous/next layer. So I want the architecture to be highly specified/customized from the get-go and I want the neural network to optimize the weights of the specified connections, while keeping everything else 0 during the forward pass AND the backpropagation (connection does not ever exist).

I am a complete beginner in neural networks. I have been recently working with tensorflow & keras to construct fully connected deep networks. Is there anything in tensorflow (or something else) that I should look into that might allow me to do this? I think with tf, I should be able to specify the computational graph such that only certain connections exist but I really have no idea yet where to start from to do this...

I came across papers/posts on network pruning, but it doesn't seem really relevant to me. I don't want to go back and prune my network to make it less over-parameterized or eliminate insignificant connections.

I want the connections to be specified and the network to be relatively sparse from the initialisation and stay that way during the back-propagation.

So I've done some more research after posting this and it looks like treating things as separate neural nets and connecting them later in subsequent hidden layers is a good idea. If anyone has other ideas that might be helpful, I'd love to hear, though!


How should I encode the input which are 5 cards from a deck of 52 cards? How should I design my input layer for the following classification problem?

Input: 5 cards (from a deck of 52 cards) in a card game;
Output: some classification using a neural network

How should I model the input layer?
Option A: 5 one-hot encodings for the 5 cards, i.e. 5 one-hot vectors of length 52 = 260 input vector. For example
[
[0,0,0,0,0,0,1,...],
[1,0,0,0,0,0,0,...],
[0,0,0,0,0,1,0,...],
[0,0,1,0,0,0,0,...],
[0,0,0,0,1,0,0,...]
]

Option B: 5 hot encoding encompassing all 5 cards in one 52 element vector
[1,0,1,0,1,1,1,...]

What are the disadvantages between A and B?

Depends on how your game is played. Is there any meaning assigned to the order of cards, or are all 5 played simultaneously? If order matters, use 5 one-hot vectors so you can choose how to order them, otherwise use a single 5-hot input vector. I would also add that if temporal order matters, you could also use a recurrent net with a 52-element input and feed the five one-hot vectors one after another.


Which AI algorithm is great for mapping between two XML files My work colleague got a project with a lot of work that is not hard or complicated.  The problem is simple but it is a lot of work.
We have two XML files with a lot of variables in it. Not only is the XML files flatten but you would have classes with classes in them that can reach an absurd amount of depth.

The problem comes in that the one file is a request that is received and the other is a response. The request needs to map its variables to the response variables. A simple tool could be built to solve this solution but the problem comes in that there are rules for certain variables. Some variables in the request have arithmetic involve, sometimes with other variables, or some don't get mapped if other variables are present.

I was thinking about Genetic Programming when I heard about this problem. If all the rules could be defined then it should be able to build a tree that would represent the desired output which is the XML response.

Will it work and if not do you think there is an AI algorithm that can solve this?

The standard tool to work with XML files is XSLT. You may not need AI to solve this problem. But.. you have to learn how to program with XSLT ;)
On Windows you can use MSXML, if you work from C++ - msxsl.exe, C# has internal supoort for XSLT. That is what I know about. There are also non-MS tools.


How does size of the dataset depend on VC dimension of the hypothesis class? This might be a little broad question, but I have been watching Caltech youtube videos on Machine Learning, and in this video prof. is trying to explain how we should interpret the VC dimension in terms of what it means in layman terms, and why do we need it in practice.
The first part I think I understand, please correct me if I am wrong. VC Dimension dictates the number of effective parameters (i.e. degrees of freedom) that model has. In other words, the number of parameters the model needs in order to cover all possible label combinations for the chosen dataset. Now, the second part is not clear to me. The professor is trying to answer the question:

How does knowing the VC dimension of the hypothesis class affect number of samples we need for training?

Again, I apologize if all of this may be trivial, but I am new to the field and wish to learn as much as I can, so I can implement better and more efficient programs in practice.

Given a hypothesis set $H$, the set of all possible mappings from $X\to Y$ where $X$ is our input space and $Y$ are our binary mappings: $\{-1,1\}$, the growth function, $\Pi_H(m)$, is defined as the maximum number of dichotomies generated by $H$ on $m$ points. Here a dichotomy is the set of $m$ points in $X$ that represent a hypothesis. A hypothesis is just a way we classify our points. Therefore with two labels we know,

$$\Pi_H(m)\leq 2^m$$

This is just counts every possible hypothesis. The VC dimension is then the largest $m$ where $\Pi_H(m)=2^m$.

Consider a 2D perceptron, meaning our $X$ is $\mathbb{R}^2$ and our classifying hyperlane is one-dimensional: a line. The VC dimension will be 3. This is because we can shatter (correctly classify) all dichotomies for $m=3$. We can either have all points be the same colour, or one point be a different colour - which is $2^3=8$ dichotomies. You may ask what if the points we are trying to classify are collinear. This does not matter because we are concerned with resolving the dichotomies themselves, not the location of the points. We just need a set of points (wherever they may be located) that exhibits that dichotomy. In other words, we can pick the points such that they maximize the number of dichotomies we can shatter with one classifying hyperplane (a triangle): the VC dimension is a statement of the capacity of our model.

To make this clear, consider $m=4$. We can represent the truth table of the XOR gate as a dichotomy but this is not resolvable by the perceptron, no matter where we choose the location of the points (not linearly separable). Therefore, we can resolve a maximum of 8 dichotomies, so our VC dimension is 3. In general, the VC dimension of perceptrons is $d+1$ where $d$ is the dimension of $X$ and $d-1$ is the dimension of the classifying hyperplane.


What are the pros and cons of using standard deviation or entropy for exploration in PPO? When trying to implement my own PPO (Proximal Policy Optimizer), I came across two different implementations :

Exploration with std

Collect trajectories on $N$ timesteps, by using a policy-centered distribution with progressively trained std variable for exploration
Train policy function on $K$ steps
Train value function on $K$ steps

For example, the OpenAI's implementation.

Exploration with entropy

Collect trajectories on $N$ timesteps, by using policy function directly
Train policy and value function at the same time on $K$ steps, with a common loss for the two models, with additional entropy bonus for exploration purpose.

For example, the PPO algorithm as described in the official paper.

What are the pros/cons of these two algorithms?

Is this specific to PPO, or is this a classic question concerning policy gradients algorithms, in general?

Both implementations may be closer than you think.

In short:

PPO has both parts: there is noisiness in draws during training (with learned standard deviation), helping to explore new promising actions/policies. And there is a term added to the loss function aiming to prevent a collapse of the noisiness, to help ensure exploration continues and we don't get stuck at a bad (local) equilibrium.

In fact, for continuous action, the term for entropy in the loss function you describe in Ex. 2, can make sense only when the actions are stochastic, i.e. when there action choice has some standard deviation the way you describe in Ex. 1.

More detail:

On one hand, PPO (at least for continuous action), trains a central/deterministic value (say the mean policy or close to mean) targeting a most profitable action path. On the other hand, along with it, a standard deviation making the actions a random draw with noise around the deterministic value. This is the part you describe in Example 1. The noise helps explore new paths and to update the policy according to the rewards on these sampled paths.
Entropy itself is a measure of the noisiness of the draws, an thus also an indirect indicator for the trained standard deviation value(s) of the policy.

Now, entropy tends to decay as training progresses, that is, the random draws become progressively less random. This can be good for reward maximization - really the best draws are taken for reward maximization - but it is bad for further improvements of policy: improvement may halt or slow down as exploration of new action paths fades.

This is where entropy encouragement comes in. PPO foresees the inclusion of entropy in the loss function: we reduce the loss by x * entropy, with x the entropy coefficient (e.g. 0.01), incentivizing the learning network to increase the standard deviations (or, to not let them drop too much). This part is what you describe in Example 2.

Further notes:

During exploitation, we'd typically turn off the noise (implicitly assuming action std = 0) and pick deterministic actions: in normal cases this increases the payoffs; we're choosing our best action estimate, rather than at a random value around it.
People are not always precise when referring to the model's entropy vs. the entropy coefficient added to the loss function.
Other RL algorithms with continuous action tend to use noisy drwas with standard deviations/entropy too.


What are the necessary components to make AI agent self-programming-capable? An AI agent is often thought of having "sensors", "a memory", "machine learning processors" and "reaction" components. However, a machine with these does not necessarily become a self-programming AI agent. Beyond the parts mentioned above, is there any other elements or details necessary to make a machine capable of being a self-programming AI agent?

For example, a paper from 2011 declared that solving the optimization problem of maximizing the intelligence is a must-have feature for the self-programming process, as quoted below:

  A system is said to carry out an instance of self-programming when it undergoes learning regarding some element of its "cognitive infrastructure", where the latter is defined as the fuzzy set of "intelligence-critical" features of the system; and the intelligence-criticality of a system feature is defined as its "feature quality," considered from the perspective of solving the optimization problem of maximizing the intelligence of a multi-feature system.

However, this description of "optimization of intelligence" is vague. Can anyone give a clear definition or better summary for the necessary components for self-programming agents?

This question is from the 2014 closed beta, with the asker having a UID of 23.

At the highest level, all it needs is for the various systems already discussed to incorporate code objects. If it can interpret its source code / model architecture from the formatted text objects underpinning them, can 'understand' them in terms of having a useful ML model, and alter the code with its reaction, then it can self-program.

That is, the basic loop behind a recursively improving intelligence is simple. It examines itself, writes a new version, and then that new version examines itself and writes a new version, and so on.

The difficult component comes at lower levels. We don't need to invent a new concept like 'sensor,' what we need to do is build very, very sophisticated sensors that are equal to the task of understanding code well enough to detect and write improvements.


How to prove that gradient descent doesn't necessarily find the global optimum? How can I prove that gradient descent doesn't necessarily find the global optimum?

For example, consider the following function

$$f(x_1, x_2, x_3, x_4) = (x_1 + 10x_2)^2 + 5x_2^3 + (x_2 + 2x_3)^4 + 3x_1x_4^2$$

Assume also that we can't find the optimal value for the learning rate because of time constraints.

You can find by yourself a counterexample that, in general, GD is not guaranteed to find the global optimum!

I first advise you to choose a simpler function (than the one you are showing), with 2-3 optima, where one is the global and the other(s) are local. You don't need neural networks or any other ML concept to show this, but only basic calculus (derivatives) and numerical methods (i.e. gradient descent). Just choose a very simple function with more than one optimum and apply the basic gradient descent algorithm. Then you can see that, if you start gradient descent close to one local optimum (i.e. you choose an initial value for $x$ or $\theta$, depending on your notation for the variable of the function) and then you apply gradient descent (for some iterations), you will end up in that close local optimum, from which you cannot escape, after having applied the gradient descent steps.

See also the question Does gradient descent always converge to an optimum? and For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?


How to train an LSTM with varying length input? I have a dataset where each of the training instances is different in the length and the data is sequential. So, I design an LSTM but I am thinking about how to train the LSTM. In fixed-length data, we just keep all of the input in an array and pass it to the network, but here the case is different. I can not store varying length data in an array and I do not want to use padding to make it fixed length. Now, should I train the LSTM where each training instance are varying in length?

You could sequentially pass in each element of your sequential data and save the hidden and cell states in a separate buffer. In a typical LSTM implementation, you input the entire sequence and the hidden and cell states are propagated internally. In the end, the final hidden and cell states returned as the output. This works if your input is all the same length. Instead, you can handle sequentially giving the next element to the LSTM as well as the hidden and cell state yourself. To keep it efficient you can batch your inputs by the batch dimension (batch_first=True in the pytorch LSTM implementation).
For example, propose you have 5 sequences of length 5, 4, 3, 2, and 1. You initialize your hidden and cell state buffers for each of the sequences and pass the first batch containing the first element of all 5 sequences. You save the output hidden and cell states in the buffers for each sequence. Next, you input the batch of the second element of the 4 sequences with length > 1, and save the states in the respective sequence buffers and so and so forth until you exhaust the sequence of greatest length.


What noteworthy contributions have Chinese AI researchers made in the field of artificial intelligence? In recent years, China has made rapid progress in manufacturing and scientific research as evidenced by their successful teleportation of a single quantum entangled photon to a satellite in orbit. My question is, what major contributions have Chinese AI researchers made in the field of Artificial Intelligence?

Well, there's Baidu with former chief scientist Andrew Ng that's doing research on AI and the Microsoft Asia team has won some ImageNet competitions.


Why is this PyTorch implementation of the actor-critic algorithm inconsistent with the mathematical formulas? This PyTorch implementation of the actor-critic algorithm calculates the losses like so:
actor_loss = -log_prob * discounted_reward
policy_loss = F.smooth_l1_loss(value, torch.tensor([discounted_reward]))

Both are different from the regular formulas which are, in the case of the actor loss (parameterized by $\theta$):
$$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$
and, in the case of the critic loss (parameterized by $w$):
$$r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t}),$$
where $r(s_t,a_t)$ is the immediate reward following taking the action.
For the actor, "the immediate critic evaluation of the transition" was replaced with "the discounted reward". For the critic, the discounted evaluation of the value from the next state $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1})$ was replaced by "the discounted reward". The $L_1$ loss is then calculated, effectively discarding the sign of the (equation) loss.
Questions:

Why did they make these changes?

Why is the sign discarded for the critic loss?

There is no 'regular' formula for calculating policy loss, the regular thing when calculating policy gradient is to multiply gradient with advantage function which can be many things. Look at section 2 of this paper for coverage on basic advantage functions. Also, the expected discounted reward is the same thing as the state-action value function (Q value).
$$Q^\pi(s_t, a_t) = \mathbb{E}_{s_{t+1:\infty}, a_{t+1:\infty}}[\sum_{l=0}^\infty \gamma^lr_{t+l}]$$
So, the variations you posted roughly calculate the same thing.
Regarding the negative sign, in policy gradient methods, we want to maximize our performance function which has the following form:
$$J(\theta) = \sum_\limits{\substack{a}} \pi(a \mid s, \theta)A(s, a)$$
So, the higher our performance the better it is. When people write code, they use minimizers that minimize the objective function, but, in this case, we want to maximize it, so maximizing the objective function is the same thing as minimizing the negative objective function therefore the negative sign.


For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return? For episodic tasks with an absorbing state, why can't $\gamma=1$ and $T= \infty$?
In Sutton and Barto's book, they say that, for episodic tasks with absorbing states that becomes an infinite sequence, then the return  is defined by:
$$G_t=\sum_{k=t+1}^{T}\gamma^{k-t-1}R_k$$
This allows the return to be the same whether the sum is over the first $T$ rewards, where $T$ is the time of termination or over the full infinite sequence, with $T=\infty$ xor $\gamma=1$.
Why can't we have both? I don't see how they can both be set to those parameters. It seems like, if you have an absorbing state, the rewards from terminal onward will just be 0 and not be affected by $\gamma$ or $T$.
Here's the full section of the book on page 57 in the 2nd edition

I think the reasoning behind this also leads to why for policy evaluation where
$$v_\pi(s)=\sum_a\pi(a|s)\sum_{s',r}p(s',r|s,a)[r+\gamma v_\pi(s')]$$
"Has an existence and uniqueness guarantee only if $\gamma < 1$ or termination is guaranteed under $\pi$"(page 74). This part I'm also a bit confused by, but seems related.

$T = \infty$ and $\gamma = 1$ cannot be both true at the same time because the return defined in equation 3.11 is supposed to be a unified definition of the return for both continuing and episodic tasks. In the case of continuing tasks, $T = \infty$ and $\gamma = 1$ cannot be true at the same time, because the return may not be finite in that case (as I think you already understood).
Moreover, note that, in that specific example of the book, they assume that the agent ends up in an absorbing state, so this specific sum is finite, no matter whether $T$ is finite or $\infty$, given that, once you enter the absorbing state, you will always get a reward of $0$. Of course, if you discount those specific rewards, the sum will still be finite. However, in general, if you had a different MDP where the absorbing state is not reachable (i.e. the episode never ends), then the return could not be finite.


Are there communities dealing with costs-vs-accuracy tradeoffs in Machine Learning? My group is working on a ML model that can work with little data (and bad accuracy) as long as there actually is little data available but can easily be extended as soon as said data becomes available (Think of interpolating existing models and then creating an individual predictor). This is due to a business requirement of the relevant application (new plants get installed over time but need to be integrated seamlessly). Transforming from a coarse yet cheap prediction to an accurate but expensive prediction takes labor effort that the company can invest as desired.

Are there research areas that take into account such "evolutionary transformation processes" which enable a tradeoff between costs and accuracy? What are the right keywords to look for?

I am looking for keywords/papers/communities.

Welcome to AI.SE ks.and1.

What you're describing could be related to several areas, but since you are looking only for some keywords, I'll suggest some:

You might be interested in anytime learning, which might better be called 'learning that can stop at any time'. A good example of this approach in machine learning is anytime learning for decision trees, as summarized in Esmeir & Markovitch's 2007 JMLR paper
Apart from this, it sounds like you are just describing the regular process of retraining a model when you get more data. For most algorithms, this just amounts to changing the input to your program so that it points at a larger file, containing more records than last time, and maybe buying a faster GPU or CPU to train the model with if it's taking too long with the extra data.

Hopefully, this can help you get started. If you have some more details about your problem, I might be able to suggest some more specific resources.


Non-differentiable reward function to update a neural network In Reinforcement Learning, when reward function is not differentiable, a policy gradient algorithm is used to update the weights of a network. In the paper Neural Architecture Search with Reinforcement Learning they use accuracy of one neural network as the reward signal then choose a policy gradient algorithm to update weights of another network.

I cannot wrap my head around the concept of accuracy as a non-differentiable reward function. Do we need to find the function and then check if it is mathematically non-differentiable?

I was wondering if I can use another value, for example silhouette score (in a different scenario) as the reward signal?

I cannot wrap my head around the concept of accuracy as a non-differentiable reward function. Do we need to find the function and then check if it is mathematically non-differentiable?

In reinforcement learning (RL), the reward function is often not differentiable with respect to any learnable parameters. In fact it is quite common to not know the function at all, and apply a model-free learning method based purely on sampling many observations of state, action, reward and next state. For reward, you don't need to know the reward function as a function of your parameters, but you do need to know how to caclulate it from each observation. That means that the reward is either provided as part of the environment, or it is clear how to calculate it from the initial state, action or next state. In the case of an agent that creates a neural network then reports the accuracy, you can probably view the NN training process and its results including accuracy on a validation set as a "black box" that reports back an arbitrary reward signal.

  In Reinforcement Learning, when reward function is not differentiable, a policy gradient algorithm is used to update the weights of a network

Policy gradient methods are one broad family of RL methods which are also often model-free. Alternatively, value-based methods are broadly the other choice (e.g. Q-learning) or can be used in combination with policy gradient methods (e.g. Actor-Critic). All of these can be model-free, and often are.

Unless you want to apply a model-based RL method, then you have no need to find the reward function in an explicit form based on any parameters. Even if you did want to use a model-based approach, the reward function does not need to be differentiable.

  I was wondering if I can use another value, for example silhouette score (in a different scenario) as the reward signal?

Yes this would likely be viable, and could be made to work similarly to the paper, provided there is some meaningful connection between the parameters you manipulate and the end result.

RL can be made to work as a generic way of optimising numeric measure influenced due to a vector of parameters. That includes indirect optimisation of neural networks by non-differentiable metrics such as accuracy. There is nothing special about the choice of accuracy here, the metric just needs to be related to the parameters being learned.

There is a catch. Indirect learning using RL can produce very noisy training data, due to e.g. covariance between all the parameters, and it can take many samples to get a clear gradient signal through all the noise and make meaningful updates. It can be inefficient compared to other optimisation methods if they are available - however, the ability to extract a gradient for the parameters even when rules are complex and non-differentiable is a nice feature of RL.


How do I find whether this heuristic is or not admissible and consistent? I was given the following problem to solve.

  Given a circular trail divided by $n> 2$ segments labeled $0 \dots n-1$. In the beginning, an agent is at the start of segment number $0$ (the edge between segments $n-1$ and $0$) and the agent's speed ($M$) is $0$ segments per minute.

  At the start of each minute the agent take one of three actions:

  speed up: the agent's speed increases by $1$ segment per minute.
  slow down:  the agent's speed decreases by $1$ segment per minute.
  keep the same speed: stay at the same speed.

  The action slow down cannot be used if the current speed is $0$ segments per minute.

  The cost of each action is $1$.

  The goal of the agent is to drive around the trail k times ($1 \leq k$)  and then park in the beginning spot (at speed 0 of course). The agent needs to do that in the minimum amount of actions.

  The heuristic given is: If agent is in segment $z$ then: $n-z$ if $z \not = 0$ or $0$ if $z=0$.

I need to find if the given heuristic is admissible (or complete) and consistent.

I think:

Regarding consistency: A heuristic is consistent if its estimate is always $\leq$ estimated distance from any given neighbour vertex to goal plus cost of reaching a goal. So, in the given problem, it is consistent because ($n>2$), so heuristic function is well defined and because of circular trail divided by $n$ segments with a constant price of $1$ for each action, then the given definition of consistency holds because estimated distance from any given neighbour vertex to goal can be looked on as a difference between segments until reaching a goal and it is consistent because again the function is well defined.
Regarding admissibility: An admissible heuristic is one that the cost to reach goal is never more than the lowest possible cost from the current point to reach the goal. I am not sure if the given heuristic is admissible because it does not help much to know the difference between trail size ($n$ = segment size) and current place. But it does not create flaws, so it is probably admissible. I am not sure this is a proof.

Is my idea correct? How could I write it as a proof?

Welcome to AI.SE @hpr16!

Your understanding of when a heuristic is admissible is correct, but your heuristic is inadmissible. An admissible heuristic must always underestimate the cost to move from a given state to a goal state.

Notice that states in the search are not the same as positions on the circle in your problem. A state needs to capture all the information about the current environment the agent is in. In your problem, agents have a speed as well as a position. A state must, therefore, contain both.

To see why your heuristic is inadmissible because the agent can move (n-z) segments in less than n-z steps: it can speed up, and do them in, for example, (n-z)/2 steps, by moving with speed 2.


Can hidden Markov models be used to model any time series data? Can HMMs be used to model any time series data? Or does the data have to be that of a Markov process?

In HTK documentation, I see that the first few lines state that it can model any time series

  HTK is a toolkit for building Hidden Markov Models (HMMs). HMMs can be used to model any time series and the core of HTK is similarly general-purpose.

Yes, you can fit any time series (with or without external variables) using HMM, but there are some constraints:

It should follow the Markov property.
There is some variance that other models are not able to capture (in other words, the system is partially observable).

Adding to point 1, for HMM, it should hold true, but the way Baum Welch algorithm works, indirectly it considers the values of more than the previous state for HMM (order-1). The state $t-1$  depends on $t-2$, which in turn depends on $t-3$. The calculation of parameters (transition, emission, starting probabilities) happens over multiple iterations and it finds parameters in such a way that holds Markov property true.

I think that, when they say 'any', they mean even when you don't have all variables needed to forecast future values.


Why does this model have 12 parameters? I guess the model shown in this image (img_1)

is the same as the one in this image (img_2)

I was trying to build a neural net like that.

This keras code is to do the job.

model = Sequential()
model.add(Dense(3, input_dim=3, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)

However, print(model.summary()) outputs

Model: "sequential_17"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense_31 (Dense)             (None, 3)                 12
_________________________________________________________________
dense_32 (Dense)             (None, 1)                 4
=================================================================

There are 3 ws and 1 b in the hidden layer. Why does this model have 12 parameters?

You have 3 inputs going to 3 nodes in the input layer. Each connection has a weight so you have  3 X 3 =9 weights. Plus each node has a bias weight so that adds 3 more weights for a total of 12. Your output layer has 3 inputs and is a single node so you have 3 weights for the inputs to the node plus a bias weight for a total of 4. So the total weights in your network is 16.


What are the main differences between a perceptron and a naive Bayes classifier? What are the main differences between a perceptron and a naive Bayes classifier?

Naive Bayes is a generative algorithm while Perceptron is a discriminative algorithm. That is the main difference.


Why is the entire area of a join probability distribution considered when it comes to calculating misclassification? In the image given below, I do not understand a few things

1) Why is an entire area colored to signify misclassification? For the given decision boundary, only the points between $x_0$ and the decision boundary signify misclassification right? It's supposed to be only a set of points on the x-axis, not an area.

2) Why is the green area with $x < x_0$ a misclassification? It's classified as $C_1$ and it is supposed to be $C_1$ right?

3) Similarly, why is the blue area a misclassification? Any $x >$ the decision boundary belongs to $C_2$ and is also classified as such...

The misclassifications that could arise if $\hat{x}$ is used as decision boundary are:

a) Classifying a point as $C_2$ when actually it was $C_1$ -- which will only happen when $x > \hat{x}$ as only the points greater than $\hat{x}$ are being classified as $C_2$.

b) Classifying a point as $C_1$ when actually it was $C_2$ -- which will only happen when $x < \hat{x}$ as only the points less than $\hat{x}$ are being classified to be from class $C_1$.

  2) Why is the green area with x < x0 a misclassification?

If we label a point -- the $x$ which is from the interval on the horizontal axis which corresponds to the green area -- which is a probability as being from either class then there is a good chance that it could be misclassified. This is because both the joint distributions' curves are above the axis for that interval on the axis and have some area (probability) for that interval on the horizontal axis.

There is a positive probability that a point that was drawn from the interval (a subset of the sample space, an event), which corresponds to the green area, was drawn from either of the two joint distributions or belongs to either of the two classes $C_1$ and $C_2$. The probability is the area under the curve.

  It's classified as C1 and it is supposed to be C1 right?

The same data point could well be generated from multiple different probability distributions. Here, you have an illustration of two probability distributions for which the author is trying to find the optimal decision boundary that will minimize the misclassification "region".

The point is not supposed to be from $C_1$, even if the density of the joint prob. distribution of $(x, C_1)$ has higher values. It could well be from $C_2$. And, when one classifies the same data point as being from class $C_1$ and make an error. This is why the whole area under the curve $p(x, C_2)$ has been painted green -- which means that there is some probability that a point could be from the distribution $p(x, C_2)$ and if we blindly label all the points to be from $C_1$ will certainly lead to some misclassifications.


When to choose Stochastic Hill Climbing over Steepest Hill Climbing? Stochastic Hill Climbing generally performs worse than Steepest Hill Climbing, but what are the cases in which the former performs better?

The steepest hill climbing algorithms works well for convex optimization. However, real world problems are typically of the non-convex optimization type: there are multiple peaks. In such cases, when this algorithm starts at a random solution, the likelihood of it reaching one of the local peaks, instead of the global peak, is high. Improvements like Simulated Annealing ameliorate this issue by allowing the algorithm to move away from a local peak, and thereby increasing the likelihood that it will find the global peak.

Obviously, for a simple problem with only one peak, the steepest hill climbing is always better. It can also use early stopping if a global peak is found. In comparison, a simulated annealing algorithm would actually jump away from a global peak, return back and jump away again. This would repeat until its cooled down enough or a certain preset number of iterations have completed.

Real world problems deal with noisy and missing data. A stochastic hill climbing approach, while slower, is more robust to these issues, and the optimization routine has a higher likelihood of reaching the global peak in comparison to the steepest hill climbing algorithm.

Epilogue: This is a good question which raises a persistent question when designing a solution or choosing between various algorithms: the performance-computational cost trade-off. As you might have suspected, the answer is always: it depends on your algorithm's priorities. If it is part of some online learning system that is operating on a batch of data, then there is a strong time constraint, but weak performance constraint (next batches of data will correct for erroneous bias introduced by first batch of data). On the other hand, if it is an offline learning task with the entire available data in hand, then performance is the main constraint, and the stochastic approaches are advisable.


What is a working configuration of a neuronal network (number of layers, lerning rate and so on) for a specific dataset? I try to solve some easy functions with a neuronal network (aforge-lib):

This is how I generate the dataset:

const int GesamtAnzahl = 200;
float[,] tempData = new float[GesamtAnzahl, 2];
float minX = float.MaxValue;
float maxX = float.MinValue;

Random rnd = new Random();
var granzen = new List<int>()
{
    rnd.Next(1, GesamtAnzahl-1),
    rnd.Next(1, GesamtAnzahl-1),
    rnd.Next(1, GesamtAnzahl-1),
    rnd.Next(1, GesamtAnzahl-1),
};
granzen.Sort();

for (int i = 0; i < GesamtAnzahl; i++)
{

    var x = i;
    var y = -1;
    if ((i > granzen[0] && i < granzen[1]) ||
        (i > granzen[2] && i < granzen[3]))
    {
        y = 1;
    }
    tempData[i, 0] = x;
    tempData[i, 1] = y;
}

So this is quite easy: The output is 1 if the input is between the 2 lower random generated "borders" or between the 2 higher numbers. Otherwise the output is 1.

The input values are standardices to fit between -1 and 1. So 0 is -1 and 200 is 1.

As a network I used a BackPropagationLearning with a BipolarSigmoidFunction and several configurations like:

Learning Rate: 0,1
Momentum: 0
Sigmoids alpha value: 2
Hidden Layer 1: 4 neurons
Hidden Layer 2: 2 neurons

Learning Rate: 0,1
Momentum: 0
Sigmoids alpha value: 2
Hidden Layer 1: 4 neurons
Hidden Layer 2: 2 neurons
Hidden Layer 3: 2 neurons

Learning Rate: 0,2
Momentum: 0
Sigmoids alpha value: 2
Hidden Layer 1: 4 neurons
Hidden Layer 2: 2 neurons
Hidden Layer 3: 2 neurons

and so on. None of them worked. As described here: https://towardsdatascience.com/beginners-ask-how-many-hidden-layers-neurons-to-use-in-artificial-neural-networks-51466afa0d3e it should be enough to have 2 hidden layers. The first one with 4 neurons and the second one with 2.

The configurations which worked best were:

Learning Rate: 0,01
Momentum: 0
Sigmoids alpha value: 2
Hidden Layer 1: 4 neurons
Hidden Layer 2: 4 neurons
Hidden Layer 3: 4 neurons

Learning Rate: 0,02
Momentum: 0
Sigmoids alpha value: 2
Hidden Layer 1: 4 neurons
Hidden Layer 2: 2 neurons

This solves the problem about 50 % of the times.

As this is a quite simple problem I wonder if I am doing something wrong. I think there has to be a configuration which has better results.

What is the best configuration for this problem and why?

Additionally I tried:

Having more data does not help. I created 5000 a dataset of 5000 points ( GesamtAnzahl = 5000). Then the networks have a even worse sucess rate.
I tried to add an extra constant input (always 1) to the dataset but this also lowered the sucess rate

I ran a lot of randomly created networks to solve this problem, but none of the structures where able to reliably "solve" this problem.

Of course, some of them where able to solve it one time, some of them even twice but there where only one which solved it 3 times:

LearningRate: 0,510141694690167
Momentum: 0,962972165068133;
Layer/Neuron-Count: 2 (14, 9)
SigmundAlphaValue: 2;