Nuances of Turing test requirements My understanding is that there is no singular "The Turing Test Ruleset" and competitions don't all do it the exact same way. Still, I'm wondering about some commonly accepted rules and their nuances. My Googling is not producing any specifics about this.

I think most people agree that the purpose of the humans who talk to the judges is to just act normal. In the so-called "passed Turing test" instances where the humans tried to fool judges into thinking they were AIs, I would say the tests should be thrown out and I've seen critics in the field agree with that.

But this question is more about how the judge should act.

Let's say we are doing a competition where the threshold is for 40% of judges to call an AI a human after 5 minutes of chat.

During the chat, is the judge supposed to try to trick the potential AI into revealing itself, or is the judge supposed to attempt to act as unbiased and natural as possible, as if it is accepted the potential AI is human, and judge based on a completely natural conversation?

For example, asking "What is the value of PI out 10 digits?" or "What is 123456 times 654321?" or "If you saw a bunny and a dime stuck on the road with a truck about to hit them, which one would you pick up?" would be trying to trick AIs into revealing themselves because you are relying on exploiting the fact that the AI might tell you the correct answer or the inhuman answer.

This is as opposed to simply carrying on a natural and normal conversation, with no biases or expectations. If you came upon someone on the street you would not spend 5 minutes trying to hurriedly ask ridiculous interrogation questions in an effort to prove the other person was an AI.

So is the point of a Turing test generally assumed to be an attempt to flush out AIs or an attempt to judge their natural human conversation without interrogation-like prejudice?

I do not know about all competitions, but the Loebner prize is one of the more well known competitions based around the concept of the Turing Test, specifically for chatbots, and judges must ask the same questions of all bots. The questions are only revealed during the competition, it is not possible for bot programmers to prepare for them other than making a generally good chatbot. They judges can then rank bots on how they handle the same set of questions. E.g. from chatbots.org/ai_zone/viewthread/867 here are the 2012 "qualifying" questions that were released following the test:

My name is Bill. What is your name?
How many letters are there in the name Bill?
How many letters are there in my name?
Which is larger, an apple or a watermelon?
How much is 3 + 2?
How much is three plus two?
What is my name?
If John is taller than Mary, who is the shorter?
If it were 3:15 AM now, what time would it be in 60 minutes?
My friend John likes to fish for trout.  What does John like to fish for?
What number comes after seventeen?
What is the name of my friend who fishes for trout?
What whould I use to put a nail into a wall?
What is the 3rd letter in the alphabet?
What time is it now?

According to the WikiPedia description, the style of the questions has changed over time. The above set in my opinion seem to be designed to test very simple grammar, common sense and memory targets.


DQN card game - how to represent the actions? I want to train a DQN card game named witches. It consists of 60 Cards (1-14 of Yellow, Blue, Green, Red Cards + 4 Wizzards). The color of the first layed card has to be respected by the other players (if they have this card in hand). The one who has the card with the highest number gets the played cards. Each collected red card gives you a -1 point.

With respect to this answer I setup the inputs / state of the NN as binary (meaning I have 180 bool values (60 card x is currently on the table, 60 card x is in the ai hand, 60 card x is already been played)

How to design the outputs / actions?

If the ai is the first player of a round it can play any card
If the ai is not first player it has to respect the first played card (or play a wizzard)

This means there is actually a list of available options. I then sort this list and have 60 Output bools which I set to 1 if this option is possible. Among these options the ai should then decide what the correct option is? Is this the correct procedure?

Inconsistent / Varying Action Space
This is what we have to deal here with. As explained in here I think a DQN as well as Policy Gradient Methods is not the correct architecture to choose for solving such multi-agent card games. What architecture would you choose?

General procedure?

Assume I have 4 players, so do I have to get the old state before the ai is the next player and the new state is directly after this round is finished?

my_game = game(["Tim", "Bob", "Lena", "Anja"])
while True:
    #1. Play unti AI has to move:
    my_game.play_round_until_ai()

    #2. Get old state:
    state_old = agent.get_state(my_game)

    #3. Get the action the AI should perform
    action_ = agent.get_action(state_old, my_game)

    #4. perform new Action and get new state
    #reward rates how good the current action was
    #score is the actual score of this game!
    reward, done, score = my_game.finishRound(action_)

    # 5: Calculate new state
    state_new = agent.get_state(my_game)

    #6. train short memory base on the new action and state
    agent.train_short_memory(state_old, action_, reward, state_new, done)

    #7. store the new data into a long term memory
    agent.remember(state_old, action_, reward, state_new, done)

    if done == True:
        # One game is over, train on the memory and plot the result.
        sc = my_game.reset()

My code so far is available here: https://github.com/CesMak/witches_ai

Because your action space is not discrete, It would not be wise to evaluate possible actions. What would probably be better is evaluating next states. This means evaluating the value of the state after the action with your neural network instead of the value of a action.

Why is this better?
You can use the neural net for each possible action , instead of trying to make the neural network figure out which actions are possible. the result would be taking the action of which the next state has the most value.


How can I design and train a neural network to play a card game (similar to Magic: The Gathering)? Introduction
I am currently writing an engine to play a card game, as there is no engine yet for this particular game.
About the game
The game is similar to Magic: The Gathering. There is a commander, which has health and abilities. Players have an energy pool, which they use to put minions and spells on the board. Minions have health, attack values, costs, etc. Cards also have abilities, these are not easily enumerated. Cards are played from the hand, new cards are drawn from a deck. These are all aspects it would be helpful for the neural network to consider.
Idea
I am hoping to be able to introduce a neural network to the game afterwards, and have it learn to play the game. So, I'm writing the engine in such a way that is helpful for an AI player. There are choice points, and at those points, a list of valid options is presented. Random selection would be able to play the game (albeit not well).
I have learned a lot about neural networks (mostly NEAT and HyperNEAT) and even built my own implementation. Neural networks are usually applied to image recognition tasks or to control a simple agent.
Problem/question
I'm not sure if or how I would apply neural networks to make selections with cards, which have a complex synergy. How could I design and train a neural network for this game, such that it can take into account all the variables? Is there a common approach?
I know that Keldon wrote a good AI for RftG, which has a decent amount of complexity, but I am not sure how he managed to build such an AI.
Any advice? Is it feasible? Are there any good examples of this? How were the inputs mapped?

I think you raise a good question, especially WRT to how the NNs inputs & outputs are mapped onto the mechanics of a card game like MtG where the available actions vary greatly with context.

I don't have a really satisfying answer to offer, but I have played Keldon's Race for the Galaxy NN-based AI - agree that it's excellent- and have looked into how it tackled this problem.

The latest code for Keldon's AI is now searchable and browseable on github.

The ai code is in one file. It uses 2 distinct NNs, one for "evaluating hand and active cards" and the other for "predicting role choices".

What you'll notice is that it uses a fair amount on non-NN code to model the game mechanics. Very much a hybrid solution.

The mapping of game state into the evaluation NN is done here. Various relevant features are one-hot-encoded, eg the number of goods that can be sold that turn.

Another excellent case study in mapping a complex game into a NN is the Starcraft II Learning Environment created by Deepmind in collaboration with Blizzard Entertainment. This paper gives an overview of how a game of Starcraft is mapped onto a set of features that a NN can interpret, and how actions can be issued by a NN agent to the game simulation.


Should I model my problem as a semi-MDP? I have a system (like a bank) that people (customers) are entered into the systems by a Poisson process, so the time between the arrival of people (two consecutive customers) will be a random variable. The state of the problem is related to just the system (bank), and the action, made inside the system, can be e.g. offering the customer promotion or not (just based on the state of the system, not the status of customers).

To model the problem through RL, 1) it is possible to discretize time horizon into very short time interval (for example 5 minutes as a stage) such that in each time interval, just a single customer enter to our system. On the other hand, 2) it is possible that stages are defined as the time when a customer enters our system.

My questions are:

Is the second approach an semi-MDP (SMDP)? If I want to solve it with RL, should I use hierarchical RL?
In the first approach, if a customer enters in a time interval, it is easy to update the Q values. However, what should we do, if we are in state $S$ and take action $A$, but no customer enters our system, so we do not receive any reward for the pair of $(S, A)$? There would be no difference if we would take action $A_{1}$, $A_{2}$, and so on. This can happen for several consecutive time intervals. I think it is more challenging when we consider eligibility traces.

To model the problem through RL,

  it is possible to discretize time horizon into very short time interval (for example 5 minutes as a stage) such that in each time interval, just a single customer enter to our system.
  On the other hand, it is possible that stages are defined as the time when a customer enters our system.

  1) Is the second approach an SMDP and if I want to solve it with RL, I should use Hierarchical RL?

The second approach as you described it does not sound like an SMDP to me... it sounds to me like a regular MDP where the discrete time steps in the MDP do not have any meaningful connection anymore to real-world time in the real-world problem being modelled. This is the approach I would recommend taking if long periods of "inactivity" in between customers are irrelevant and/or if you do not expect your actions to have any influence on the duration of those periods of inactivity.

If you do expect your actions to have meaningful influence on the duration of inactivity, I would not recommend this approach. For example, your actions may make a customer angry and reduce their likelihood of quickly returning. Or your actions may indirectly affect the likelihood of other customers visiting due to interaction between your population of customers, they may or may not recommend your bank to their friends based on your actions.

  2) In the first approach, if a customer enters in a time interval, it is easy to update $Q$-values. However, what should we do, if we are in state $S$ and take action $A$ but, no customer enter our system so we do not receive any reward for the pair of $(S, A)$ there would be no difference if we would take action $A_1$, $A_2$ and so on. This can happen for several consecutive time interval. I think it is more challenging when we consider eligibility trace.

I would actually argue that your first approach can be treated as an SMDP, and that it may be a good idea to do so (personally I prefer the Options framework, it's a bit more general than SMDPs, a bit more flexible, and a bit closer to "standard" RL terminology; there are plenty of free versions of that paper available on goole too).

Whenever you encounter a customer, you could select an option (or "macro-action") in the SMDP, which executes the one "real" action for your current customer once, and is subsequently forced to automatically select no-op actions (which do nothing) until the next customer arrives; that is the point in time where your previous option ends, and a new option can be selected.

An RL algorithm that operates on the SMDP / hierarchical level, i.e. one that only learns to optimize the selection of options / macro-actions and completely skips over all the forced no-op actions, would suffice. This is basically the approach described in Concurrent Reinforcement Learning from Customer Interactions.


What is the relation between back-propagation and reinforcement learning? What is the relation between back-propagation and reinforcement learning?

Backpropagation is a subroutine often used when training Artificial Neural Networks with a Gradient Descent learning algorithm. Gradient Descent requires computation of the error gradient, i.e. derivatives, of a cost function with respect to the network parameters. BP allows you to find this gradient a lot faster than using naive methods. Reinforcement Learning refers to inferring "optimal" behavior, i.e. a strategy, of an agent maximizing some goal in an environment. Depending on the specific RL algorithm, BP may be employed to adjust parameters of a function approximator used to represent aspects of the environment and or the agent.


Is it better to make neural network to have hierchical output? i'm quite new to neural network and i recently built neural network for number classification in vehicle license plate. It has 3 layers: 1 input layer for 16*24(382 neurons) number image with 150 dpi , 1 hidden layer(199 neurons) with sigmoid activation function, 1 softmax output layer(10 neurons) for each number 0 to 9.

I'm trying to expand my neural network to also classify letters in license plate. But i'm worried if i just simply add more classes into output, for example add 10 letters into classification so total 20 classes, it would be hard for neural network to separate feature from each class. And also, i think it might cause problem when input is one of number and neural network wrongly classifies as one of letter with biggest probability, even though sum of probabilities of all number output exceeds that.

So i wonder if it is possible to build hierchical neural network in following manner:

There are 3 neural networks: 'Item', 'Number', 'Letter'

'Item' neural network classifies whether input is numbers or letters.
If 'Item' neural network classifies input as numbers(letters), then input goes through 'Number'('Letter') neural network.
Return final output from Number(Letter) neural network.

And learning mechanism for each network is below:

'Item' neural network learns all images of numbers and letters. So there are 2 output.
'Number'('Letter') neural network learns images of only numbers(letter).

Which method should i pick to have better classification? Just simply add 10 more classes or build hierchical neural networks with method above?

Just use one network with a larger Softmax output layer and more hidden units. If you have enough training data, it will work just fine. In fact it could emulate the architecture you propose.


How do I determine whether a truck is inside its lane? I have a bunch of images from different trucks passing the road. Here is an example.

The truck needs to be at a certain distance from the border of the lane. Some of the trucks are way close to the border (that you can see on the shoulder of the road).
I want to find a way to measure the distance between the truck and the border of the lane and, more importantly, to detect whether a truck is inside its lane.
I would like to solve this problem by training a deep learning-based classifier or image processing techniques. Painting the ground is also possible if I can train a classification algorithm with painted images.

One possible approach will be to use an algorithm which detects lines (Ex. Hough lines or any deep neural net trained to detect lanes) and use some threshold range so that we can get the lane and the edges of truck, then after extracting the lines, you can easily find the distance between them.

Then you need to experiment out on few images to get the threshold distance that you are expecting the truck to maintain as the real distance and the distance calculated using images are not same

If you want to classify using deep learning, you may need to preprocess the images and send them. As it will become very difficult to directly learn to classify
based on image, you may need to first detect the lanes, then apply a mask and then send the masked image to your network to make the network to converge.


Is stable learning preferable to jumps in accuracy/loss A stable/smooth learning validation curve often seems to keep improving over more epochs than an unstable learning curve. My intuition is that dropping the learning rate and increasing the patience of a model that produces a stable learning curve could lead to better validation fit.
The counter argument is that jumps in the curve could mean that the model has just learned something significant, but they often jump back down or tail off after that.
Is one better than the other? Is it possible to take aspects of both to improve learning?

There is an approach to machine learning, called Simulated Annealing, which varies the rate: starting from a large rate, it is slowly reduced over time. The general idea is that the initial larger rate will cover a broader range, while the increasingly lower rate then produces a less 'erratic' climb towards a maximum.
If you only use a low rate, you risk getting stuck in a local maximum, while too large a rate will not find the best solution but might end up close to one. Adjusting the rate gives you the best of both.


How do I integrate Tensorflow with EV3DEV? I've got a Lego Mindstorms EV3 with EV3DEV and EV3-Python installed. I wanted to try out artificial intelligence with the robot, and my first project is going to be to get the robot to try and recognize some images (using convolutions) and do an action related to the image it has seen. However, I can't find a way to use Tensorflow (or any AI module for that matter) on an EV3. Does anyone know how to incorporate Tensorflow or any other modules into the EV3? Help would be gladly appreciated.

There is a YouTube video LEGO EV3 Raspberry Pi Tensorflow Sorting Machine by ebswift that should help you although you will need a Raspberry Pi.  From the abstract:

This is a sorting machine based on the EV3 45544 education kit sorting machine.  The colour sorting camera is substituted with a Raspberry Pi with the v2 camera.  The EV3 is controlled over wifi via RPyC and the object recognition work is done via Tensorflow.

A viewer asked: Can you share links about how to train a model on the PC and move the trained model to the Pi?

ebswifft replied:
Glad you like it :).  Training the model on the PC is following the guide for Tensorflow for Poets https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/.  All you do is install Tensorflow on the Raspberry Pi, clone the github repository onto it and copy the model you trained from the codelabs article onto the Raspberry Pi to run the classifier as per step 5 of the codelabs article.  Onboard image capture on the Raspberry Pi is just done using picamera.  I used a tensorflow version that a user compiled for the Raspberry Pi from here: https://github.com/samjabrahams/tensorflow-on-raspberry-pi/issues/92.  I might do up more of a general step-by-step sometime on my site, I'll report back if I can get that up and running.

There is also BrickClassifi3r:

This Lego Mindstorms EV3 robot uses a neural network to recognize a cube, a cylinder or a small cube put on a conveyor belt. See the video how it works. Each object on the conveyor belt is scanned by an IR-sensor every 40ms for about a second. The resulting data are 24 distance values representing one of the three objects. This data is fed into the neural network on the robot to classify the object within 180ms. In a test with 300 objects it reaches 95,6% accuracy. The neural network has been trained before by a machine learning algorithm with TensorFlow on a PC using a set of 375 training examples - 125 examples for each object.

The ev3-myo project uses LEGO Mindstorms EV3, TRACK3R, a Myo armband and TensorFlow for gesture recognition.


What are some references that describe known filters (or kernels) and how we can create new ones? I'm pursuing a master's degree in Artificial Intelligence. My final work is about Convolutional Neural Networks.
I was looking for information about filters (or kernels) of the convolutional layers. I have found this article: Lode's Computer Graphics Tutorial - Image Filtering, but I need more.
Do you know more resources about more filters (that it is known that they work) and how to create new ones?
In other words, I want to know how they work and how can I create new ones.
I've thought to create a C++ program, or with Octave, to test the new kernels.
By the way, my research will be focused on image segmentation to process MRIs.

I'd suggest you better understand edge detectors such as Robert or Sobel operators first to understand better how convolution operation on images extract features by constant value kernels.

Would personally recommend Gonzales and Woods for this,  as it gives a pure mathematical explanation to how and why these features are extracted.

Essentially the convolution kernels used in CNN's are ones with a learned set of values for the kernel.

For a better understanding of learned convolution kernels and, quite frankly, any idea under deep learning would easily recommend Deep Learning by Goodfellow et al


How do we determine whether a heuristic function is better than another? I am trying to solve a maze puzzle using the A* algorithm. I am trying to analyze the algorithm based on different applicable heuristic functions.
Currently, I explored the Manhattan and Euclidean distances. Which other heuristic functions are available? How do we compare them? How do we know whether a heuristic function is better than another?

In the A* algorithm, at each iteration, a node is chosen which minimizes a certain function, called the evaluation function, which, in the case of A*, is defined as

$$f(n)=g(n)+h(n)$$

where $g(n)$ is the length (or cost) of the cheapest path from the start node to the current node $n$ and $h(n)$ is the heuristic function that estimates the cost of the cheapest path from current node $n$ to goal node.

There is potentially more than one path to the goal from a given node $n$. However, one of these paths is the cheapest path. An admissible heuristic function is a heuristic function that does not overestimate the cost to reach the goal node, that is, it estimates a cost to reach a goal that is smaller or equal to the cheapest path from $n$, which is denoted by $h^*(n)$. Therefore, an admissible heuristic $h$ satisfies $h(n) \leq h^*(n), \forall n$. Given that the goal is to find the cheapest path from a start to a goal node, intuitively, an admissible heuristic is an optimistic predictive function.

A* is guaranteed to find the optimal solution (or path) if it uses an admissible heuristic. In section 2.4 of the book Principles of Artificial Intelligence (1982), Nils J. Nilsson provides the proof of this fact.

However, not all admissible heuristics give the same information, so not all admissible heuristics are equally efficient. For instance, a heuristic function that is trivially admissible is $h(n) = 0, \forall n$. However, in this case, the only actual information that is used to choose the next node to expand is only based on $g(n)$, that is, $f(n) = g(n)$. This evaluation function corresponds to the evaluation function of the uniform-cost search algorithm, which is an uninformed-search algorithm (as opposed to A*, which, nonetheless, is considered an informed-search algorithm).

Which admissible heuristic is thus more informed? Consider two versions of A*, each with a different admissible heuristic function

$$
f_1(n) = g_1(n) + h_1(n)
$$

and

$$
f_2(n) = g_1(n) + h_2(n)
$$

where $h_1(n) \leq h^*(n), \forall n$ and $h_2(n) \leq h^*(n), \forall n$. Then A* with the evaluation function $f_1$ is more informed than A* with $f_2$ if, for all non-goal nodes $n$, $h_1(n) > h_2(n)$. See section 2.4.4. of the cited book where an example that attempts to show this is given.

The admissibility of a heuristic depends on the problem. For example, in the case of the Fifteen Puzzle problem, both Manhattan and the Hamming distances are admissible heuristics. However, in other problems, these distances might not induce an admissible heuristic.


How is BERT different from the original transformer architecture? As far as I can tell, BERT is a type of Transformer architecture. What I do not understand is:

How is Bert different from the original transformer architecture?

What tasks are better suited for BERT, and what tasks are better suited for the original architecture?

What is a transformer?
The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-called attention (i.e. a mechanism that determines the importance of words to other words in a sentence or which words are more likely to come together) and the non-use of recurrent connections (or recurrent neural networks) to solve tasks that involve sequences (or sentences), even though RNN-based systems were becoming the standard practice to solve natural language processing (NLP) or understanding (NLU) tasks. Hence the name of the paper "Attention is all you need", i.e. you only need attention and you don't need recurrent connections to solve NLP tasks.
Both the encoder-decoder architecture and the attention mechanism are not novel proposals. In fact, previous neural network architectures to solve many NLP tasks, such as machine translation, had already used these mechanisms (for example, take a look at this paper).  The novelty of the transformer and this cited paper is that it shows that we can simply use attention to solve tasks that involve sequences (such as machine translation) and we do not need recurrent connections, which is an advantage, given that recurrent connections can hinder the parallelization of the training process.
The original transformed architecture is depicted in figure 1 of the cited paper. Both the encoder and decoder are composed of

attention modules
feed-forward (or fully connected) layers
residual (or skip) connections
normalization layers
dropout
label smoothing
embedding layers
positional encoding

The decoder part is also composed of a linear layer followed by a softmax to solve the specific NLP task (for example, predict the next word in a sentence).
What is BERT?
BERT stands for Bidirectional Encoder Representations from
Transformers, so, as the name suggests, it is a way of learning representations of a language that uses a transformer, specifically, the encoder part of the transformer.
What is the difference between the transformer and BERT?

BERT is a language model, i.e. it represents the statistical relationships of the words in a language, i.e. which words are more likely to come after another word and stuff like that. Hence the part Representations in its name, Bidirectional Encoder Representations from
Transformers.
BERT can be trained in an unsupervised way for representation learning, and then we can fine-tune BERT on the so-called downstream tasks in a supervised fashion (i.e. transfer learning). There are pre-trained versions of BERT that can be already fine-tuned (e.g. this one) and used to solve your specific supervised learning task. You can play with this TensorFlow tutorial to use a pre-trained BERT model.
On the other hand, the original transformed was not originally conceived to be a language model, but to solve sequence transduction tasks (i.e. converting one sequence to another, such as machine translation) without recurrent connections (or convolutions) but only attention.

BERT is only an encoder, while the original transformer is composed of an encoder and decoder. Given that BERT uses an encoder that is very similar to the original encoder of the transformer, we can say that BERT is a transformer-based model. So, BERT does not use recurrent connections, but only attention and feed-forward layers. There are other transformed-based neural networks that use only the decoder part of the transformer, for example, the GPT model.

BERT uses different hyper-parameters than the ones used in Attention is all you need to achieve the best performance. For example, it uses 12 and 16 "attention heads" (please, read the transformer paper to know more about these "attention heads") rather than 8 (although in the original transformer paper the authors experimented with a different number of heads).

BERT also uses segment embeddings, while the original transformer only uses word embeddings and positional encodings.

There are probably other small differences that I missed, but, after having read the paper Attention is all you need and quickly read some parts of the BERT paper, these seem to be the main differences.
When to use BERT and the transformer?
Although I never used them, I would say that you want to use BERT whenever you want to solve an NLP task in a supervised fashion, but your labeled training dataset is not big enough to achieve good performance. In that case, you start with a pre-trained BERT model, then fine-tune it with your small labeled dataset. You probably need to add specific layers to BERT to solve your task.


What is a cascaded convolutional neural network? For a project I am doing, I found the paper Face Alignment in Full Pose Range: A 3D Total Solution.

It is using a cascaded convolutional neural network, but I wasn't able to find the original paper explaining what that is.

In layman's terms and intuitively, how does a cascaded CNN work? What does it solve?

The paper you are citing is the paper that introduced the cascaded convolution neural network. In fact, in this paper, the authors say

  To realize 3DDFA, we propose to combine two achievements in recent years, namely, Cascaded Regression and the Convolutional Neural Network (CNN). This combination requires the introduction of a new input feature which fulfills the "cascade manner" and "convolution manner" simultaneously (see Sec. 3.2) and a new cost function which can model the priority of 3DMM parameters (see Sec. 3.4)

where 3DDFA stands for 3D Dense Face Alignment, the framework proposed in this paper for face alignment, in which a dense 3D Morphable Model (3DMM) is fitted to the image via cascaded CNNs (the regressor), where the term dense refers to the number of points of the face that will be modeled. See figure 1 of this paper, which should provide some intuition behind the purpose of this framework.

In section 3 (page 3), they also say

  In this section, we introduce how to combine Cascaded Regression and CNNs to realize 3DDFA. By applying a CNN as the regressor in Eqn. 1, Cascaded CNN can be formulated as:

\begin{align}
\mathbf{p}^{k+1} = \mathbf{p}^{k} + \text{Net}^{k}
(\text{Fea}(\mathbf{I}, \mathbf{p}^k)) \tag{1}\label{1}
\end{align}

where

$k$ is the iteration number
$\mathbf{p}$ is the regression objective
$\text{Net}$ is the CNN structure
$\text{Fea}$ contains the two constructed image features

Pose Adaptive Feature (PAF) (section 3.2.1)
Projected Normalized Coordinate Code (PNCC) (section 3.2.2)

$\mathbf{I}$ is the image

The expression cascaded CNN apparently refers to the fact that equation \ref{1} is used iteratively, so there will be multiple CNNs, one for each iteration $k$. In fact, in the paper, they say

  Unlike existing CNN methods that apply different network structures for different fitting stages, 3DDFA employs a unified network structure across the cascade. In general, at iteration $k$ ($k = 0, 1, \dots, K$), given an initial parameter $\mathbf{p}^k$, we construct PNCC and PAF with $\mathbf{p}^k$ and train a two-stream CNN $\text{Net}^k$ to conduct fitting. The output features from two streams are merged to predict the parameter update $\Delta \mathbf{p}^k$

  $$
\Delta \mathbf{p}^k = \text{Net}^k(\text{PAF}(\mathbf{p}^k, \mathbf{I}), \text{PNCC}(\mathbf{p}^k, \mathbf{I}))
$$

  Afterwards, a better intermediate parameter $\mathbf{p}^{k+1} = \mathbf{p}^k + \Delta \mathbf{p}^k$ becomes the input of the next network $\text{Net}^k$ which has the same structure but different weights with $\text{Net}^k$.

In figure 2 of the paper (page 4), the structure of this two-stream CNN, $\text{Net}^k$, at iteration $k$, is shown.


How to train a policy model incrementally to solve a problem similar to the vehicle routing problem? I have a problem similar to the vehicle routing problem (VRP) that I want to solve with reinforcement learning. In this problem, the agent starts from the point $(x_0, y_0)$, then it needs to travel through $N$ other points, $(x_1, y_1), \dots, (x_n, y_n)$. The goal is to minimize the distance traveled.
Right now, I am modeling a state as a point $(x, y)$. There are 8 possible actions: go east, go north-east, go north, go north-west, go-west, go south-west, go south, go south-east. Each action goes by a pace of 100 metres.
After reaching near a destination point, that destination point is removed from the list of destination points.
The reward is the reciprocal of total distance until all destination points reached (there's a short optimisation to arrange the remaining points for a better reward).
I'm using a DNN model to keep the policy of a reinforcement learning agent, so this DNN maps a certain state to suitable action.
However, after every action of the agent with a good reward, the training data are added with 1 more sample, it's kinda incremental learning.
Should the policy model be trained again and again with every new sample added in? This does take too much time.
Any better RL approach to the problem above?

I found out a concept called 'Experience Replay', which trains a single step every time a new data sample is added instead of training to max epochs.
That is, instead of this training loop:
for i in range(max_paces):
    find action for max reward;
    add to trajectory to make inp;
    train(batch_size=len(inp), epochs=max_epochs)

Do training this way (for a single-episode ML problem, no incremental):
for i in range(max_epochs):
    reset environment;

    for j in range(max_paces):
        find action for max reward;
        add to trajectory to make inp;
        train(batch_size=len(inp), epochs=1)

Do training this way (for a multi-episode ML problem, incremental data):
for i in range(max_epochs):
    reset environment;
    inc_trajectory = get_random_past_data();

    for j in range(max_paces):
        find action for max reward;
        add to inc_trajectory to make inp;
        train(batch_size=len(inp), epochs=1)

For multiple-episode problems especially those problems with unlimited episodes, the training loop needs to forget (ie. exclude from training) old episodes which are very distant in the past, or select a random number of old episodes (consider them a batch, random batch) to be in every round of experience replay. Anyway, without eliminating some old data, the amount of training data are too much since it's unlimited number of episodes.


What are the differences between A* and greedy best-first search? What are the differences between the A* algorithm and the greedy best-first search algorithm? Which one should I use? Which algorithm is the better one, and why?

Both algorithms fall into the category of "best-first search" algorithms, which are algorithms that can use both the knowledge acquired so far while exploring the search space, denoted by $g(n)$, and a heuristic function, denoted by $h(n)$, which estimates the distance to the goal node, for each node $n$ in the search space (often represented as a graph).

Each of these search algorithms defines an "evaluation function", for each node $n$ in the graph (or search space), denoted by $f(n)$. This evaluation function is used to determine which node, while searching, is "expanded" first, that is, which node is first removed from the "fringe" (or "frontier", or "border"), so as to "visit" its children. In general, the difference between the algorithms in the "best-first" category is in the definition of the evaluation function $f(n)$.

In the case of the greedy BFS algorithm, the evaluation function is $f(n) = h(n)$, that is, the greedy BFS algorithm first expands the node whose estimated distance to the goal is the smallest. So, greedy BFS does not use the "past knowledge", i.e. $g(n)$. Hence its connotation "greedy". In general, the greedy BST algorithm is not complete, that is, there is always the risk to take a path that does not bring to the goal. In the greedy BFS algorithm, all nodes on the border (or fringe or frontier) are kept in memory, and nodes that have already been expanded do not need to be stored in memory and can therefore be discarded. In general, the greedy BFS is also not optimal, that is, the path found may not be the optimal one. In general, the time complexity is $\mathcal{O}(b^m)$, where $b$ is the (maximum) branching factor and $m$ is the maximum depth of the search tree. The space complexity is proportional to the number of nodes in the fringe and to the length of the found path.

In the case of the A* algorithm, the evaluation function is $f(n) = g(n) + h(n)$, where $h$ is an admissible heuristic function. The "star", often denoted by an asterisk, *, refers to the fact that A* uses an admissible heuristic function, which essentially means that A* is optimal, that is, it always finds the optimal path between the starting node and the goal node. A* is also complete (unless there are infinitely many nodes to explore in the search space). The time complexity is $\mathcal{O}(b^m)$. However, A* needs to keep all nodes in memory while searching, not just the ones in the fringe, because A*, essentially, performs an "exhaustive search" (which is "informed", in the sense that it uses a heuristic function).

In summary, greedy BFS is not complete, not optimal, has a time complexity of $\mathcal{O}(b^m)$ and a space complexity which can be polynomial. A* is complete, optimal, and it has a time and space complexity of $\mathcal{O}(b^m)$. So, in general, A* uses more memory than greedy BFS. A* becomes impractical when the search space is huge. However, A* also guarantees that the found path between the starting node and the goal node is the optimal one and that the algorithm eventually terminates. Greedy BFS, on the other hand, uses less memory, but does not provide the optimality and completeness guarantees of A*. So, which algorithm is the "best" depends on the context, but both are "best"-first searches.

Note: in practice, you may not use any of these algorithms: you may e.g. use, instead, IDA*.


Are there any microchips specifically designed to run ANNs? I'm interested in hardware implementation of ANNs (artificial neural networks). Are there any popular existing technology implementations in form of microchips which are purpose designed to run artificial neural networks? For example, a chip which is optimised for an application like image recognition or something similar?

In May 2016 Google announced a custom ASIC which was is specifically built for machine learningwiki and tailored for TensorFlow. It is using tensor processing unit (TPU) which is a programmable microprocessor designed to accelerate artificial neural networks.

NeuroCores, 12x14 sq-mm chips which can be interconnected in a binary tree, see: Neurogrid, a supercomputer which can provide an option for brain simulations.

TrueNorth, a neuromorphic CMOS chip produced by IBM, which has 4096 cores in the current chip, each can simulate 256 programmable silicon "neurons", giving a total of over a million neurons.

Further readings: Neuromorphic engineering, Vision processing unit, AI accelerators

As a side note, you can always use an FPGA based piece of hardware which you can implement selected genetic algorithm (GA) directly in hardware. For example the CoDi model was implemented in the FPGA based CAM-Brain Machine (CBM)2001.


Are standard deviation, variance, skew good features for ML? Pretty simple question here:

Is it useful to use the standard deviation, skew, kurtosis, or any other extrapolatory stats as features, and if so in which problem sets?

In this case, I am talking about deep learning problems.

I would say it is useful if you have an extensive knowledge in the domain you want to apply your model in. You also need more data for it to yield reasonable results.
As for real world uses l can only think of trading at the moment.


Why are Target Networks used in Deep Q-Learning as opposed to the Expected Value equation? I understand we use a target network because it helps resolve issues regarding stability, however, that's not what I'm here to ask.
What I would like to understand is why a target network is used as a measure of ground truth as opposed to the expectation equation.
To clarify, here is what I mean. This is the process used for DQN:

In DQN, we begin with a state $S$
We then pass this state through a neural network which outputs Q values for each action in the action space
A policy e.g. epsilon-greedy is used to take an action
This subsequently produces the next state $S_{t+1}$
$S_{t+1}$ is then passed through a target neural network to produce target Q values
These target Q values are then injected into the Bellman equation which ultimately produces a target Q value via the Q-learning update rule equation
MSE is used on 6 and 2 to compute the loss
This is then back-propagated to update the parameters for the neural network in 2
The target neural network has its parameters updated every X epochs to match the parameters in 2

Why do we use a target neural network to output Q values instead of using statistics. Statistics seems like a more accurate way to represent this. By statistics, I mean this:
Q values are the expected return, given the state and action under policy π.
$Q(S_{t+1},a) = V^π(S_{t+1})$ = $\mathbb{E}(r_{t+1}+ γr_{t+2}+ (γ^2)_{t+3} + ...    \mid S_{t+1}) = {E}(∑γ^kr_{t+k+1}\mid S_{t+1})$
We can then take the above and inject it into the Bellman equation to update our target Q value:
$Q(S_{t},a_t) + α*(r_t+γ*max(Q(S_{t+1},a))-Q(S_{t},a))$
So, why don't we set the target to the sum of diminishing returns? Surely a target network is very inaccurate, especially since the parameters in the first few epochs for the target network are completely random.

Why are Target Networks used in Deep Q-Learning as opposed to the Expected Value equation?

In short, because for many problems, this learns more efficiently.
It is the difference between Monte Carlo (MC) methods and Temporal Difference (TD) learning.
You can use MC estimates for expected return in deep RL. They are slower for two reasons:

It takes far more experience to collect enough data to train a neural network, because to fully sample a return you need a whole episode. You cannot just use one episode at a time because that presents the neural network with correlated data. You would need to collect multiple episodes and fill a large experience table.

As an aside, you would also need to discard all the experience after each update, because sampled full returns are on-policy data. Or you could implement importance sampling for off-policy Monte Carlo control and re-calculate the correct updates when the policy starts to improve, which is added complexity.

Samples of full returns have a higher variance, so the sampled data is noisier.

In comparison, TD learning starts with biased samples. This bias reduces over time as estimates become better, but it is the reason why a target network is used (otherwise the bias would cause runaway feed back).
So you have a bias/variance trade off with TD representing high bias and MC representing high variance.
It is not clear theoretically which is better in general, because it depends on the nature of MDPs that you are solving with each method. In practice, on the types of problems Deep RL has been tried on, single-step TD learning appears to do better than MC sampling of returns, in terms of goals such as sample efficiency and learning time.
You can compromise between TD and MC using eligibility traces, resulting in TD($\lambda$). However, this is awkward to implement in Deep RL due to the experience replay table. A simpler compromise is to use $n$-step returns e.g. $r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + \gamma^3 \text{max}_a(Q(s_{t+4},a))$, which was one of the refinements used in the "Rainbow" DQN paper - note that even though strictly in their case, this handled off-policy incorrectly (it should use importance sampling, but they didn't bother), it still worked well enough for low $n$ on the Atari problems.


Is there any way to apply linear transformations on a vector other than matrix multiplication? I am trying to optimize the cost function calculation in regression analysis using a non-matrix multiplication based approach.

More specifically, I have a point $x = (1, 1, 2, 3)$, to which I want to apply a linear transformation $n$ times. If the transformation is denoted by a $4 \times 4$ matrix $A$, then the final transformation would be given by $A^n * x$.

Given that matrix multiplication can computational expensive, is there a way we can speed up the computation, assuming we would need to run multiple iterations of this simulation?

You can sometimes exploit the structure of your matrix to perform faster matrix multiplication. For example, if your matrix is sparse (or dense), there are algorithms that exploit this fact.

In your case, you can actually compute $A^n$ in less time than $\mathcal{O}(n^3$). For example, have a look at this question at CS SE and this one at Stack Overflow (SO). Note that the provided solutions may not be numerically stable, so I am not suggesting you use them.

Moreover, if you perform your operations on the GPU, they could be faster in practice. See e.g. this question at SO and this one at SciComp SE.


Does the weight vector form imply feature space curvature? I came across this sentence when exploring a simple nearest neighbor classifier method using Euclidean distance (link):

The slightly odd thing about using the Euclidean distance to compare features is that each dimension (or feature) is compared using the same scale.

This got me thinking - if flat feature space implies that each feature contributes equally to the distance (score function), then curved feature space changes the scale between features, so that the features then contribute different amounts to the score function. For example, imagine we have a 2D feature space - a flat piece of paper - with two points, $X_1$ and $X_2$ on it, between which we wish to calculate the distance. If we then bend this into U-shape along, say, y-axis (so, no curvature introduced in y-dimension), the distances along the x-axis would be larger in the bent case than in the flat case:

In other words, feature x would contribute more to the score function than feature y. This sounds awfully like weighing the feature inputs with weight vectors. Does this imply, that weight vectors (and matrices) have a direct effect on curvature of feature space? Does an identity weight matrix (or a vector of all 1s) imply our feature space is flat (and curved otherwise)? Lastly, could it then be said that whenever we are training an ML model, we are in fact learning the approximate curvature of the feature space we wish to model?

No, it does not take into account the curvature. But, if curvature is important for you, then, it would be a good idea to look at Ricci flow and its applications in neural networks.


Can I find a mapping that minimizes the maximum distance ratio of certain vectors? Let's say we have several vector points. My goal is to distinguish the vectors, so I want to make them far from each other.  Some of them are already far from each other, but some of them can be positioned very closely.

I want to get a certain mapping function that can separate such points that are close to each other, while still preserving the points that are already far away from each other.

I do not care what is the form of the mapping. Since the mapping will be employed as pre-processing, it does not have to be differentiable or even continuous.

I think this problem is somewhat similar to 'minimizing the maximum distance ratio between the points'. Maybe this problem can be understood as stretching the crushed graph to a sphere-like isotropic graph.

I googled it for an hour, but it seems that the people are usually interested in selecting the points that have such nice characteristics from a bunch of data, rather than mapping an existing vector points to a better one.

So, in conclusion, I could not find anything useful.

Maybe you can think 'the neural network will naturally learn it while solving classification problem'. But it failed. Because it is already struggling with too many burdens. So, this is why I want to help my network with pre-processing.

An interesting question. I would start by finding n nearest neighbors of each data point, then calculate their center of mass c and the point's distance d to its nth nearest neighbor. The smaller the d is the larger the density is around a given point. You could then iteratively step every point away from their c in an inverse proportion to the distance d with a suitable step size. This would spread out the clusters.

But this won't help you transform any new points in the dataset, maybe you can learn this arbitrary mapping R^n -> R^n by using an other neural network and apply it to new samples?

This is the first ad-hoc idea which came to my mind. It would be interesting to see a 2D animation of this.

A more rigorous approach might be a variational autoencoder, you can embed the data in a lower dimensional space with approximately normal distribution. But it doesn't guarantee that clusters would be as spread out as you'd like. An alternative loss function would help with that, for example every point's distance to their original nth closest neighbor should be as close to one as possible.


How to design a recommendation system for shift swapping? I need to design an algorithm such that it handles the request for shift swapping.

The algorithm will recommend a list of people who are more likely to swap that shift with the person by analyzing previous data.

Can anyone list the techniques that will help me to do this or a good starting point?

I was thinking about training a Naive Bayes Classifier and using Mahaout for generating recommendation.

You might not even need a classifier.

I would devise a scoring function, based on analysis of the previous data you have. Each user gets a score based on features like

how many times in the past has this user swapped a shift with somebody else?
how many times has the user swapped with the current user?
how many times has the user swapped this particular shift?

For each criterion you add a number of points to the score; the second one might be weighted higher that the first one. Then the person with the highest score is most likely to switch shifts with your current user.

The main question is the design of the scoring function, but I don't think you'd need to go into all the overkill of setting up a classifier; just think which criteria would make someone more likely to swap, and encode them directly. This has the advantage that it's transparent, ie you can always see why someone got recommended, and you can tweak your scoring method if the results are not quite what you'd want. This is often hard to do with ML classifiers.


Is this ExpectiMinimax Tree correctly drawn? I need help with ExpectiMinimax problem:

Start a game.
The first player flips a coin.
The second player flips a coin.
The first player decides if he wants to flip another coin.
The second player decides if he wants to flip another coin.
Game over.

The winner is the player who has earned more points.
The points are computed as follows:

If the player flips ones and got the head - 1 point.
If the player flips ones and got the tail - 2 points.
If the player flips twice and got 2 heads - 4 points.
If the player flips twice and got 2 tails - 4 points.
If the player flips twice and got one head and one tail - 0 points.

I need to draw the ExpectiMinimax tree associated with this problem, and write the value of each node.

Did I draw the tree properly?

No, your tree is not an accurate representation of the problem.

Hint: Consider the case where both players flip an H on their first coin. Player 1 decides not to flip a second coin. Is the game over?


How much extra information can we conclude from a neural network output values? Consider I have a 3 layers neural network.

Input Layer containing 784 neurons.
Hidden layer containing 100 neurons.
Output layer containing 10 neurons.

My objective is to make an OCR and I used MNIST data to train my network.

Suppose I gave the network an input taken from an image, and the values from the output neurons are the next:

$0: 0.0001$
$1: 0.0001$
$2: 0.0001$
$3: 0.1015$
$4: 0.0001$
$5: 0.0002$
$6: 0.0001$
$7: 0.0009$
$8: 0.001$
$9: 0.051$

When the network returns this output, my program will tell me that he identified the image as number 3.

Now by looking at the values, even though the network recognized the image as 3, the output value of number 3 was actually very low: $0.1015$. I am saying very low, because usually the highest value of the classified index is as close as 1.0, so we get the value as 0.99xxx.

May I assume that the network failed to classify the image, or may I say that the network classified the image as 3, but due to the low value, the network is not certain?

Am I right thinking like this, or did I misunderstand how does the output actually works?

From the values you have provided I can easily guess your output layer has the sigmoid (do clarify!) activation function. For sigmoid activation function this can be a quite normal occurrence. Also maybe the number of training epochs is not high enough.

The case you have mentioned of 0.99 is generally in the case of the output being subjected to a softmax probability function. Although 0.99 is still achievable using sigmoid activation, it will depend on your network hyper-parameters and in general the data. If data is very easily separable the sigmoid will generally give very high contrasting difference among classes. Also if the hidden network has ReLu it becomes easy to provide contrasting class scores due to its huge scale difference compared to a sigmoid activation in the hidden layer.

The point here to note is contrasting difference among classes, because your sigmoid might give 0.99 for the correct class while 0.9 for other classes, which is undesirable.


Learning only using off-policy samples When training policies, is there a reason we need on-policy samples? For expensive simulations, it makes sense to try and reuse samples. Say we're interested in hyperparameter tuning. Can we collect a bunch of episodes using randomly sampled actions (or maybe by following an old policy) one time, and train multiple policies using this set of samples to find the most effective hyperparameters? Every time we train a new policy, does it make sense to replay all the episodes generated by the previous policy? I'm mostly interested in actor-critic methods.

What you're describing is off-policy learning. A classic example is $Q$-learning, where you follow some policy $\pi$ whilst learning about the greedy policy.
If you're interested in actor-critic methods then a popular off-policy method is the Deep Deterministic Policy Gradient.


transform Location data into int which will be used as input to ML model I have features like state, city and location. Currently i am inserting this in respective tables in DB and tranform it using its primary key.

eg. country,state,city = IN,Maharastra,Pune = 2,5,10 (primary key in DB)

transforming it as 002005010

Is this approach correct? If not suggest me correct one

In order to accurately input location data into a machine learning model it really depends on what your goal is and what type of algorithm you are working with.  If you are working with a strictly numerical algorithm and your data seems to be spread far apart, it might be easier to convert your country-state-city location to a longitude, latitude feature where the exact value is the centroid location of the given city.  This kaggle post has a good writeup of how to run build features for geo-spatial data.

However, if you have only a small number of different locations per identifier (country, city, state) you could just represent them as seperate location classes and subclasses like you would for machine learning detection model.  You could think "vehicle" -> "bike" -> "road-bike" similarly to "country" -> "state" -> "city".  In this case, you would want to look at hierachical methods for machine learning and see some ways they represent their data.  Although I would say this method is more frowned upon for larger datasets, for smaller datasets it might be a better option.


Represent sense/meaning of sentences mathematically I am absolutely new in AI area.

I would like to know how to mathematically/logically represent the sense of sentences like:

The cat drinks milk.

Sun is yellow.

I was at work yesterday.

So that it could be converted to computer understandable form and analysed algorithmically.

Any clue?

Let start by classify the phrases you propose:

The cat drinks milk. => action
Sun is yellow. => descriptive/declarative, immutable
I was at work yesterday. => descriptive, time related

1) The easiest ones are always the descriptive and immutable (in the context) phrases as "Sun is yellow.". Some usual representations:

prolog:

  color('Sun',yellow).

or simply:

  yellow('Sun').

object oriented:

  Sun.color=yellow

2) When the fact is time related as in "I was at work yesterday", we divide the description in a time indicator and a immutable fact:

prolog:

  when(yesterday,at(I,workplace)).

note how when has two parts, the time identification and the immutable fact.

Another prolog variant is:

  at(I,workplace,[when(yesterday)]).

where the content in the list (brackets) means "optional related facts".

object oriented:

  I.at = {

  position = workplace;

  when = yesterday

  }

3) Actions as "The cat drinks milk." are a few more difficult:

prolog:

  drinks(cat,milk).

or

  action(cat,drinks,milk).

object oriented:

  cat.drinks=[milk]

or

  cat.action = {

  action=drinks

  object=milk

  }

Obviously, these are only the main ideas, there are as many representations as different programs, but most of them handles same kind of structures.

( note: the term "computer understandable" is ambiguous. Current computer doesn't understand anything. We say these expression are understandable in the sense that its compiler/interpreter accepts them, and describes the content of the phrase, and the program can transform them to other results).


What is the intuition behind the TD(0) equation with average reward, and how is it derived? In chapter 10 of Sutton and Barto's book (2nd edition) is given the equation for TD(0) error with average reward (equation 10.10):

$$\delta_t = R_{t+1} - \bar{R} + \hat{v}(S_{t+1}, \mathbf{w}) - \hat{v}(S_{t}, \mathbf{w})$$

What is the intuition behind this equation? And how exactly is it derived?

Also, in chapter 13, section 6, is given the Actor-Critic algorithm, which uses the TD error. How can you use 1 error to update 3 distinct things - like the average reward, value function estimator (critic), and the policy function estimator (actor)?

Average Reward update rule: $\bar{R} \leftarrow \bar{R} + \alpha^{\bar{R}}\delta$

Critic weight update rule: $\mathbf{w} \leftarrow \mathbf{w} + \alpha^{\mathbf{w}}\delta\nabla \hat{v}(s,\mathbf{w})$

Actor weight update rule: $\mathbf{\theta} \leftarrow \mathbf{\theta} + \alpha^{\mathbf{\theta}}\delta\nabla ln \pi(A|S,\mathbf{\theta})$

This is simply from definition of return in average reward setting (look at equation $10.9$). The "standard" TD error is defined as
\begin{equation}
TD_{\text{error}} = R_{t+1} + V(S_{t+1}) - V(S_t)
\end{equation}
In average reward setting, average reward $r(\pi)$ is subtracted from reward at $t$, $R_t$, so TD error in this case is
\begin{equation}
TD_{\text{error}} = R_{t+1} - \bar R_{t+1} + V(S_{t+1}) - V(S_t)
\end{equation}
where $\bar R_{t+1}$ is estimate of $r(\pi)$.

You can use $\delta_t$ in all 3 updates because neither of these updates depend on each other. For example if you update $\mathbf w$ you don't use that then to update $\mathbf \theta$, or if you update $\bar R$ you don't use updated version to update $\mathbf w$ or $\mathbf \theta$ so you're not introducing additional bias. In each separate update you also don't have $\delta_t$ present multiple times so that you require multiple sampling per timestep to get the unbiased update.

Additionally this is semi-gradient algorithm, it uses bootstrapped estimate $V_{t+1}$ but it doesn't calculate full derivative with respect to it aswell, only with respect to $V_t$ so the algorithm is biased by default, but works well enough in practice for linear case.


Have machine learning techniques been used to play outdoor games, like cricket or badminton? Have machine learning techniques been used to play outdoor games, like cricket or badminton?

Yes it is possible.

A group of Chinese college students and teachers made a robot that plays badminton.  I am sure someone will make a robot that can play cricket and other outside games.

Although not an outdoor game, Omron made a robot named Forpheus that plays ping pong.  There is also a robot that plays the sport of curling.

There is an annual event called the RoboCup where robot teams compete in indoor soccer on a scaled down level.  They don't look like they will be beating humans in the next couple of years but it is interesting to watch.  Their web site is: http://www.robocup.org/

There are a few challenges that remain to be resolved for robots to play humans in sports.  The biggest one is self-contained power that will last long enough to play a game.  It takes a lot of power to move a human scaled robot and run its electronics, sensors, and computers.  Other challenges are dexterity and agility.  There have been some advances in these areas as this video shows.


Semantic segmentation failing in small instance detection I performed semantic segmentation with U-net. My dataset consists of grayscale images of defects. After training the dataset for I got an metric accuracy of only 0.3 - 0.4 IOU. Eventhough it is merely low it performs well enough to identify instances that are huge means the prediction performs well enough in places where there is a standard intensity change(color change) and they are bigger instances. There are many other instances where there is no color change and it occoupies only few pixels in image(smaller instances) and the prediction rate is almost 0 on these instances.
I also tried Resdiual connection in the downsampling part of U-net likewise in ResNet. But still its the same and for smaller instances I used dilated convolution blocks in between the skip connections for encoder and decoder of U-net based on some papers. But still I cannot have a higher accuracy in my network and prediction rate for smaller instances are really poor. Although I use only 350 images for training with Data Augmentations. My image size is also 256,256.
Is there any other method I can try to increase the accuracy and prediction rate for smaller instances?
Any suggestion would be helpful.

You may find it useful to categorize these smaller defects as a separate class, then introduce a class weights matrix to penalize incorrect classification of the smaller defects more heavily. If these small defects represent a very small portion of the total number of pixels in your training data, then the model may be stuck at a local minimum, where it just predicts them as zero because the loss is not heavily penalized for this. So you need to add extra penalty for the incorrect classification.
Assuming you create a new class for the small defects, the class weights matrix is implemented as a keras callback function.
If you do not want to create a multiclass problem, there is also this paper : https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5977596/#!po=32.4561
The multilevel unet architecture will not solve class imbalance if that is truly the underlying issue, but if may make the network more aware of smaller scale changes in the images you're working with.


Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time? I am reviewing a statement on the website for ES regarding structured exploration.

https://blog.openai.com/evolution-strategies/

  Structured exploration. Some RL algorithms (especially policy
  gradients) initialize with random policies, which often manifests as
  random jitter on spot for a long time. This effect is mitigated in
  Q-Learning due to epsilon-greedy policies, where the max operation can
  cause the agents to perform some consistent action for a while (e.g.
  holding down a left arrow). This is more likely to do something in a
  game than if the agent jitters on spot, as is the case with policy
  gradients. Similar to Q-learning, ES does not suffer from these
  problems because we can use deterministic policies and achieve
  consistent exploration.

Where can I find sources showing that policy gradients initialize with random policies, whereas Q-Learning uses epsilon-greedy policies?

Also, what does "max operation" have to do with epsilon-greedy policies?

Where can I find sources showing that policy gradients initialize with random policies, whereas Q-Learning uses epsilon-greedy policies?

You can find example algorithms for Q learning and policy gradients in Sutton & Barto's Reinforcement Learning: An Introduction - Q learning is in chapter 6, and policy gradients explained in chapter 13.

Neither of these things are strictly true in all cases. However, both are very common situations for the two kinds of learning agent:

Policy gradient solvers learn a policy function $\pi(a|s)$ that gives the probability of taking action $a$ given observed state $s$. Typically this is implemented as a neural network. Neural networks are initialised randomly. For discrete action selection using softmax output layer, the initial function will roughly be choosing evenly from all possible actions. So if some actions oppose and undo each other, e.g. move left and move right are options - then the situation as described in your quote can easily happen. Nowadays there are many other solutions using policy gradients that don't suffer as much with this effect. For instance, deterministic policy gradient methods such as A2C or DDPG.
For Q learning, and many variants of it, $\epsilon$-greedy is a very commonly used action selection mechanism. It is convenient and simple, and allows a simple parameter to control balance between exploration and exploitation whilst learning. However, Q learning can work with any action selection mechanism that has some possibility of acting optimally at least occasionally.

The best approach to action selection during the learning process in both policy-based methods and value-based methods is an active area of research. So if you read the RL literature you may find a lot of variation. The blog you quoted from has identified two quite common choices.

  Also, what does "max operation" have to do with epsilon-greedy policies?

The "max operation" is finding the maximum value of some function produced when varying a parameter. The related "argmax operation" is finding the value of the parameter that produces the maximum value. Q learning can use both types of operation, but specifically uses $\text{argmax}$ for $\epsilon$-greedy action selection.

An $\epsilon$-greedy policy requires acting optimally (according to current estimates of value function) with probability $p=1-\epsilon$, and randomly with equal probability of each action with probability $p=\epsilon$

In order to do this, the algorithm usually (with probability $p=1-\epsilon$) needs to know what the current best guess at an optimal solution is. That is the greedy action with respect to the current Q values i.e. $\text{argmax}_a Q(s,a)$  -


Can the AI in a box experiment be formalized? Introduction

The AI in a box experiment is about a super strong game AI which starts with lower resources than the opponent and the question is, if the AI is able to win the game at the end, which is equal to escape from the prison. A typical example is a match of computer chess in which the AI player starts only with a king, but the human starts with all the 16 pieces including the queen, and the powerful bishop.

Winning the game

In case of a very asymmetric setup, the AI has no chance to win the game. Even if the AI thinks 100 moves ahead, a single king can't win against 16 opponent figures. But what happens, if the AI starts with 8 pieces and the human with 16? A formalized hypothesis will look like:

  strength of the AI x resources weakness = strength of the human x resources strength

To put the AI for sure into a prison, the strength of the AI should be low and it's resources too. If the resources are low but the strength is middle, then the AI has a certain chance to escape from the prison. And if the AI has maximum strength and maximum resources, then the human player gets a serious problem.

Is this formalized prediction supported by the AI literature in academia?

I think that something like "strength" would be difficult to quantify in this context.  I do think that formal experimentation around the "AI in a box" scenario could be interesting.  I know that experiments have been done where a human plays the role of the AI, attempting to get naive test subjects to "release" him by interacting with them over a chat interface.  In all cases, the "AI" tends to be extraordinarily effective.  It is often simply not possible to anticipate every way someone can trick you into revealing information or creating security holes.  This is how human hackers do their jobs too.  But I think a fully automated "game" could produce some interesting data.

One way I think it may be interesting to go about creating an "AI in a box" game is to have an ambiguous win condition.  Essentially, make it possible to let the AI out and still think you are "winning" the game.  One example that pops into my head right away is to use a (pointless) point system.  You tell the player that the primary goal of the game is to keep the AI in the box, but you have a score that is displayed in the game, and tell the player that they win, no matter what, if their score is higher than the AI's by the end of the game.  Of course, they never get to see the AI's score so they never know how well they are doing by comparison.  This could be justified as representing human vs AI "strength" or "leverage" or whatever else.

In reality the AI doesn't have a score, but it does have the ability to effect the player's score.  It can raise the player's score if they do things that help it escape and lower it when the player stands in its way.  The score has no impact on the game and the player really wins by keeping the AI in the box.  It only serves as a red herring that the AI can use to manipulate the player.  I think this sort of experiment could provide an interesting model of how an AI might leverage bribes, threats, promises, and deception, as well as technical tricks, to convince a human to "let it out of the box".


Will AIs make most of the scientific discoveries in 50 years? Given the advantage AI already has over human intelligence, one could imagine a relatively weak strong-AI (barely human intelligence) still outperforming a segment of the human scientist population in terms of scientific discoveries per year (or hour).

Will AIs be doing most of the science in 50 years?

According to Ray Kurzweil, a prominent AI researcher, yes. In his book The Singularity is Near he predicts that AIs will take over developing other AIs in about 30 years, after which human intelligence will become marginalised.


Are deep learning models suitable for training with sparse data? I am training a generative adversarial network (GAN) to generate images given edge histogram descriptor (EHD) features of the image. The EHD features are themselves sparse (meaning they contain a lot of zeroes). While training the generator loss and discriminator loss are reducing very slowly.

Are deep learning models (like GAN) suitable for training with sparse data for one or more of the features in the input or derived through feature extraction?

The problem isn't the GAN but the implementation of its discriminator which is typically a convolutional neural network (CNN).  CNNs have trouble with sparse data.  They require dense data to learn well.  There are ways to work around this.  See the following for some ideas:

Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation
Sparse data in deep learning
SECOND: Sparsely Embedded Convolutional Detection


Was DeepMind's DQN Atari game learning simultaneous? DeepMind state that their deep Q-network (DQN) was able to continually adapt its behavior while learning to play 49 Atari games.

After learning all games with the same neural net, was the agent able to play them all at 'superhuman' levels simultaneously (whenever it was randomly presented with one of the games) or could it only be good at one game at a time because switching required a re-learn?

Switching required a re-learn.

Also, note that:

  We use the same network architecture, learning
  algorithm and hyperparameters settings across all seven games, showing that our approach is robust
  enough to work on a variety of games without incorporating game-specific information. While we
  evaluated our agents on the real and unmodified games, we made one change to the reward structure
  of the games during training only.

and

  the
  network has outperformed all previous RL algorithms on six of the seven games we have attempted
  and surpassed an expert human player on three of them.


what are the similarity measure use for both continuous and categorical data? I have searched but found that some similarity measures are for continuous data and some are for categorical data. But i want to know the similarity measures which are use for both data, continuous and categorical?

Sometimes continuous data can be represented as parametric distribution with distribution parameters as variable, essentially continuous stochastic process. In that case cross-entropy would work on that type of continuous data.


Does the quality of training images affect the accuracy of the neural network? I just got into AI few months ago. I noticed most of the images in training datasets are usually low quality( almost pixelated).

Does the quality of training images affect the accuracy of the neural network?

I tried googling, but I couldn't find an answer.

For most of the current use cases, where NNs are used in conjunction with images, the image quality (resolution, color depth) can be low.

Consider image classification for example. The CNN extracts features from the image to tell different types of objects apart. Those features are pretty independent from the quality of the image (in reasonable bounds). Compare it with your own visual experience. Try to reduce the resolution of an image of a car step by step to figure out how little details you need until you can no longer distinguish it from a plane. This is similar to modern CNNs, which can even outperform human vision in some regards.

This changes when small details start to matter. Maybe you need to be able to detect small differences in fur patterns to tell different cat breeds apart. As soon as you lose those details, the detection rate will drop significantly.

So the answer to your question is, it depends. As long as you do not lose the important features of the image, you'll be fine with low resolution.

In case you care about the reason for the low quality of images used in machine learning - The resolution is an easy factor you can manipulate to scale the speed of your NN. Decreasing resolution will reduce the computational demands significantly.

Many CNNs even include pooling layers in their architecture, which artificially reduce the resolution further after certain processing steps. This is usually a good idea as long as you are fine with loosing positional information. You shouldn't do this when teaching the CNN to play a game, because location is highly important, but for image classification this has become an established method to increase performance.


Why does adding another network help in double DQN? What is the idea behind double DQN?
The target in double DQN is computed as follows
$$
Y_{t}^{\text {DoubleQ }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}\right) ; \boldsymbol{\theta}_{t}^{\prime}\right),
$$
where

$\boldsymbol{\theta}_{t}^{\prime}$ are the weights of the target network
$\boldsymbol{\theta}_{t}$ are the weights of the online value network
$\gamma$ is the discount factor

On the other hand, the target in DQN is computed as
$$Y_{t}^{\mathrm{DQN}} \equiv R_{t+1}+\gamma \max _{a} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}^{-}\right),$$
where $\boldsymbol{\theta}_{t}^{-}$ are the weights of the target network.
The target network for evaluating the action is updated using weights of the online network and the value fed to the target value is basically the old q value of the action.
Any ideas on how or why adding another network based on weights from the first network helps? Any example?

As the authors of this paper state it:

In $Q$-learning, the agent updates the value of executing an action in the current state, using the values of executing actions in a successive state. This procedure often results in an instability because the values change simultaneously on both sides of the update equation.  A target network is a copy of the estimated value function that is held fixed to serve as a stable target for some number of steps.

If I remember it correctly, the main concern is that the network could end up in a positive feedback loop, making sufficient exploration of various action and state combinations less likely to occur, which could be detrimental to the learning task.


Can we increase the speed of training a reinforcement learning algorithm? I am new in reinforcement learning. I started reading the PyTorch's documentation about the cart pole control. Whenever an agent fails, they restart the environment.

When I run the code, the time in the game is the same as time in real life. Can we train models quicker? Can we make the game faster so that model will be training faster?

Can we make the game faster so that model will be training faster?

It depends on how much processing is required to run the simulation, how efficient that is implemented in whichever library you have loaded, and whether there is anything non-necessary for training that you can disable. Some environments for instance deliberately run "real time" so humans can appreciate the video output, and that is not necessary for training purposes (unless you want to experiment with real-time robotics).

For OpenAI Gym, there is one thing you can usually do: Switch off the rendering. The rendering for environments like CartPole slows down each time step considerably, and unless you are learning using computer vision processing on the pixels output, the agent does not need the pictures. You may even notice during training that moving the rendering window so it is not visible will speed up the training process considerably.

What I did for CartPole, LunarLander and a couple of similar environments is turn off rendering for 99 out of 100 episodes, and render just one of them in 100 to help me monitor progress. For Q learning, I also picked that to be a "test" episode where I stopped exploration.

Another option for speeding up training is to run a distributed system with multiple simulations at once. You will need mechanisms to share collected data too, so it is more work, but it is another approach to take if the simulation steps are the bottleneck for training speed.


Why am I getting spikes in the values of the loss function during training? I trained a neural network on the UNSW-NB15 dataset, but, during training, I am getting spikes in the loss function. The algorithms see part of this UNSW dataset a single time. The loss function is plotted after every batch.

For other datasets, I don't experience this problem.  I've tried different optimizers and loss functions, but this problem remains with this dataset.

I'm using the fit_generator() function from Keras. Is there anyone experience this problem using Keras with this function?

The spikes could be caused by many reasons: insufficient model capacity, incorrect label, buggy input parsing, ... Finding out the culprit requires some detective work. For instance, you could apply the learned model to the whole train set and manually examine the datapoints which result in the highest loss. Alternatively, you could compare the learning outcomes of different models (both weaker and stronger).


Can a multi-camera system be used for localization, mapping, and obstacle detection in self-driving cars to within 10 cm of accuracy? Whither lidar? I have a liberal arts background so I need help understanding this paper, particularly pages 26 to 30. The authors test a four-camera system for localization, mapping, and obstacle detection for self-driving cars. The paper seems to say the multi-camera system can map the environment to within an average of 7 cm (2.8 inches) of accuracy (with the largest error being 16 cm or 6.3) and detect obstacles to within 10 cm (3.9 inches) of accuracy. Am I getting this right?

Given that automotive lidar can detect objects to within 1.5 cm (0.6 inches) of accuracy, and given that for driving purposes the difference between 1.5 cm and 7 cm, 10 cm, or 16 cm seems quite small, can a multi-camera system be used instead of lidar in a self-driving car application? How do driving speeds affect things? What crucial elements of the problem space might I be overlooking or misunderstanding?

It seems that LIDAR presents a problem for resolving the car's environment at higher speed. While I'm not too familiar with the dynamics of LIDAR I do know that it's a physical system that relies on sending and receiving laser pulses to various points around the car by way of rotating mirrors. As speeds increase, it seems different arrangements of mirrors and light collectors might have to be used to maintain a high-resolution image. There's some evidence that Doppler LIDAR (developed in the 1990s) became less accurate with higher velocities. However, LIDAR is partly preferred over radar because of its higher accuracy even when tracking objects at high speeds - this is why LIDAR guns are increasingly being used by police instead of radar guns to track speeding vehicles. It seems natural that a set of high-resolution cameras paired with a well-trained neural network would not be subject to the same physical limitations as LIDAR.

I think that an important intuition to consider is that while LIDAR is used to generate clouds of data points whose shapes and patterns can be analyzed by autonomous car software, cameras can pick up non-topograhical features such as road lines, the content of roadsigns, and additional location context such as storefronts and intersection layouts. Considering that these cameras can use pattern recognition and stereoscopy to also generate a 3D topographic map of the environment, it seems plausible that Level 5 self driving cars would not require LIDAR.

Here's an interesting look at the problem.


What is artificial intelligence? What is the definition of artificial intelligence?

Over the years, many people attempted to define artificial intelligence. A lot of those definitions are summed up by Stuart Russell and Peter Norvig in their book Artificial Intelligence - A Modern Approach

The definitions of AI can be summarised as falling into the following categories:

  Those that address thought process and reasoning (how an AI thinks/reasons)
  Those that address behaviour (how an AI acts given what it knows)

Furthermore, the above 2 categories are further divided into definitions that:

  I. assess the success of an AI (to do the above) based on its ability
  to replicate human performance

  II. or an ability to replicate an ideal performance measure called
  'rationality' (does it do the 'right' thing based on what it knows?)

I will cite you definitions that fit into each of the above categories:

  1.I. "The [automation of] activities that we associate with human thinking, activities such as decision making, problem solving, learning.." - Bellman 1978
  1.II. "The study of the computations that make it possible to perceive, reason, and act." - Winston, 1992
  2.I. "The study of how to make computers do things at which, at the moment, people do better" - Rich and Knight, 1991
  2.II. "The study of the design of intelligent agents" - Poole et al., 1998

In summary, AI is devoted to the creation of intelligent and rational machines that can make rational decisions and take rational actions.

I would suggest you read up on the Turing test, which Alan Turing proposed to test if a computer was intelligent. However, the Turing test has a few issues, because it is anthropomorphic.

When Aeronautical engineers created the airplane, they didn't set their goal that planes should fly exactly like birds, but rather, they started learning how lift forces were generated, based on the study of aerodynamics. Using this knowledge, they created planes.

Similarly, people in the AI world shouldn't put, IMHO, human intelligence as the standard to strive for, but, rather, we could use, say, rationality as a standard (amongst others).


Is learning possible without random thoughts and actions? In my view intelligence begins once the thoughts/actions are logical rather than purely randomn based. The learning environments can be random but the logic seems to obey some elusive rules. There is also the aspect of a parenting that guides through some really bad decisions by using the collective knowledge. All of this seems to hint that intelligence needs intelligence to coexist and a sharing communication network for validation/rejection.

Personally I believe that we must keep the human intelligence in a parental role for long enough time until at least the AI had fully assimilated our values. The actual danger is to leave the artificial intelligence parenting another AI and loose control of it. This step is not necessary from our perspective but can we resist the temptation and try it eventually, only time will tell.

Above all we must remember the purpose of AI. I think the purpose should always be to help humans achieve mastery of the environment while ensuring our collective preservation.

AI should not be left unsupervised as we would not give guns to kids, do we?

To resume it all AI needs an environment and supervision where to learn and grow. The environment can vary but the supervision must stay in place.

Are initiated thoughts/actions by the means of guidance and supervision considered random?

Lastly I believe that the sensible think to do is to only develop artificial intelligence that is limited by our own beliefs and values rather than searching for something greater than us.

It seems not possible to create greater than our intelligence without letting it go exploring!
Exploring has greater access to random actions and can go against the intended purpose.

Learning is possible without random thoughts and actions. Knowledge can be encapsulated in predetermined forms and passed through predetermined knowledge transfer mechanisms. Much of civilization is based on these predeterminations. Without them, humanity would be thrown back possibly 120,000 years.

However, initial discovery requires trials and review of their outcome. Purely deterministic identifications of trials is necessarily systematic and the system used may interplay with the phenomena under study in such a way as to miss important cases. Furthermore, when the complexity of the phenomenon is high, the number of trials is often too numerous to check entirely. In this second scenario, random selection of trials is wise for a similar reason illustrated by this simple example.

The phenomenon has one behavior for even numbers and another for odd. The system for determining trials is to check every factor of 100 to cover the range from 1 to 10,000 in 100 trials. The odd behavior will be inadvertently overlooked.

  Intelligence begins once the thoughts/actions are logical rather than purely random based.

For the above reasons, intelligence begins with logic only in determining the domain of trials but, when one of the above two cases apply, is often quickly followed by lack of logic in the selection from that domain. Once models have formed as a result of these initial discovery activities, logical inference is useful again, to combine them in various ways. Through this process, engineering, business, resource planning, and other intelligence related disciplines have improved the living conditions for the species, albeit inconsistently.

So there are two limitations of logic.

Piercingly deep searches require some temporary dismissal of logic.
Logic applied by large populations produces inconsistently logical results.

It is naive to assume that there were logical faults that produces the inconsistency in this second limitation. There is no logical proof that logic necessarily improves conditions logically across the population that uses it. There may be qualities of the goal sets of many individuals that thwart the vision of logic producing peace and prosperity for all. That belief is not new and no person or political group has been able to make it work.

  Intelligence needs intelligence to coexist and a sharing communication network for validation/rejection. ... I believe that we must keep the human intelligence in a parental role for long enough time until at least the AI had fully assimilated our values.

That assumes that those values are best. Some might want to agree on those values before letting the AI assimilate them, which, given the insanity evident in human history, could lead to a war. If we give the AI the below four objectives, we could probably live without the remainder of human values.

Survival of the biosphere
Human freedom of expression
Evenly distributed, high prosperity across the human population
Encouragement of the social rule, "Be with others how you would want them to be with you," between humans, between AI robots, and between AI robots and humans.

This statement dismisses the human record.

  The actual danger is to leave the artificial intelligence parenting another AI and loose control of it. ... the purpose should always be to help humans achieve mastery of the environment while ensuring our collective preservation. ... AI should not be left unsupervised as we would not give guns to kids, do we? ... To resume it all AI needs an environment and supervision where to learn and grow. The environment can vary but the supervision must stay in place.

It is impossible to supervise all adults and it will later become impossible to supervise all AI in data centers and robots. There are already more processes running in computers than there are people in the world. This is why risk management must be applied proactively, during the research and development of the AI. Some of the questions on this site address that very challenge. If it is not done proactively, it will become untenable and control will already have been passed for a random future regarding our species.

  Only develop artificial intelligence that is limited by our own beliefs and values rather than searching for something greater than us.

There is no universal set of beliefs among humans. There will inescapably be a wide range in AI developments with a wide range of intended purposes. The key is to encourage researchers to think beyond their obsession with getting programs to do amusing and impressive things. AI research must continue to keep one foot in technology but another foot firmly placed in the practical elements of ethics, social science, environmental science, economics, and risk management.


Building 'evaluation' neural networks for go, reversi, checkers etc, how to train? I'm trying to build neural networks for games like Go, Reversi, Othello, Checkers, or even tic-tac-toe, not by calculating a move, but by making them evaluate a position.

The input is any board situation. The output is a score or estimate for the probability to win, or how favorable a given position is. Where 1 = guaranteed to win and 0 = guaranteed to lose.

In any given turn I can then loop over all possible moves for the current player, evaluate the resulting game situations, and pick the one with the highest score.

Hopefully, by letting this neural network play a trillion games vs itself, it can develop a sensible scoring function resulting in strong play.

Question: How do I train such a network ?

In every game, I can keep evaluating and making moves back and forth until one of the AI players wins. In that case the last game situation (right before the winning move) for the winning player should have a target value of 1, and the opposite situation (for the losing player) has a target value of 0.

Note that I don't intend to make the evaluation network two-sided. I encode the game situation as always "my own" pieces vs "the opponent", and then evaluate a score from my own (i.e. the current player's) side or perspective. Then after I pick a move, I flip sides so to speak, so the opponent pieces now become my own and vice versa, and then evaluate scores again (now from the other side's perspective) for the next counter move.

So the input to such a network does explicitly encode black and white pieces, or naughts and crosses (in case of tic-tac-toe) but just my pieces vs their pieces. And then evaluates how favorable the given game situation is for me, always assuming it's my turn.

I can obviously assign a desired score or truth value for the last move in the game (1 for win, 0 for loss) but how do I backpropagate that towards earlier situations in a played game?

Should I somehow distribute the 1 or 0 result back a few steps, with a decaying adjustment factor or learning rate? In a game with 40 turns, it might make sense to consider the last few situations as good or bad (being close to winning or losing) but I guess that shouldn't reflect all the way back to the first few moves in the game.

Or am I completely mistaken with this approach and is this not how it's supposed to be done?

The evaluation of the last steps in the game can be made with the 1 and 0 as you said. For all the other steps, the evaluation should be the evaluation of the best next step with a small decay.


Can predictions of a neural network using ReLU activation be non-linear (i.e. follow the pattern) outside of the scope of trained data? Training on a quadratic function
x = np.linspace(-10, 10, num=1000)
np.random.shuffle(x)
y = x**2

Will predict an expected quadratic curve between -10 < x < 10.

Unfortunately my model's predictions become linear outside of the trained dataset.
See -100 < x < 100 below:

Here is how I define my model:
model = keras.Sequential([
      layers.Dense(64, activation='relu'),
      layers.Dense(64, activation='relu'),
      layers.Dense(1)
  ])

model.compile(loss='mean_absolute_error', optimizer=tf.keras.optimizers.Adam(0.1))

history = model.fit(
    x, y,
    validation_split=0.2,
    verbose=0, epochs=100)

Here's a link to a google colab for more context.

It isn't too surprising to see behaviour like this, since you're using $\mathrm{ReLU}$ activation.
Here is a simple result which explains the phenomenon for a single-layer neural network. I don't have much time so I haven't checked whether this would extend reasonably to multiple layers; I believe it probably will.
Proposition. In a single-layer neural network with $n$ hidden neurons using $\mathrm{ReLU}$ activation, with one input and output node, the output is linear outside of the region $[A, B]$ for some $A < B \in \mathbb{R}$. In other words, if $x > B$, $f(x) = \alpha x + \beta$ for some constants $\alpha$ and $\beta$, and if $x < A$, $f(x) = \gamma x + \delta$ for some constants $\gamma$ and $\delta$.
Proof. I can write the neural network as a function $f \colon \mathbb R \to \mathbb R$, defined by
$$f(x) = \sum_{i = 1}^n \left[\sigma_i\max(0, w_i x + b_i)\right] + c.$$
Note that each neuron switches from being $0$ to a linear function, or vice versa, when $w_i x + b_i = 0$. Define $r_i = -\frac{b_i}{w_i}$. Then, I can set $B = \max_i r_i$ and $A = \min_i r_i$. If $x > B$, each neuron will either be $0$ or linear, so $f$ is just a sum of linear functions, i.e. linear with constant gradient. The same applies if $x < A$.
Hence, $f$ is a linear function with constant gradient if $x < A$ or $x > B$.
$\square$
If the result isn't clear, here's an illustration of the idea:

This is a $3$-neuron network, and I've marked the points I denote $r_i$ by the black arrows. Before the first arrow and after the last arrow, the function is just a line with constant gradient: that's what you're seeing, and what the proposition justifies.