Why does potential-based reward shaping seem to alter the optimal policy in this case? It is known that every potential function won't alter the optimal policy [1]. I lack of understanding why is that.
The definition:
$$R' = R + F,$$ with $$F = \gamma\Phi(s') - \Phi(s),$$
where, let's suppose, $\gamma = 0.9$.
If I have the following setup:

on the left is my $R$.
on the right my potential function $\Phi(s)$
the top left is the start state, the top right is the goal state

The reward for the red route is: $(0 + (0.9 * 100 - 0)) + (1 + (0.9 * 0 - 100)) = -9$.
And the reward for the blue route is: $(-1 + 0) + (1 + 0) = 0$.
So, for me, it seems like the blue route is better than the optimal red route and thus the optimal policy changed. Do I have erroneous thoughts here?

The same $\gamma = 0.9$ that you use in the definition $F \doteq \gamma \Phi(s') - \Phi(s)$ should also be used as the discount factor in computing returns for multi-step trajectories. So, rather than simply adding up all the rewards for your different time-steps for the different trajectories, you should discount them by $\gamma$ for every time step that expires.

Therefore, the returns of the blue route are:

$$0 + (0.9 \times -1) + (0.9^2 \times 0) + (0.9^3 \times 1) = -0.9 + 0.729 = -0.171,$$

and the returns of the red route are:

$$(0 + 0.9 \times 100 - 0) + 0.9 \times (1 + 0.9 \times 0 - 100) = 90 - 89.1 = 0.9.$$


What would be the result of an AI that does not find any pattern according its available methods of pattern recognition? What would be the result of an AI subject to a data stream or information stream, fed to it from events in an "external world", if it could not find any pattern within that stream by the methods it has in determining patterns?
Would the response be because of the limitation of its methods in determining patterns or would you be able to conclude that the information from those events are undetermined and therefore could be stated as random?
Or would the AI be unable to ever determine using its methods whether the information fed to it was from a random undetermined world because it needed an eternity of information to determine whether the data was random or not?
So what I guess I'm asking is the pursuit of creating AI by mankind on a similar level of limitation by the accumulation of "Knowledge" over time as is the other pursuits of mankind within science etc, and is only going to be achieved simultaneously with the acquisition of "knowledge" from these other pursuits. If the determination of a simulated model comes from the physics of a determined world or not will there ever be a model because of the eternity of determining whether the information from the "external world" is determined or not.

During my master thesis I struggled with this issue.

In strict sense, a real AI would be able to say, my sensors (or some of them) are gathering data i cannot understand (yet), and would ideally have a routine what to do whith such cases and ideally one day, even understand that data. But it would most probably now, if that is happening outside in the real world, or if it's due to damaged or wrong sensors.
But that broadly depends how it was programmed, so your use case would have to be further defined: what specific AI are we talking about, what can it do, what not, how is it built.

AI is actually able to reproduce
a) a subset of an adult's intelligence (speach, face recognition, etc.) but just on a case by case basis
b) or, if you seak to build a full grown intelligence, it has to go thorugh the development stages as humans (eyesight, voice, imitation, learning, etc.)
So AI is still in babysteps and is not even able to reproduce let's say an adolescent behaviour, or to reproduce the human evolution of a baby to an adolescent.

So, after a long speach, to come back to your original question: At the very moment an AI is not able to decide by itself, if it has no method to analize the data, if the data analized are too random, or if one of it's sensors is giving random input.
Perhaps in some years there are better methods, but at the moment, machines, as babys or childs, need adults hands to teach and show them what to do in what case and how to difference each use-case. Which often means that they would need to add new modules to the AI, which, nowadays is still far away from building anything we could really define as "intelligent" or even "artificially intelligent" (PD: Intelligence here means by ways much more than plain rationality)

I deeply recommend some books of Andy Clark concerning filosofical issues and AI:
e.g. "Being there: Putting Brain, Body and World together". Andy Clark is a filosofer which takes a deeper look at AI, the human brain, cognition sciences, filosofy, psychology, sociology, etc.


What does it mean for a neuron in a neural network to be activated? I just stumbled upon the concept of neuron coverage, which is the ratio of activated neurons and total neurons in a neural network. But what does it mean for a neuron to be "activated"? I know what activation functions are, but what does being activated mean e.g. in the case of a ReLU or a sigmoid function?

A neuron is said activated when its output is more than a threshold, generally 0.

For examples :
\begin{equation}
y = Relu(a) > 0
\end{equation}
when
\begin{equation}
a = w^Tx+b > 0
\end{equation}

Same goes for sigmoid or other activation functions.


I need to predict ball position from set of Images I am a novice developer in AI. Any help appropriated.

I have a set of images and from that I want to predict position(x,y co-ordinates) of the Ball.

Thanks in advance.

There are so many challenges here for a novice or even a very seasoned AI developer!  There hasn't been a lot of development in this area.  I assume you are working with broadcast video.  Your biggest challenges will involve occlusions of the ball and changes in the view as broadcasters constantly switch cameras which have very different positions on the field or around the stadium.  You will need to track the ball and the players that take action on the ball.  The ball can be in motion from a player heading, kicking, or throwing the ball.  The ball can also be in motion from a bounce off a player or the ground.  To get started I recommend building off some work that at least can track the players and the ball and can identify certain actions such as a kick.  If you start there you can build some type of ball trajectory predictor.

The following paper Soccer Event Detection, by Abdullah Khan1, Beatrice Lazzerini, Gaetano Calabrese and Luciano Serafini, presents an event detector for "Ball possession" and "Kicking the ball".

A paper entitled The Newton Scheme for Deep Learning by Junqing Qiu, Guoren Zhong, Yihua Lu, Kun Xin, Huihuan Qian, and Xi Zhu, addresses predicting the trajectory of a ball in sports.  They address the complexities of a ball in motion.  They address situations where "the ball is flying with a self-spinning, resulting in a Magnus force on it, which makes it difficult to predict the position by traditional methods".

Some related articles and papers you might find of interest:

Identifying Player Possession in Spatio-Temporal Data
Deep Learning using CNNs for Ball-by-Ball Outcome Classification in Sports
Detecting events and key actors in multi-person videos
Learning to Track and Identify Players from Broadcast Sports Videos
Video Artificial Intelligence Powers Creation
AI predicts how athletes will react in certain situations
How AI-Based Sports Analytics Is Changing the Game


Adding a dense layer after a conv2d layer in a convolutional autoencoder I am trying to implement a convolutional autoencoder with a dense layer at the bottleneck do to some dimensional reduction. I have seen two approaches for this which arent particularly scalable. The first was to introduce 2 dense layers (one at the bottle neck and one before & after that has the same number of nodes as the conv2d layer that precedes the dense layer in the encoder section:

input_image_shape=(200,200,3)
encoding_dims = 20

encoder = Sequential()
encoder.add(InputLayer(input_image_shape))
encoder.add(Conv2D(32, (3,3), activation="relu, padding="same"))
encoder.add(MaxPooling2D((2), padding="same"))
encoder.add(Flatten())
encoder.add(Dense(32*100*100, activation="relu"))
encoder.add(Dense(encoding_dims, activation="relu"))

#The decoder
decoder = Sequential()
decoder.add(InputLayer((encoding_dims,)))
decoder.add(Dense(32*100*100, activation="relu"))
decoder.add(Reshape((100, 100, 32)))
decoder.add(UpSampling2D(2))
decoder.add(Conv2D(3, (3,3), activation="sigmoid", padding="same"))

Its easy to see why this approach blows up as there are two densely connected layers with (32*100*100) nodes each or more or in that ballpark which is nuts.

Another approach I have found which makes sense for b/w images such as the mnist stuff is to introduce an arbitrary number of encoding dimensions and reshape it (https://medium.com/analytics-vidhya/building-a-convolutional-autoencoder-using-keras-using-conv2dtranspose-ca403c8d144e). The following chunk of code is copied from the link, I claim no credit for it:

#ENCODER
inp = Input((28, 28,1))
e = Conv2D(32, (3, 3), activation='relu')(inp)
e = MaxPooling2D((2, 2))(e)
e = Conv2D(64, (3, 3), activation='relu')(e)
e = MaxPooling2D((2, 2))(e)
e = Conv2D(64, (3, 3), activation='relu')(e)
l = Flatten()(e)
l = Dense(49, activation='softmax')(l)
#DECODER
d = Reshape((7,7,1))(l)
d = Conv2DTranspose(64,(3, 3), strides=2, activation='relu', padding='same')(d)
d = BatchNormalization()(d)
d = Conv2DTranspose(64,(3, 3), strides=2, activation='relu', padding='same')(d)
d = BatchNormalization()(d)
d = Conv2DTranspose(32,(3, 3), activation='relu', padding='same')(d)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(d)

The question is then is there a more rigorous way/ accepted of adding a conv2d layer after a 2d layer?

Hi for me this worked perfectly. I encoded with conv2d and dense and then I flatten I and reshape in the decoder after the dense layer so the encoder and decoder are symmetrical. The only difference is that in my case I use images (224,224,1)
        # create encoder
        # 28,28 -> 1st conv2d (filter 3x3,relu activation, padding, strides == 'jumps')
        self.encoder = tf.keras.Sequential([layers.Input(shape=(224,224,1)),
                                            layers.Conv2D(16,kernel_size=3,activation='relu',padding='same',strides=2),
                                            layers.Conv2D(8,kernel_size=3,activation='relu',padding='same',strides=2),
                                            layers.Conv2D(4,kernel_size=3,activation='relu',padding='same',strides=2),
                                            layers.Flatten(),
                                            layers.Dense(units=3136,activation='sigmoid')]) # (28,28,4)

        # deconvolution -> decoding
        self.decoder = tf.keras.Sequential([layers.Input(shape=(3136)),
                                            layers.Dense(units=3136,activation='sigmoid'),
                                            layers.Reshape((28,28,4)),
                                            layers.Conv2DTranspose(4,kernel_size=3,activation='relu',padding='same',strides=2),
                                            layers.Conv2DTranspose(8,kernel_size=3,strides=2,activation='relu',padding='same'),
                                            layers.Conv2DTranspose(16,kernel_size=3,strides=2,activation='relu',padding='same'),
                                            layers.Conv2D(1,kernel_size=(3,3),activation='sigmoid',padding='same')])


What are the differences between uniform-cost search and greedy best-first search? What are the differences between the uniform-cost search (UCS) and greedy best-first search (GBFS) algorithms? How would you convert a UCS into a GBFS?

In the case of UCS, the evaluation function (that is, the function that is used to select the next node to expand) is $f(n) = g(n)$, where $g(n)$ is the cost of the path from the initial node to $n$, while in the case of the greedy BFS it is $f(n) = h(n)$, where $h(n)$ is the heuristic function that estimates the cost of the path from $n$ to the goal node. In other words, in the case of UCS, nodes are expanded only using the experience (in the form of $g(n)$), while, in the case of GBFS, nodes are expanded only using the estimate of the cost to the goal. Note that, in both cases, the node that is chosen to be expanded is the one with the smallest $f(n)$ value.


If we achieve sentience using mutable hardware, will it be possible to make a copy of that "brain" and its active state? Hardware comes in two forms, basically: immutable, such as RAM, and mutable, such as FPGAs.

In animals, neurological connections gain in strength by changing the physical structure of the brain. This is analogous to FPGAs whereby signal strength is increased by changing the pathways themselves.

If we achieve sentience using mutable hardware (e.g., neuromemristive systems), will it be possible to make a copy of that "brain" and its active state?

For this question, assume that the brain is how the hardware has "reconfigured" [or etched, if you will] its pathways to strengthen them and the brain's state is captured by how electrons are physically flowing throughout those pathways.

Theoretically, there shouldn't be a problem copying either of the artificial brains in any state.

Difficulty in measuring a state doesn't seem to really be a problem until you get down to the quantum level, where the means of measurement affect the state.

The configuration of the artificial brains, including pathway structures and states, should be reducible to a single string, which could then be used to reconfigure the artificial brain the information is being copied to.

Definitely look into the concept of a Turing Machine and Universal Turing Machine.


Is there a good ratio between the positive and negative rewards in reinforcement learning? Is there an ideal ratio in reinforcement learning between the positive and negative rewards?

Suppose I have the scenario of moving a robot across the river. There are two options, walk across the bridge or walk across the river. If it walks across the river then the robot breaks so the idea is to reinforce the robot to walk across the bridge. What would be the best rewards values? Does this ratio vary between cases?

option1:

Bridge: +10
River: -10

Option2:

Bridge: +10
River: -1

Option3:

Bridge: +1
River: -10

It usually does not matter, but I'm sure there are situations where it could matter.  In theory, if a reward for good behavior is higher than the rewards for bad behavior, then the neural network will be trained such that the higher rewards are preferred, even if those higher rewards are negative.  For example, if a bad reward is -100, then a relatively good reward could be -50, and the network will then be more likely to choose the action with -50 reward over the action with -100 reward.


How are the "Link Strength true", "Link Strength blind" and "Mutual Information" calculated in this report on Bayesian networks? I'm trying to understand how to calculate the strength of every arc in a Bayesian Network.
I came across this report Measuring Connection Strengths and Link Strengths in Discrete Bayesian Networks, but I got lost in the calculation.
In particular, how are the values of Link Strength true, Link Strength blind, and Mutual Information computed in Table 1?

The $MI$ is easy enough.
For $MI(X \to Z)$, we get $U([0.5, 0.5]) - 0.5 U([0.9, 0.1]) - 0.5 U([0.1, 0.9])$. The $[0.5, 0.5]$ comes from $0.5 [0.9, 0.1] + 0.5[0.1, 0.9]$ -- this is how to calculate $P(Z)$ from $P(Z|X)$ and $P(X)$.
For $MI(X \to Y)$, we need to marginalize out $Z$. This takes a bit of work, but is not too hard. I manually verified all of the reported values in the figure, and they are all correct.
The $\mathrm{LS}{\mathrm{true}}$ values are a bit trickier, but still feasible. I managed to get the same values. It is important when calculating $\mathrm{LS}{\mathrm{true}}(X \to Z)$ that $Y$ is not a parent, so it does not come into it -- which is why the MI gives exactly the same value.
The "blind" version took me a bit more effort to implement, but again, I am getting the same values as reported in the table.
I used the "simple formula" at the top of page 5.


Why does the adversarial search minimax algorithm use Depth-First Search (DFS) instead of Breadth-First Search (BFS)? I understand that the actual algorithm calls for using Depth-First Search, but is there a functionality reason for using it over another search algorithm like Breadth-First Search?

The primary reason is that Breadth-First Search requires much more memory (and this probably also makes it a little bit slower in practice, due to time required to allocate memory, jumping around in memory rather than working with what's still in the CPU's caches, etc.). Breadth-First Search needs memory to remember "where it was" in all the different branches, whereas Depth-First Search completes an entire path first before recursing back -- which doesn't really require any memory other than the stack trace. This is assuming we're using a recursive implementation for DFS -- which we normally do in the case of minimax.

You can clearly see this if you look at pseudocode for the two approaches (ignoring the minimax details here, just presenting pseudocode for straightforward searches):

BreadthFirstSearch(start):
    Q = new queue()
    Q.append(start)
    while Q is not empty:
        node = Q.pop()
        if node is leaf:
            do something with leaf
        else:
            for each child of node:
                Q.append(child)

DepthFirstSearch(start):
    if start is leaf:
        do something with leaf
    for each child of start:
        DepthFirstSearch(child)
        // probably do something with return value from the recursive DFS call

You see that the BFS requires a queue object that explicitly stores a bunch of stuff in memory, whereas DFS doesn't.

There's more to the story once you get to extensions of Minimax, like Alpha-Beta pruning and Iterative Deepening... but since the question is just about Minimax, I'll leave it at that for now.


How to draw bounding boxes for gender classification? I wonder what is the better way of drawing rectangles on images for gender classification. My task is to create a classifier (CNN based) to detect gender from pictures of entire bodies (not just faces). When I started labeling pictures I noticed that I am not sure whether I should draw it around an entire person like example 1 (including hands and legs and some background space between them) or just the inner part like example 2 (where there is almost no background), in order to achieve better results?

Example 1

Example 2

In state-of-the-art papers. researcher managed to achieve 90-100% accuracy on gender classification on just human faces, so both method may work just fine, with minor improvements maybe using the first method. However using the image of the entire body enhances overfitting as there is more input features and increases overfitting. For a very large dataset it might be better to use the first method as chances of overfitting are low while for a smaller one you should probably go for the second method or even just the face as that won't make such a huge differnce.

A better way to input the data to the network is through landmarks of a body.

Inputting the landmarks of the body helps reduce the features to the minimal and also keeps only the relevant features. The main difference in body between women and men is perhaps the torso and shoulder width, which you can get from body landmark detection. You can input both the image of the face and the body landmark as input to the network, and it would probably increase in accuracy. However this solution is not an one stage method, meaning it requires an extra model to predict the landmark.

Hope I can help you.


Normalization of possibly not fully representative data I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:

outliers, but I could fight them or use RobustScaler instead.
I am not sure that some features in my dataset include all possible ranges. Like I have max(feature_A) == 10, but with the data update, it could become 20. And if I'll preprocess data the same way I will get bad prediction results.
Some features do not have a limit at all and will only increase with time, like how many years patients were treated, for example. I could suppose that this value is !>100years, for example, but if my mean value is 10 years, it will squeeze feature values a lot.

My dataset is pretty large, like millions of observations, thus there is a pretty good chance that it is representative, though. But I am concerned with the small-time range, like all those observations are for the 2 years only, thus, some feature values (like how many years patients were treated) could still grow their bounds.
How should I handle this?
My concerns example:
import pandas as pd
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

#### like, initial state
df1 = pd.DataFrame({'A': [1, 2, 3, 4, 5], 'B': [10, 40, 60, 80, 100]})
""" output:
   A    B
0  1   10
1  2   40
2  3   60
3  4   80
4  5  100
"""

scaler.fit_transform(df1)
""" output:
array([[0.        , 0.        ],
       [0.25      , 0.33333333],
       [0.5       , 0.55555556],
       [0.75      , 0.77777778],
       [1.        , 1.        ]])
"""

#### new data arrived, while preprocessing is the same
df2 = pd.DataFrame({'A': [1, 2, 3, 4, 5, 10, 10], 'B': [10, 40, 60, 80, 100, 120, 140]})
""" output:
    A    B
0   1   10
1   2   40
2   3   60
3   4   80
4   5  100
5  10  120
6  10  140
"""

# now 5 in "A" scaled to 0.4 instead of 1, same in "B"
scaler.fit_transform(df2)
""" output:
array([[0.        , 0.        ],
       [0.11111111, 0.23076923],
       [0.22222222, 0.38461538],
       [0.33333333, 0.53846154],
       [0.44444444, 0.69230769],
       [1.        , 0.84615385],
       [1.        , 1.        ]])
"""

PS: I've duplicated this question in different communities (question in ai got most of views):

https://stats.stackexchange.com/questions/523097/normalization-of-possibly-not-fully-representative-data
https://datascience.stackexchange.com/questions/94095/normalization-of-possibly-not-fully-representative-data

A friend of mine answered on this question in different social media on different language, I'll post his answer here:

1. scaler should be saved in this case.
You do fit_transform in the example the second time you run it, but you should just transform. Scaler should be "fit" once on data train and not change it afterwards. Then you will get 5 in 1 in both cases. And when 10 appears, scaler will convert it to 2.
2. It's not a fact that this will break the model.
I think everything is clear here, but just in case:
Let us consider the simplest case. Suppose there is a linear dependence y = 3 * x, we don't know about it, but there is a dataset:
X Y
1 3
2 6
3 9

If we assume that the model has the form y = a * x, in the process of learning it turns out that
a = 3 gives the best score and when suddenly x = 100 appears, it has no problem mapping to 300, even though 100 is strongly out of dataset (1,2,3).
3. You can check if this breaks the model now.
Suppose there is a chip that takes values 0, 1, 2, 3, 4, 5, 6, 7, 8 ...
You are afraid that after some time samples with values greater than the current maximum (9, 10, 11, ...) will appear and the model will behave strangely on them. You can try to "simulate" this situation.
Split dataset into train and test so samples with values from 0 to 6 will go into train, and samples with values from 7 to 8 will go into test. If accuracy doesn't fail at the test stage, then we can assume that when all the 0-8 ranks get into train, then the appearance of 9-11 won't break anything.

Also, there is answer for the second question: https://stackoverflow.com/questions/46744076/machine-learning-normalizing-features-with-no-theoretical-maximum-value


How can I find words in a string that are related to a given word, then associate a sentiment to that found word? I came up with an NLP-related problem where I have a list of words and a string. My goal is to find any word in the list of words that is related to the given string.
Here is an example.
Suppose a word from the list is healthy. If the string has any of the following words: healthy, healthier, healthiest, not healthy, more healthy, zero healthy, etc., it will be extracted from the string.
Also, I want to judge whether the extracted word/s is/are bearing positive/negative sentiment.
Let me further explain what I mean by using the previous example.
Our word was healthy. So, for instance, if the word found in the string was healthier, then we can say it is bearing positive sentiment with respect to the word healthy. If we find the word not healthy, it is negative with respect to the word healthy.

There are many ways to solve this problem.  One way is to apply stemming or lemmatization to reduce your words. Using NLTK's Porter stemmer for example on healthy, healthier, healthiest, not healthy, more healthy, and zero healthy gives:
healthi , healthier , healthiest , not healthi , more healthi , zero healthi

This can help make word comparisons easier.
Sentiment analysis on the phrases will provide positive, neutral, and negative scores.  There are a lot of algorithms for doing this but a common one is Valence Aware Dictionary and sEntiment Reasoner (VADER).  Here is a recent article with code using NLTK and the VADER lexicon:

How To Run Sentiment Analysis In Python Using VADER by George Pipis

The following article also does sentiment analysis using NLTK and includes stemming and lemmatization.  Instead of VADER they use a Naive Bayes classifier on a labeled data set of tweets: How To Perform Sentiment Analysis in Python 3 Using the Natural Language Toolkit (NLTK) by Saumik Daityari.


Is it beneficial to represent a neural net as a matrix? A neural network is a directed weighted graph. These can be represented by a (sparse) matrix. Doing so can expose some elegant properties of the network.

Is this technique beneficial for examining neural networks?

It depends on the type of neural networks you are dealing with.

For medium sized neural nets, the matrix approach is a very good way to do quick computations and even backpropogation of errors. One can even exploit sparse matrixes for understanding the sparse architecture of some neural nets.

But, for very large neural nets, using matrix computations would be computationally very intensive. So, relevant methods like graph-based stores, etc are used for them depending on the purpose and the architecture.


How to normalize images before training? I have seen people normalize images by just dividing 255. But why? Why not use mean normalization or Z-score Normalization? I also came across this StackOverflow topic while searching but the answers there were not enough enlightening for me.

tl;dr subtracting the mean and dividing by the standard deviation is theoretically more sound, but is impractical compared to dividing by $255$.

Explanation
As you know neural networks perform better when their input is scaled. The 2 most common ways to perform scaling are:

normalization, where you want to scale the image to the $[0, 1]$ range.
standardization, where you want to bring the mean to $0$ and the standard deviation to $1$.

Theoretically, I think, the latter has some advantages, however in practice there is not a significant difference, especially if the network has normalization layers.
Images are usually stored in 8-bit color mode, meaning that they take integer values from 0 to 255. Because you know the minimum, and maximum values that each pixel can take, it is extremely easy to normalize an image (i.e. you simply have to divide each pixel value by $255$).
On the other hand it is much less practical to compute the mean and standard deviation of the pixel intensities of the training set. Keep in mind that in most cases the training set doesn't even fit into memory, and you have to approximate the true mean/std by moving averages (i.e. load a few images, compute their mean/std, close them, load some more, add their mean to the MA, ...). But why bother if you can similar results by simply dividing each image by $255$.


Is there a way to understand neural networks without using the concept of brain? Is there a way to understand, for instance, a multi-layered perceptron without hand-waving about them being similar to brains etc?

For example: it is obvious that what a perceptron does is approximating a function; there might be many other ways, given a labelled dataset, to find the separation of the input area into smaller areas that correspond to the labels; however, these ways would probably be computationally rather ineffective, which is why they cannot be practically used. However, it seems that the iterative approach of finding such areas of separation may give a huge speed-up in many cases; then, natural questions arise why this speed-up may be possible, how it happens and in which cases.

One could be sure that this question was investigated. If anyone could shed any light on the history of this question, I would be very grateful. So, why are neural networks useful and what do they do? I mean, from the practical and mathematical standpoint, without relying on the concept of "brain" or "neurons" which can explain nothing at all.

tl;dr I always like to think of Neural Networks as a generalization of logistic regression.

I too don't like that, traditionally, when introducing Neural Networks, books start with biological neurons and synapses, etc. I think its more beneficial to start from statistics and linear regression, then logistic regression and then neural networks.

A perceptron is essentially a simple binary logistic regressor (if you threshold the output). If you have many perceptrons that share the same input (i.e. a layer in a neural network), you can think of it as a multi-class logistic regressor. Now, by stacking one such layer after an other, you create a Multi-Layer Perceptron (MLP), which is a Neural Network with two layers. There is equivalent to two multi-class logistic regressors stacked one after the other. One notable thing that changes is the training technique here, i.e. backpropagation (because you don't have direct access to the targets from the hidden layer). Another thing that can change is the activation function (it's not always sigmoid in Neural Networks)

Introduce sparse connectivity and weight sharing and you get a Convolutional Neural Network. Add a connection from a layer to its self (for the next timestep) and you get a Recurrent Neural Network. Likewise, you can reproduce any Neural Network through this reasoning.

I know this is an over-simplified way of presenting them, but I think you get the point.


What is the reason for mode collapse in GAN as opposed to WGAN? In this article I am reading:

$D_{KL}$ gives us inifity when two distributions are disjoint. The value of $D_{JS}$ has sudden jump, not differentiable at $\theta=0$. Only Wasserstein metric provides a smooth measure, which is super helpful for a stable learning process using gradient descents.

Why is this important for a stable learning process? I have also the feeling this is also the reason for mode collapse in GANs, but I am not sure.
The Wasserstein GAN paper also talks about it obviously, but I think I am missing a point. Does it say JS does not provide a usable gradient? What exactly does that mean?

I don't have a definite answer, but only a suspicion/idea:
Looking at Figure 1 from the WGAN paper, we clear see that the JS divergence on the right is not continuous at $0$, hence not differentiable at $0$. However, the EM plot on the left is continuous also at $0$. You could now argue that we have a kink there, so it should not be differentiable there either, but they might have a different notion of differentiability, I am honestly not sure about it right now.


What do these numbers represent in this picture of a surface? The following image is a screenshot from a video tutorial that illustrates the concept of gradient descent algorithm with a 3D animation.

Do the numbers on the top of the balls pointed out by the red arrows represent the gradient?

It represents the value of the loss function J(x1, x2; θ); the valley has value 0 in the video. You can see that the lowest ball with value 3.13 is on a steep point with a high gradient, so it's not the gradient.


How can the importance sampling ratio be different than zero when the target policy is deterministic? In the book Reinforcement Learning: An Introduction (2nd edition) Sutton and Barto define at page 104 (p. 126 of the pdf), equation (5.3), the importance sampling ratio, $\rho _{t:T-1}$, as follows:
$$\rho _{t:T-1}=\prod_{k=t}^{T-1}\frac{\pi(A_k|S_k)}{b(A_k|S_k)}$$
for a target policy $\pi$ and a behavior policy $b$.
However, on page 103, they state:

The target policy $\pi$ [...] may be deterministic [...].

When $\pi$ is deterministic and greedy it gives $1$ for the greedy action and 0 for all other possible actions.
So, how can the above formula give something else than zero, except for the case where policy $b$ takes a path that $\pi$ would have taken as well? If any selected action of $b$ is different from $\pi$'s choice, then the whole numerator is zero and thus the whole result.

You're correct, when the target policy $\pi$ is deterministic, the importance sampling ratio will be $\geq 1$ along the trajectory where the behaviour policy $b$ happened to have taken the same actions that $\pi$ would have taken, and turns to $0$ as soon as $b$ makes one "mistake" (selects an action that $\pi$ would not have selected).

Before importance sampling is introduced in the book, I believe the only off-policy method you will have seen is one-step $Q$-learning, which can only propagate observations back along exactly one step. With the importance sampling ratio, you can often do a bit better. You're right, there is a risk that it turns to $0$ rather quickly (especially when $\pi$ and $b$ are very different from each other), at which point it essentially "truncates" your trajectory and ignores all subsequent experience... but that still can be better than one-step, there is a chance that the ratio will remain $1$ for at least a few steps. It will occasionally still only permit $1$-step returns, but also sometimes $2$-step returns, sometimes $3$-step returns, etc., which is often better than only having $1$-step returns.

Whenever the importance sampling ratio is not $0$, it can also give more emphasis to the observations resulting from trajectories that would be common under $\pi$, but are uncommon under $b$. Such trajectories will have a ratio $> 1$. Emphasizing such trajectories more can be beneficial, because they don't get experienced often under $b$, so without the extra emphasis it can be difficult to properly learn what would have happened under $\pi$.

Of course, it is also worth noting that your quote says (emphasis mine):

  The target policy $\pi$ [...] may be deterministic [...]

It says that $\pi$ may be deterministic (and in practice it very often is, because we very often take $\pi$ to be the greedy policy)... but sometimes it won't be. The entire approach using the importance sampling ratio is well-defined also for cases where we choose $\pi$ not to be deterministic. In such situations, we'll often be able to propagate observations over significantly longer trajectories (although there is also a risk of excessive variance and/or numeric instability when $b$ selects actions that are highly unlikely according to $b$, but highly likely according to $\pi$).


Is there any literature on the design of dialogue systems for interviews and questionnaire administration? For my master thesis I am working on a dialogue system that should be deployed in hospitals to administer simple questionnaires to patients. I already did literature research and I'm fine with what I found since I don't have to replicate something which has been already done, but I noticed that there are really few papers regarding this specific 'robot interviewer' topic.

Let me explain the task a bit more in details: in a real interview a human interviewer usually starts with greetings and with an explanation of the questionnaire to administer, and then he/she starts asking some more or less structured questions to the person to interview. The idea here is to replace a human interviewer with a dialogue system.

Now, at a first glimpse it seems like a task that can be easily hand coded, and indeed lot of real application simply use systems in which specific questions are stored in memory in association with some already made answers to choose from (here's an example), and the system simply show them (or read in the case of humanoid robots) to the people being interviewed, the system then wait for the answer and then move on with the next questions.

The point is that in real interviews the conversation flow is obviously much more smooth and natural. A human being can detect doubts in the voice of the interviewed person, which can also explicitly ask for explanations, a human being can also understand when an answer comes with emotional implications ('yes I feel sad every day') and we are able to automatically react to these hidden implications with emotional fillers ('I'm sorry to hear that'). All these aspects require to train some natural language understanding module in order to be replicated in an artificial agent (and this is actually what I'm currently working on), so I though I would have found more papers on this.

Now, despite having found a tons of paper related to open domain dialogue systems, affective and attentive systems, even systems able to reply with irony, I did not found many papers about dialogue systems for smooth interviews or questionnaire administration, which in my opinion sounds like an much easier task to tackle (especially if compared to open domain conversations).
The only two papers that I found which truly focused on interviewer systems are:

SimSensei Kiosk: A Virtual Human Interviewer for Healthcare Decision Support
A job interview dialogue system with autonomous android ERICA

So my question is: did I miss something, like some specific keywords? Or is there actually a gap in the literature with regard to the design of dialogue systems for interviews and questionnaire (or surveys) administration? I'm interested in any link or hint from anyone working on similar applicants, thank you in advance!

[Disclaimer: I work for a company that provides a platform for developing conversational AI systems]

The platform used by the company I work for has a sentiment analysis component, so you can recognise if the user input expresses certain emotions. The dialogues are encoded in 'flows', which are graphs with an initial trigger consisting of output nodes and transitions that correspond to user input. In order to react to emotions expressed by the user you'd have a trigger that does not react to the user input text, but instead to a setting of the sentiment flags.

For example, in a call centre bot, if the user repeatedly uses angry sentiment in their utterances, the bot could detect this and trigger a flow that says "It seems you are not happy with my responses -- please wait while I transfer you to a human agent".

This all depends on the mechanics of organising the dialogues. If you have a ML model, this will probably not be as easy as if you have a manually designed graph structure (which gives you more control over such matters).

Being a commercial system, there are obviously no academic papers on this. But as it's so easy to do within the platform, it doesn't seem too much of a problem in the first place.


What exactly is the advantage of double DQN over DQN? I started looking into the double DQN (DDQN). Apparently, the difference between DDQN and DQN is that in DDQN we use the main value network for action selection and the target network for outputting the Q values.
However, I don't understand why would this be beneficial, compared to the standard DQN. So, in simple terms, what exactly is the advantage of DDQN over DQN?

In $Q$-learning there is what is known as a maximisation bias. That is because the update target is $r + \gamma \max_a Q(s,a)$. If you slightly overestimate your $Q$-value then this error gets compounded (there is a nice example in the Sutton and Barto book that illustrates this). The idea behind tabular double $Q$-learning is to have two $Q$-networks, $Q_1,Q_2$, and you choose an action $a$ from them, e.g. from $Q_1 + Q_2$. You then flip a coin to decide which to update. If you choose to update $Q_1$ then the update target becomes $r + \gamma Q_2(s', \arg\max_a Q_1(s',a))$.
The idea is that if you overshoot your estimate on one $Q$ network then having the second will hopefully control this bias when you would take the max.
In Deep Double $Q$-learning the idea is essentially the same but instead of having to maintain and train two $Q$-networks, they use the target network from vanilla DQN to provide the target. To make this more concrete, the update target they use is
$$r + \gamma Q(s', \arg\max_aQ(s',a;\theta);\theta^-)\;,$$
where $Q(s,a;\theta^-)$ denotes the target network whose parameters are only updated to the current networks every $C$ time steps.
As before, the idea is that if we have overestimated our value of being state $s'$ in our current network when taking the max action, using the target network to provide the target will help control for this bias.
Maximisation Bias
I will here explain maximisation bias from the simple example given from the Sutton and Barto book.

The Markov Decision Process in the image is defined as follows: we start in state A and can take the 'right' action which gives us 0 reward and immediately leads to termination. If we choose 'left' we get 0 immediate reward where we then move to state B. From there, we have an arbitrary number of action we can take where they all lead to the terminal state and the reward is drawn from a Normal(-0.1,1) distribution.
Clearly, the optimal action is always to move to the right from state A as this gives 0 expected future returns. Taking the left action will give a $\gamma \times -0.1$ expected future returns (the $\gamma$ is our discount factor).
Now, if we got into state $B$ and took some random action our initial reward could be bigger than 0 -- after all it is drawn from a Normal(-0.1,1) distribution.
Now, consider we are updating our $Q$-function for state A and taking the left action. Our update target will be $0 + \gamma \max_a Q(B,a)$. Because we are taking the max over all possible actions, this will lead to a positive reward and so we are backing up the belief of our expected future rewards from taking action left in state A to be something positive -- clearly this is wrong since we know it should be -0.1. This is what is known as the maximisation bias, because it gives us a kind of 'optimistic' estimate of the action value!
I've attached an image below that shows the %age of time the agent chose the left action, which it shouldn't be choosing). As you can see, it takes normal $Q$-learning along time to even start to correct itself, whereas double $Q$-learning corrects the mistake almost immediately.


How to find the optimal number of neurons per layer? When you're writing your algorithm, how do you know how many neurons you need per single layer? Are there any methods for finding the optimal number of them, or is it a rule of thumb?

There is no direct way to find the optimal number of them: people empirically try and see (e.g., using cross-validation). The most common search techniques are random, manual, and grid searches.

There exist more advanced techniques such as Gaussian processes, e.g. Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act Classification, IEEE SLT 2016.


Why do we use a delay when feeding our input data to the echo state network? I'm new to working with neural networks and have recently began implementing neural networks for time series forecasting in some of my work. I've been particularly using Echo State Networks and have been doing some reading to understand how they work. For the most part, things seem pretty straight forward, but I'm confused as to why we use a 'delay' when feeding our input data (the delay concept mentioned in the paper Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication)?

I'm looking at some source code on Github, and they implement this delay as well (they feed two arrays inputData & targetData, into the network where one is delayed by one element relative to the other). I am noticing that the larger the delay, the worse the fit.

Why is this done?
My interest is eventually to forecast past sample data.

It is not a 'delay' but a 'look ahead' as some people call it.  This is how far in the future the model can forecast (or predict).

In reference to the GitHub source, the code uses a look ahead ('delay') of one.  The echo state network learns on the 'inputData' using a future version of itself called 'targetData'.

Here is an analogy.  It is early on Friday, June 5, and I have the last 51 days worth of Amazon stock prices.  I can use the first 50 days of prices, which go up to and include Wednesday, June 3, to train a network to predict Thursday, June 4.  If I have a lot of historical data on Amazon, I can train many 50 day windows.  Each 50 day window would have as its target a stock price one day in the future.

Performance generally degrades the further into the future you try to forecast.  One reason is due to the build up of error over time.  This is true in life.  We usually know fairly accurately what we will do tomorrow, less so next week, and it would be a wild guess to predict what we will be doing in ten years.


Hand computing feed forward and back propagation of neural network I used to treat back propagation as a black box but lately I want to understand more about it. I have used mattmuzr's and DuttA's explanaiton as a guide to hand compute a simple neural network. I have computed feed forward and back propagation to a network similar to this one with one input, one hidden and one output

Here are my computations

Is my computations correct?

*Full latex code

One important point I missed in first review: error is a summatory, its derivative is also a summatory.

About offsets "b": usually they are different in each cell (if not fixed to some value, as 0). Thus, replace them by b1 and b2. Moreover, they should be optimized in the same way that the weights.


Do off-policy policy gradient methods exist? Do off-policy policy gradient methods exist?

I know that policy gradient methods themselves using the policy function for sampling rollouts. But can't we easily have a model for sampling from the environment? If so, I've never seen this done before.

Absolutely, it’s a really interesting problem. Here is a paper detailing off policy actor critic. This is important because this method can also support continuous actions.

The general idea of off-policy algorithms is to compare the actions performed by a behaviour policy (which is actually acting in the world) with the actions the target policy (the policy we want to learn) would have chosen. Using this comparison we can determine a ratio ($0 \leq \rho \leq 1$) which can scale the update to the target policy by the probability of the target policy taking that action. A higher $\rho$, the more alike the two policies are, and this increases the magnitude of the learning update for the target policy for that step. A $\rho$ of $0$, and the update is ignored.


Why do we calculate the mean squared error loss to improve the value approximation in Advantage Actor-Critic Algorithm? class AtariA2C(nn.Module):
    def __init__(self, input_shape, n_actions):
        super(AtariA2C, self).__init__()

        self.conv = nn.Sequential(
            nn.Conv2d(input_shape[0], 32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=4, stride=2),
            nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=3, stride=1),
            nn.ReLU(),
        )

        conv_output_size = self. _get_conv_out(input_shape)

        self.policy = nn.Sequential(
            nn.Linear(conv_output_size, 512),
            nn.ReLU(),
            nn.Linear(512, n_actions),
        )

        self.value = nn.Sequential(
            nn.Linear(conv_output_size, 512),
            nn.ReLU(),
            nn.Linear(512, 1),
        )

    def _get_conv_out(self, shape):
        o = self.conv(T.zeros(1, *shape))
        return int(np.prod(o.shape))

    def forward(self, x):
        x = x.float() / 256
        conv_out = self.conv(x).view(x.size()[0], -1)
        return self.policy(conv_out), self.value(conv_out)

In Maxim Lapan's book Deep Reinforcement Learning Hands-on, after implementing the above network model, it says

  The forward pass through the network returns a tuple of two tensors:
  policy and value. Now we have a large and important function, which
  takes the batch of environment transitions and returns three tensors:
  batch of states, batch of actions taken, and batch of Q-values
  calculated using the formula $$Q(s,a) = \sum_{i=0}^{N-1} \gamma^i r_i
 + \gamma^N V(s_N)$$ This Q_value will be used in two places: to calculate mean squared error (MSE) loss to improve the value
  approximation, in the same way as DQN, and to calculate the advantage
  of the action.

I am very confused about a single thing. How and why do we calculate the mean squared error loss to improve the value approximation in Advantage Actor-Critic Algorithm?

I believe that the author is referring to how the networks are trained in Deep RL. Consider Deep Q-Learning where the $Q(s,a)$ is approximated using a neural network. Then the loss function used to train the network is
$$\mathbb{E}[(r + \gamma \max_{a'} Q(s',a') - Q(s,a))^2]\;.$$
Here, $r + \gamma \max_{a'} Q(s',a')$ is your target, what you want your network to aim towards, and $Q(s,a)$ is what your network predicted. (Note that I have left off some details that can be found in the Nature paper for simplicity).

As for actor-critic methods, most popular actor-critic methods will use the value function to 'replace' the action-value function by using the following relationship:
$$\mathbb{E}[r + \gamma v_\pi(s')] = Q_\pi(s,a)\;.$$
This relationship can be proved by looking at exercise 3.13 (or somewhere around there) in the Sutton and Barto textbook. This looks like what the author is doing in the textbook you are reading.

Based on what I said at the start regarding how state-action value functions are trained, it is analogous to train a critic network that approximates the value function in the same way.

e1: spelling.

e2: added link to nature paper.


Multiple GPUs one expensive GPU, which gpu to buy for real time processing (not training) I am trying to decide what GPU or GPUs to buy to run tf-pose pose detection and yolo3 object detection on several cameras. I need to keep an acceptable frame rate too.

what kind of GPU configuration should I go for?

does having multiple GPUs have an effect on what I am trying to do.

can you please explain the difference between using 2-4 1080ti(any GPU in that price range) cards and using something 10 times more expensive than that such as v100.in training and using the networks both

Here's a link to some benchmarks that should give you some insights.

In my experience (I've used systems with both 1080s and V100s) I've found that as of about a year ago, a lot of the common tools couldn't use the V100s well.  Until we started doing some manual optimization, the 1080s were comparable if not better on common tasks.  Of course, once we put in the time to optimize, recompile drivers, etc., the V100's really came to life and outperformed everything else by several times.


Isolate the speech of two people in an audio record with two people only I would like to find a way to isolate the speech of each of the people in an audio record so I can create a file of that form :

[
   {
       "voice_fingerprint": "701066EDD3A0A40A2F53F61EAFD0E6AB",
       "sentences": {
           {
               "sentence": "do you like red apples",
               "position": 1.39 // Seconds. Time position in the audio record
           },
           {
               "sentence": "and how do you feel about time shifts",
               "position": 7.21
           }
       }
   },
   {
       "voice_fingerprint": "8FFEA051AF3E3FB9A80A51A98FE05896",
       "sentences": {
           {
               "sentence": "yes I do like them",
               "position": 4.54
           },
           {
               "sentence": "i feel well about traveling",
               "position": 10.18
           }
       }
   }
]

This may be an interview record.

The problem IS NOT the Speech to Text, but to isolate the two people's sentences. Preferably in Python.

Have you ever worked on this ? Do you have any hints ?

The task of isolating 2 or more speakers is called speaker diarization,
here a list of softwares and useful resources.

Once you have the 2 or more audio files containing the individual voices, you could run some speech-to-text network that also outputs time information.


How does one create a non-classifying CNN in order to gain information from images? How do I program a neural network such that, when an image is inputted, the output is a numerical value that is not the probability of the image being a certain class? In other words, a CNN that doesn't classify. For example, when an input of an image of a chair is given, the model should not give the chance that the image is a chair but rather give the predicted age of the chair, the predicted price of the chair, etc. I'm currently not sure how to program a neural net like this.

This can be thought of as a loss function design problem. If you optimize your network weights for something like multi-class classification, then expect your network to learn weights for this task (You will use cross entropy loss for this task). If you optimize your network to output a single value at the last layer and treat it as a regression problem for age prediction, then your network can learn weights for this particular task (You may use something like Mean Square Error loss here).

Let me give you a weak guidline on how to do this. Suppose say your input is images of chairs and you want to predict their age using a pretrained resnet, this is how you may do it in pytorch.

Definition
X: Input pictures
Y: List of ground truth values which is age here. i.e. : [1.4, 2.5, 2.2, ....]

Modify your neural network to give one output at the last layer
model = torchvision.models.resnet18(pretrained=True)
num_ftrs = model.fc.in_features
model.fc = torch.nn.Linear(num_ftrs, 1)

Design your loss function appropriately. You can treat age prediction as a standard regression problem. Of course, there are better loss designs here.
criterion =  torch.nn.MSELoss()

So use this loss during training
loss = criterion(output, target) where output is your neural network prediction and target is your ground truth values.

This is how you can modify an existing architecture for your task. Hope it helps.


How to use Genetic Algorithm for varying lengths of solutions Until now, I always thought that Genetic Algorithm can be used for problems of which the solution space can be encoded (modeled) as a chromosome of a specific length. However, some people claim that they used GA for this game and this game. They are basically games in which we control an agent on a 2-dimensional area.

Obviously, the length of the genome sequence depends on how fast the game is finished. So, how is GA used for such games?

If you think GA is not the most suitable method for this kind of problems can you explain why and give better alternatives?

Before I answer your question, it is important to frame what a GA is, so please allow me to cover some history.

Friedman and Fraser performed some of the earliest evolutionary computation experiments. Fraser presented the case of diploid organisms represented by binary strings of a given length. Each bit in the string represented a gene. Fraser proposed the process of single-point crossover for binary string reproduction.

American computer scientist John Henry Holland and his colleagues at the University of Michigan formalised the evolutionary algorithms originally implemented by Bremermann, Fraser, Friedman and Friedberg.

Holland's formal algorithm is known as a Genetic Algorithm (GA). A GA has a fixed length chromosome structure. Traditionally it was binary based. However due to hamming distances of various bit strings a GA using a binary encoded chromosome string can get stuck at local minima. Holland experimented with integer based GAs too.

Getting back to your question, theoretically and in its strictest sense, a Genetic Algorithm has a fixed length chromosome vector. The primary reason for this is that each gene represents something. If it was variable then the meaning of the gene must be encoded into the chromosome, which makes evolution more difficult. However, there are other chromosome structures that are variable length. The chromosome structure itself implies meaning.

Koza experimented with an alternative version of a GA called a genetic programming (GP). A GP replaces the fixed length vector structure used in a genetic algorithm with a variable length, non-linear, hierarchical abstract data structure, known as a tree.

The tree consists of branch nodes and leaf nodes. Leaf nodes have no children and implement a number or constant from a terminal set. A leaf node is also known as a terminal node. A branch node is connected to other branch nodes or leaf nodes and implement an operator from a functional set. A branch node is also known as a non-terminal node.

The tree structure naturally accommodates a game rule. A tree is not restricted to a fixed length. A tree is a natural structure to represent hierarchical decisions rules, and a tree can store both binary, logical, and numeric functions within its nodes. Logical functions allow the GP to evolve conditional execution of game play functions not easily implemented using a fixed length vector. Tree does need a traversal path and grammar (syntax and semantics).

In short to answer your questions:

A GA (fixed length) is generally not used for games, but rather a GP (variable with decisions).
A GA is not the most suitable method for this kind of problem as it is fixed
length, and each gene corresponds to a rule or value.

I would implement a GP, where the operators of mutation and crossover are adapted for a GP. I would also use the game itself as a fitness function, where a pool of individuals either using a single population, or co-evolution evolve players that gradually increase their score.

The paragraphs above were taken from Nicholls' MSc thesis. His thesis was on stock market trading rule generation which is similar to game rule generation. His thesis is available here (Only available after April 2019).


Is the gradient at a layer independent of the activations of the previous layers? Is the gradient at a layer (of a feed-forward neural network) independent of the activations of the previous layers?

I read this in a paper titled Mean Field Residual Networks: On the Edge of Chaos (2017). I am not sure how far this is true, because the error depends on those activations.

Yes, this is the premise of back-propagation, the gradient at layer $j_{n}$ is not impacted by the gradient at layer $j_{n-1}$. This allows you to start with a gradient at the output layer and propagated it back through the network to the input layer.

It is however impacted by the gradient at $j_{n+1}$, which the back-prop algorithm also follows.


How can I develop an object detection system that counts the number of objects and determines their position in an image? I want to create a simple object detection tool. So, basically, an image will be provided to the tool, and, from that image, it has to detect the number of objects.
For example, an image of a dining table that has certain items present on it, such as plates, cups, forks, spoons, bottles, etc.
The tool has to count the number of objects, irrespective of the type of object. After counting, it should return the position of the object with its size, so that I can draw a border over it.
I would like not to use any library or API present such as TensorFlow, OpenCV, etc., given that I want to learn the details.
If the process is very difficult to be created without using an API then the number of/type of objects which it will count as an object can also be limited but since this project will be for my educational/learning purpose can anyone help me understand the logic using which this can be achieved? For example, it may ignore a napkin present in the table to be counted as an object.

If you want to get experience, you should probably start with some easier task. Object detection and localization are relatively hard and writing a neural network and image processing pipeline from scratch will take you a long time.
If you want to build up an intuition about how NN's work, you might want to code some simple task from scratch. This is an example.
When you got some intuition about how NN's work, then you should proceed to your task. In here, you have a similar question and answer provided. The current state-of-the-art approach for your task would probably be point 3, that is object detection network, like YOLO or Faster-RCNN.


Why aren't compiled languages as popular as Python in AI? One of if not the most popular programming language for data science and AI today is Python, with R being a frequently cited runner-up. However, both of them are interpreted languages, which do not execute as fast as compiled languages. Why is that the case? The main advantage of AI over humans is in computing speed, and with real world AI applications today handling big data, execution time will already increase considerably as it is. Why aren't compiled languages preferred for this very reason?

Sure, the key argument and strength going for Python is its vast range of third party libraries available for AI, like scikit etc. but such communities can take root and grow anywhere under the right circumstances. Why did this community end up growing around Python and not a faster, equally common compiled language like C++, C# or Java?

There are a few advantages of interpreted languages (compared to compiled languages)

platform independence (you only need the interpreter for your platform, even though this is not true if e.g. your interpreted language is only a wrapper library around a library written in another programming language)
dynamic typing (no need to specify the types of the variables)
dynamic scoping (e.g. you can access variables in other scopes)
automatic memory management (but there are compiled languages, like Java, that also have a garbage collector)
Rapid prototyping (for various reasons, including dynamic typing), hence software can be written more quickly

Hence, the main advantages of an interpreted language compared to a compiled language are flexibility and dynamism. Given that AI is still an evolving field, these characteristics are widely appreciated.

There is also at least one disadvantage of interpreted languages (compared to compiled languages)

Slower running times compared to compiled languages, which, once compiled, are quite fast, because they are often compiled to a code that is quickly executable by the machine or virtual machine

Python and R are widely used in data science and artificial intelligence because of the advantages above, which possibly contributed to the rapid growth of the communities around them and the development of software libraries.

However, note that the core of the most common machine learning libraries today, including TensorFlow and PyTorch, is written in a compiled language like C and C++. In the specific case of TensorFlow, Python is just a wrapper library. Consequently, under the hood, the code is not executed by the Python interpreter, but first compiled, which implies that, when you're using e.g. TensorFlow, your code will run (more or less) as fast as if you were using a compiled language like C. A similar argument can be made for libraries like NumPy, where Python is just a wrapper library.


Solution to exercise 3.22 in the RL book by Sutton and Barto The goal is to find an optimal deterministic policy for this MDP:

There are two possible policies: left (L) and right (R). What is the optimal policy, when different discounts are used:

A $\gamma = 0$

B $\gamma = 0.9$

C $\gamma = 0.5$

The optimal policy $\pi_* \ge \pi$ if $v_{\pi^*}(s) \ge v_{\pi}(s), \forall s \in S$, so to find the optimal policy, the goal is to check which one of those results in the largest state value function for all states in the system given discount factors (A,B,C).

The Bellman equation for the state value function is

$v(s) = E_\pi[G_t | S_t= s] = E_\pi[R_{t+1} + \gamma v(S_{t+1}) | S_t = s]$

The suffix $_n$ marks the current iteration, and $_{n+1}$ marks the next iteration. The following is valid if the value function is initialized to $0$ or some random $x \ge 0$.

A) $\gamma = 0$

$v_{L,n+1}(S_0) = 1 + 0 v_{L,n}(S_L) = 1$

$v_{R,n+1}(S_0) = 0 + 0 v_{R,n}(S_R) = 0$

$L$ is optimal in case A.

B) $\gamma = 0.9$

$v_{L,n+1}(S_0) = 1 + 0.9 v_{L,n}(S_L) = 1 + 0.9(0 + 0.9 v_{L,n}(S_0)) = 1 + 0.81v_{L,n}(S_0)$

$v_{R,n+1}(S_0) = 0 + 0.9 v_{R,n}(S_R) = 0 + 0.9(2 + 0.9 v_{R,n}(S_0)) = 1.8 + 0.81v_{R,n}(S_0)$

$R$ is optimal in case B.

C) $\gamma = 0.5$

$v_{L,n+1}(S_0) = 1 + 0.5 v_{L,n}(S_L) = 1 + 0.5(0 + 0.9 v_{L,n}(S_0)) = 1 + 0.45v_{L,n}(S_0)$

$v_{R,n+1}(S_0) = 0 + 0.5 v_{R,n}(S_R) = 0 + 0.5(2 + 0.9 v_{R,n}(S_0)) = 1 + 0.45v_{R,n}(S_0)$

Both $R$ and $L$ are optimal in case C.

Question: Is this correct?

Your answer is correct but I am not sure exactly on how you arrived at it, as e.g. in the last case you don't know that $v_{L,n}(S_0) = v_{R,n}(S_0)$.

I will show for case B when $\gamma = 0.9$ as case A is trivial and hopefully you can apply what I've done in case B to case C so that you get exact answers.

Now, as you stated $v(s) = \mathbb{E}[R_{t+1} + \gamma v(S_{t+1}) | S_t = s]$. Assuming that $\gamma = 0.9$ we can calculate the values for each state under the policy of taking the left action. Note that because we are looking for deterministic policies and the environment is deterministic then a lot of the expectations can be disregarded as nothing random is happening.

'Left Policy'

\begin{align}v(s_0) &= 1 + 0.9 \times v(s_L) \\ v(s_L) &= 0 + 0.9 \times v(s_0) \\ v(s_R) &= 2 + 0.9 \times v(s_0)
\end{align}
We can solve this set of linear equations to get $v(s_0) = \frac{100}{19}, v(s_L) = \frac{90}{19}, v(s_R) = \frac{128}{19}\;.$

'Right Policy'

\begin{align}v(s_0) &= 0 + 0.9 \times v(s_R) \\ v(s_L) &= 0 + 0.9 \times v(s_0) \\ v(s_R) &= 2 + 0.9 \times v(s_0)
\end{align}
We can again solve these to obtain $v(s_0) = \frac{180}{19}, v(s_L) = \frac{162}{19}, v(s_R) = \frac{200}{19}\;.$

As we can see, for each of the states the value function is larger for all of the states under the policy 'go right', thus this is the optimal policy for the case of $\gamma = 0.9$.

It is important to note that if we take the 'left' action in state $s_0$ then our policy would never take us to state $s_R$, and the same for the right action and state $s_L$, however due to the definition of an optimal policy requiring $v_{\pi ^*}(s) \geq v_{\pi}(s)\; \forall s \in \mathcal{S}$ then we must evaluate the value function for all states, even ones that would not be visited under a policy you are evaluating. This means that the state for $s_R$ will change whether we go right or left, because the value of this state depends on the value of $s_0$, which clearly changes depending on whether we go right or left.


Applying NNs to 2d image noise reduction? I'd like to explore possibilities of applying deep learning on image noise reduction problem, more on photographic camera noise. What's a good NN architecture to solve problems like this?

EDIT 25,Nov,2017:
I have a small dataset of clean/noisy reference (~15K 4Kres images) acquired from digital camera. The target is to denoise other images from this camera type but without a reference photo.

You might want to look into autoencoders. What they do is learn the relationship between the input and a simplified version of it, and then try to reconstruct the original version of the input.

The good news about this architecture is that it is not a supervised learning technique, so you can feed it a lot of data without having to construct both input and output.

Hopefully, if you train it right, it will learn that the parts of the image that are not important in the result are the noisy parts that you intend to remove.

Note that autoencoders had already been used to reduce noise in audio and image with great success.

Update: Here's a quick not-so-technical explanation on why they work.

In the general architecture that auto-encoder networks have, you'll usually find an input layer, a certain amount of hidden layers which usually have less nodes than the input layer, and then a certain amount of hidden layers that gradually grow to have the same amount of input layers in the output layer.

Why they learn to reproduce the input in the output layer, the data representation needs to necessarily go through the narrow middle layer -- at that point, you can consider that the input has been "compressed", usually with a lossly approach to have a representation of the data that can later on use to reproduce the input.

(Side note: this compression feature has been tested in replacement of compression algorithms but they are not so good at performing compression as particular algorithms designed for it.)

When reducing the error between input and output, the network then starts forgetting the pieces that don't add as much information to the input as other bits of data in it. For instance, if you always train these images with figures in white backgrounds, it is likely that the encoded representation of the image will not have any information about the white background itself, since it is not required to be recreated.

As such, the most common pieces will start falling apart, and with the right tuning, a general amount of noise will go away too. Uniform noise is easier to remove (for the very same reason as the white background example), but variable noise can be worked around too with the right training.


What are some of the possible future applications of intelligent agents? I am trying to do some experiments with some intelligent agents, but I'm not sure how significant they will be in the future.

What are some possible interesting applications or use-cases of intelligent agents in the future?

For instance, it can be used as a virtual assistant instead of a real call agent. But what can be a more appealing application in the future?

At the moment, what I can think of are the following applications, but there are potentially a lot more.

Decision Maker: If you have any problem making a decision, intelligent agents can be used to weight evidence and give you statistics to rule out bad decisions.
Online Teacher: In the far future, intelligent agents may acquire human-like skills, they maybe used to teach different students (from different backgrounds) at once, but the advantage here is that the intelligent agent can evaluate every student's level (skills, personality, intelligence, etc), and use this data to decide how to convey information to him (differently). It would be like a virtual world were children receive information in parallel, while taking different data about them into consideration: They can be used to better education.
Social Media: I think intelligent agents will be integrated into social media, to analyze messages and comments, and warn users that some friends will dislike this or that or find it offensive, to give him time for making a decision about what to say. This would help limit bullying on social media.
Language Learning: in the future (like in 10 - 20 years), intelligent agents will have the ability to generate human-level language, that would make it possible for them to generate infinite sentences and structures just like humans can do, and again they can evaluate user's level to generate listening and reading material. For a human, it would be just like talking to a native speaker of that language (or even better, since this "native speaker" would know everything about your level).


What is chaotic behavior and how it is achieved in non-linear regression and artificial networks? I'm finding it hard to understand the relationship between chaotic behavior, the human brain, and artificial networks.  There are a number of explanations on the web, but it would be very helpful if I get a very simple explanation or any references providing such simplifications.

It looks like you have some common misconceptions about AI and neural networks.

First, AI programs generally do not try to imitate the human behaviour of a human brain. Instead, they try to imitate some higher-level behaviour. For example, they might imitate the reasoning process that you go through when you make a plan. In this context, the building-blocks (silicon or flesh) don't matter too much.

Second, artificial neural networks are also (mostly) not intended to imitate the human brain. Although they are inspired by the arrangement of neurons in a human brain, the networks used in most ANN systems have very little to do with real brains. The main similarity is that both systems have a lot of simple little computational units connected in patterns such that signals passed from one to another lead to interesting computations. However, real neurons produce lots of different kinds of signals, are connected in arbitrary ways, and randomization and transmission times play a significant role. Artificial neurons generally are deterministic, produce only one kind of signal (or sometimes a couple different kinds), are connected in extremely regular ways, and usually simulate instantaneous transmissions between neurons.


Is this proof of $\epsilon$-greedy policy improvement correct? The text book being referred to, in this question is "Reinforcement Learning: An introduction" by Richard Sutton and Andrew Barto (second edition, 2018). For your convenience, I have enclosed the following part of a paragraph about $\epsilon$-greedy policies in the book, to convey my question with a better clarity. This paragraph can be found at the end of Pg 100, under section 5.4 .

So, the non-greedy actions are given the probability $\frac{\epsilon}{|\mathscr{A}(s)|}$, and the greedy action is given the probability $1-\epsilon+\frac{\epsilon}{|\mathscr{A}(s)|}$. All clear upto this point. However, I have a doubt in the policy improvement theorem that is mentioned in Pg 101, under section 5.4. I have enclosed a copy of this proof for your convenience:
$$q_\pi(s, \pi'(s)) = \sum_a \pi'(a|s)q_\pi(s,a) \\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ = \frac{\epsilon}{|\mathscr{A}(s)|}\sum_aq_\pi(s,a) + (1-\epsilon)\max_a q_\pi(s,a) \\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \geq \frac{\epsilon}{|\mathscr{A}(s)|}\sum_aq_\pi(s,a) + (1-\epsilon)\sum_a\frac{\pi(a|s) - \frac{\epsilon}{|\mathscr{A}(s)|}}{1-\epsilon}q_\pi(s,a) \\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \  \ \ = \frac{\epsilon}{|\mathscr{A}(s)|}\sum_aq_\pi(s,a) - \frac{\epsilon}{|\mathscr{A}(s)|}\sum_aq_\pi(s,a) + \sum_a \pi(a|s)q_\pi(s,a) \\ = v_\pi(s)$$
My question is, shoundn't the greedy action be chosen with a probability of $1-\epsilon + \frac{\epsilon}{|\mathscr{A}(s)|}$? The weighing factors do not add up to 1, as they are probability values. With this argument, the proof (with a slight modification) would be:
$$q_\pi(s, \pi'(s)) = \sum_a \pi'(a|s)q_\pi(s,a) \\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ = \frac{\epsilon}{|\mathscr{A}(s)|}\sum_aq_\pi(s,a) + (1-\epsilon + \frac{\epsilon}{|\mathscr{A}(s)|})\max_a q_\pi(s,a) \\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \geq \frac{\epsilon}{|\mathscr{A}(s)|}\sum_aq_\pi(s,a) + (1-\epsilon + \frac{\epsilon}{|\mathscr{A}(s)|})\sum_a\frac{\pi(a|s) - \frac{\epsilon}{|\mathscr{A}(s)|}}{1-\epsilon + \frac{\epsilon}{|\mathscr{A}(s)|}}q_\pi(s,a) \\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \  \ \ = \frac{\epsilon}{|\mathscr{A}(s)|}\sum_aq_\pi(s,a) - \frac{\epsilon}{|\mathscr{A}(s)|}\sum_aq_\pi(s,a) + \sum_a \pi(a|s)q_\pi(s,a) \\ = v_\pi(s)$$
Though the end result isn't changed, I just want to know what I am conceptually missing, in order to understand the proof that is originally provided. I am extremely sorry, if this is something very elementary that I am not able to fathom.
Thank you so much for your time.

The weights do sum to one. Note that in the second line where we have
$$\frac{\epsilon}{|\mathcal{A}(s)|} \sum_a q_{\pi}(s,a) + (1-\epsilon)\max_aq_{\pi}(s,a) \; ,$$
the sum is over the whole action space, including the greedy action, so the sum of the weights will be $\frac{\epsilon}{|\mathcal{A}(s)|} \times |\mathcal{A}(s)| + (1-\epsilon) = 1$.


How exactly does adversarial training help in handling mode-collapse in generative networks? Of my understanding mode-collapse is when there happen to be multiple classes in the dataset and the generative network converges to only one of these classes and generates images only within this class. On training the model more, the model converges to another class.

In Goodfellows NeurIPS presentation he clearly addressed how training a generative network in an adversarial manner avoids mode-collapse. How exactly do GAN's avoid mode-collapse? and did previous works on generative networks not try to address this?

Apart from the obvious superior performance (generally), is the fact that GAN's address mode-collapse make them far preferred over other ways of training a generative model?

I don't think he said that at all. Going back to the talk you'll see he mentions mode collapse comes from the naivete of using alternating gradient-based optimization steps because then $min_{\phi}max_{\theta}L(G_\phi, D_\theta)$ starts to look a lot like $max_{\theta}min_{\phi}L(G_\phi, D_\theta)$.

This is problematic because in the latter case the generator has an obvious minimum of transforming all generated output into a single-mode that the discriminator has considered acceptable.

Since then a lot of work has been done to deal with this point of failure. Examples include Unrolled GANs (he mentions this one in the talk), where you essentially make the generator optimize what the discriminator will think $K$ steps in the future to ensure the ordering of the $min \ max$ game, and Wasserstein GANs, where you focus on a different metric that still has the same global minimum but allows for side by side training completely eliminating the ordering and failure mode, to begin with. On top of this, other work has been done as well, these are just two important examples.

Regarding how they fare against other generative models, like VAEs, there is no one is better than the other. The recent empirical success of GANs is why they are so popularly used, but we still see others being used in practice as well.


How can I train an AI to find known and new patterns in a log file? I'm developing an AI tool to find known equipment errors and find new patterns of failure. This log file is time-based and has known messages (information and error).
I'm using a JavaScript library, but my real question is how to train the AI to find the known patterns and find new possible patterns.
I have some requirements:

The tool shall either a. has no dependence on extra environment installation or b. the less the better (the perfect scenario is to run the tool entirely on the browser in standalone mode);

Possibility to make the pattern analyzer fragmented, a kind of modularity, one module per error;

What are the recommended kind of algorithm to do this (neural network, genetic algorithm, etc.)? Is there anything to achieve that end in JavaScript? If not, what is the best language to make this AI?

Correlation Between Entries

The first recommendation is to ensure that appropriate warning and informational entries in the log file are presented along with errors into the machine learning components of the solution.  All log entries are potentially useful input data if it is possible that there are correlations between informational messages, warnings, and errors.  Sometimes the correlation is strong and therefore critical to maximizing the learning rate.

System administrators often experience this as a series of warnings followed by an error caused by the condition indicated in the warnings.  The information in the warnings is more indicative of the root cause of failure than the error entry created as the system or a subsystem critically fails.

If one is building a system health dashboard for a piece of equipment or an array of machines that inter-operate, which appears to be the case in this question, the root cause of problems and some early warning capability is key information to display.

Furthermore, not all poor system health conditions end in failure.

The only log entries that should be eliminated by filtration prior to presentation to the learning mechanism are ones that are surely irrelevant and uncorrelated.  This may be the case when the log file is an aggregation of logging from several systems.  In such a case, entries for the independent system being analyzed should be extracted as an isolate from entries that could not possibly correlate to the phenomena being analyzed.

It is important to note that limiting analysis to one entry at a time vastly limits the usefulness of the dashboard.  The health of a system is not equal to the health indications of the most recent log entry.  It is not even the linear sum of the health indications of the most recent N entries.

System health has a very nonlinear and very temporally dependent relationships with many entries.  Patterns can emerge gradually over the course of days on many types of systems.  The base (or a base) neural net in the system must be trained to identify these nonlinear indications of health, impending dangers, and risk conditions if a highly useful dashboard is desired.  To display the likelihood of an impending failure or quality control issue, an entire time window of log entries of considerable length must enter this neural net.

Distinction Between Known and Unknown Patterns

Notice that the identification of known patterns is different in one important respect than the identification of new patterns.  The idiosyncrasies of the entry syntax of known errors has already been identified, considerably reducing the learning burden in input normalization stages of processing for those entries.  The syntactic idiosyncrasies of new error types must be discovered first.

The entries of a known type can also be separated from those that are unknown, enabling the use of known entry types as training data to help in the learning of new syntactic patterns.  The goal is to present syntactically normalized information to semantic analysis.

First Stage of Normalization Specific to Log Files

If the time stamp is always in the same place in entries, converting it to relative milliseconds and perhaps removing any 0x0d characters before 0x0a characters can be done before anything else as a first step in normalization.  Stack traces can also be folded up into tab delimited arrays of trace levels so that there is a one-to-one correspondence between log entries and log lines.

The syntactically normalized information arising out of both known and unknown entries of error and non-error type entries can then be presented to unsupervised nets for the naive identification of categories of a semantic structure.  We do not want to categorize numbers or text variables such as user names or part serial numbers.

If the syntactically normalized information is appropriately marked to indicate highly variable symbols such as counts, capacities, metrics, and time stamps, feature extraction may be applied to learn the expression patterns in a way that maintains the distinction between semantic structure and variables.  Maintaining that distinction permits the tracking of more continuous (less discrete) trends in system metrics.  Each entry may have zero or more such variables, whether known a priori or recently acquired through feature extraction.

Trends can be graphed against time or against the number of instances of a particular kind.  Such graphics can assist in the identification of mechanical fatigue, the approach of over capacity conditions, or other risks that escalate to a failure point.  Further neural nets can be trained to produce warning indicators when the trends indicate such conditions are impending.

Lazy Logging

All of this log analysis would be moot if software architects and technology officers stopped leaving the storage format of important system information to the varying convenient whims of software developers.  Log files are generally a mess, and the extraction of statistical information about patterns in them is one of the most common challenges in software quality control.  The likelihood that rigor will ever be universally applied to logging is small since none of the popular logging frameworks encourage rigor.  That is most likely why this question has been viewed frequently.

Requirements Section of This Specific Question

In the specific case presented in this question, requirement #1 indicates a preference to run the analysis in the browser, which is possible but not recommended.  Even though ECMA is a wonderful scripting language and the regular expression machinery that can be a help in learning parsers is built into ECMA (which complies with the other part of requirement #1, not requiring additional installations) un-compiled languages are not nearly as efficient as Java.  And even Java is not as efficient as C because of garbage collection and inefficiencies that occur by delegating the mapping of byte code to machine code to run time.

Many experimentation in machine learning employs Python, another wonderful language, but most of the work I've done in Python was then ported to computationally efficient C++ for nearly 1,000 to one gains in speed in many cases.  Even the C++ method lookup was a bottleneck, so the ports use very little inheritance, in ECMA style, but much faster.  In typical kernel code traditional, C structures and function pointer use eliminates vtable overhead.

The second requirement of modular handlers is reasonable and implies a triggered rule environment that many may be tempted to think is incompatible with NN architectures, but it is not.  Once pattern categories have been identified, looking for the most common ones first in further input data is already implied in the known/unknown distinction already embedded into the process above.  There is a challenge with this modular approach however.

Because system health is often indicated by trends and not single entries (as discussed above) and because system health is not a linear sum of the health value of individual entries, the modular approach to handling entries should not just be piped to the display without further analysis.  This is in fact where neural nets will provide the greatest functional gains in health monitoring.  The outputs of the modules must enter a neural net that can be trained to identify these nonlinear indications of health, impending dangers, and risk conditions.

Furthermore, the temporal aspect of pre-failure behavior implies that an entire time window of log entries of considerable length must enter this net.  This further implies the inappropriateness of ECMA or Python as a choice for the computationally intensive portion of the solution.  (Note that the trend in Python is to do what I do with C++:  Use object oriented design, encapsulation, and easy to follow design patterns for supervisory code and very computationally efficient kernel-like code for actual learning and other computationally intensive or data intensive functions.)

Picking Algorithms

It is not recommendable to pick algorithms in the initial stages of architecture (as was implied at the end of the question).  Architect the process first.  Determine learning components, the type of them needed, their goal state after training, where reinforcement can be used, and how the wellness/error signal will be generated to reinforce/correct desired network behavior.  Base these determinations not only on desired display content but on expected throughput, computing resource requirements, and minimal effective learning rate.  Algorithms, language, and capacity planning for the system can only be meaningfully selected after all of those things are at least roughly defined.

Similar Work in Production

Simple adaptive parsing is running in the lab here as a part of social networking automation, but only for limited sets of symbols and sequential patterns.  It does scale without reconfiguration to an arbitrarily large base linguistic units, prefixes, endings, and suffixes, limited only by our hardware capacities and throughput.  The existence of regular expression libraries was helpful to keep the design simple.  We use the PCRE version 8 series library fed by a ansiotropic form of DCNN for feature extraction from a window moving through the input text with a configurable windows size and move increment size.  Heuristics applied to input text statistics gathered in a first pass produce a set of hypothetical PCREs arranged in two layers.

Optimization occurs to apply higher probabilistic weights to the best PCREs in a chaotically perturbed text search.  It uses the same gradient descent convergence strategies used in NN back propagation in training.  It is a naive approach that does not make assumptions like the existence of back-traces, files, or errors.  It would adapt equally to Arabic messages and Spanish ones.

The output is an arbitrary directed graph in memory, which is similar to a dump of an object oriented database.

قنبلة -> dangereux -> 4anlyss
bomba -> dangereux
ambiguïté -> 4anlyss -> préemption -> قنبلة

Although a re-entrant algorithm for a reinforcement version is stubbed out and the wellness signal is already available, other work preempted furthering the adaptive parser or working toward the next step to use the work for natural language: Matching the directed graphs to persisted directed graph filters representing ideas, which would mimic the idea recollection aspect of language comprehension.

Final Comments

The system has components and process architecture similar to the log analysis problem and prove the concepts listed above.  Of course, the more disorganization in the way logging is done between developers of the system doing the logging, the more difficult it is for a human or artificial agent to disambiguate the entries.  Some system logging has been so poorly quality control for so long that the log is nearly useless.


How to train a sequence labeling model with annotations from three annotators? I have a dataset of movie reviews annotated by 3 persons. The following example contains one sentence with corresponding annotations from 3 different persons.
sentence = ['I', 'like', 'action', 'movies','!']
annotator_1 = ['O','O', 'B_A', 'I_A', 'O']
annotator_2 = ['O','O', 'B_A', 'I_A', 'O']
annotator_3 = ['O','O', 'B_A', 'O', 'O']

The labels follow the BIO format. That is, B_A means the beginning of aspect-term (action) and I_A indicates inside of aspect-term (movie).
Unfortunately, the annotators do not agree always together. While the first two persons assigned the right labels for aspect-term (action movie), the last one mislabeled the token (movies).
I am using Bi-LSTM-CRF sequence tagger to train the model. However, I am not sure if am using the training data correctly.
Is it correct to feed the model the same sentence with annotations from 3 persons? Then test it in the same way, i.e., the same sentence with different annotations?
Another question.
I merged the annotations in one final list of labels as follows:
final_annotation = ['O','O', 'B_A', 'I_A', 'O']

In this case, the final label is chosen based on the majority of labels among three annotators.
Is it right to feed the model the same sentence with corresponding annotations from all users during the testing phase?

Both ways are valid. It depends on what you want from the model and expect from the data. Generally though I would use 1 assumption and stick with it (unless there was a specific reason not to), so I would use all lines for test if training done that way, and same for majority.
Also note if you ever get more than 3 people, you can choose to do a variance based approach (use data if only x% agree, throw away otherwise (or you could even weigh controversial labels lower))


How can a neural network work with continuous time? I have an ANN model that receives an input and produces an output. The output is an action that interacts with the environment and changes the input accordingly. The network has a desired environment state which, in any turn, decides the desired response and trains the network on that basis.

Currently, the network works in discrete time. How can I make this network work in a continuous manner? Can you provide some resources and links if there is any past or current research on continuous AI?

To be more concrete, the system starts with the current environment state, for example, [1 1 1], then produces an output. In current system, the next step takes the final state of the system as input, for example, [1 2 2], but we know that such a thing doesn't happen in physical world and the system goes from [1 1 1] to, for example, [1 1 2], and then to [1 2 2], and that middle step is something that a discrete-time AI can't figure out.

The very case that I'm working on is the simulation for an autopilot cart in which the model is incapable to take subtle things like "the maximum speed that you can turn the steering wheel" into consideration. I don't want to add these complexities to the model since if the model is perfect, then the result is deterministic and there is no need for AI. I want the AI to be able to make a decision in each step based on the current state of the system in continuous time.

By the way you have explained things above, it seems more like a problem with your code and not the something to do with the environment. The term discrete and continuous is used to define, how the outside environment is acting, rather than how your code is taking its steps. These are some lines from the book, Artificial Intelligence: A Modern Approach:

  The discrete/ continuous distinction applies to the state of the environment, to the way time is handled, and to the percepts and actions of the agent. For example, the chess environment has a finite number of distinct states (excluding the clock). Chess also has a discrete set of percepts and actions. Taxi driving is a continuous state and continuous-time problem: the speed and location of the taxi and of the other vehicles sweep through a range of continuous values and do so smoothly over time. Taxi-driving actions are also continuous (steering angles, etc.). Input from digital cameras is discrete, strictly speaking, but is typically treated as representing continuously varying intensities and locations.

So, continuous or discrete is not something that should be talked about as a problem of the code. It is basically, what an environment is. Your concern with the device is regarding the code. I will suggest that you upload the code on git and ask people to improve it.
I hope this helps!


Neural network design when amount of input neurons vary I'm looking to design a neural network that can predict which runner wins in a sports game, where the amount of runners varies between 2-10. In each case, specific data about the individual runners would be fed into the neural network.

What design would be most advantageous for such a neural network?

Essentially this is a ranking problem where the amount of inputs and outputs are variable.

The best option in your case would probably be zero padding or padding up. This is simply zeroing out inputs for cases in which there is no data. It's done a lot on the borders of images for CNNs.

That or you could just use a RNN which can handle your variable length inputs with ease.


Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously? Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously? Is there an agreed up protocol on how to extract this information?
What combination of layers works: convolution + LSTM? What would be the alternatives?

Yes, there are different ways. What I think you are looking for is under the research field of Localization and Mapping. Which divides in the following subfields:

For getting current (the robot) position and trajectory go to models for Odometry Estimation
For getting a representation of the world around the robot go to models for Mapping
If you want both of them (I am guessing you want). Go for SLAM (Simultaneous Localization and Mapping) models

Here it is the amazing survey that links you to tons of papers with different models for each category. If you want to know what are the most common architectural blocks (LSTM, ConvLSTM, RNN...) used for your problem, read the most promising papers under your target category.
References:
Survey: https://arxiv.org/abs/2006.12567