Are there some notions of distance between two policies? I want to determine some distance between two policies $\pi_1 (a \mid s)$ and $\pi_2 (a \mid s)$, i.e. something like $\vert \vert \pi_1 (a \mid s) - \pi_2(a \mid s) \vert \vert$, where $\pi_i (a\mid s)$ is the vector $(\pi_i (a_1 \mid s), \dots, \pi_i(a_n \mid s))$. I am looking for a sensible notion for such a distance.
Are there some standard norms/metrics used in the literature for determining a distance between policies?

Given that policies are probability distributions, in principle, you can use any metric or measure of distance that can be used to compare two probability distributions. (Note that notions of distance are not necessarily metrics in a mathematical sense).
A common measure is the Kullback–Leibler divergence (which is not a metric, in a mathematical sense, given that it does not satisfy certain required conditions for being a metric). For example, in section 4 of the PPO paper, the KL divergence is used as a regulariser (which is actually quite common, for instance, in the context of variational Bayesian neural networks). The TRPO also uses the KL divergence.
The Wasserstein metric has also been used in RL, for instance, in distributional RL (but, in this case, not to compare policies but distributions over value functions).
You can find more info about statistical distances here. The specific distance that you use may depend on the problem that you want to solve and the properties that you want your distance to have. For example, the KL divergence is unbounded above, so, if that's not desirable, you could choose another one. The paper On choosing and bounding probability metrics (2002, by Gibbs and Su) may also be useful. Here I also talk about the KL divergence and total variation.


How to exclude sections of bad data from time-series data before training an LSTM network I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks.
This results in several multiple sections [continuous streak of bad data for several days until the problem is fixed].
I need to exclude this bad data section before feeding it to model training.
Since I am using LSTM-RNN network, it requires to do an un-roll data based on the previous records.

How can I properly exclude this bad data?
I thought of an approach as training model separately using each batch of good data, and use subsequent good-data batch for fine-tuning the model.
Please let me know if this is a good approach? or is there a better method?

example data:
"1-01",266.0
"1-02",145.9
"1-03",183.1
"1-04",0  [bad data]
"1-05",0  [bad data]
"1-06",0  [bad data]
"1-07",0  [bad data]
"1-08",224.5
"1-09",192.8
"1-10",122.9
"1-11",0  [bad data]
"1-12",0  [bad data]
"2-01",194.3
"2-02",149.5

Your idea is a good one.
Another idea is to upsample or aggregate your data.  For example, average by week if you generally have a couple of missing days in every week.
A similar question on Stack Exchange:
https://stats.stackexchange.com/questions/374935/how-to-deal-with-really-sparse-time-series-data-for-a-binary-classification-task


Training Custom object detection network using tensor-flow object detection API? I was just wondering if some one could provide a nice tutorial on how to use the Recent tensor-flow object detection API to train custom network say like VGG-16? (Just USE the VGG-16, VGG-19, Inception-v3 etc as a fixed feature extractor in the Faster RCNN implementation).

From their Documentation I know it can be done. As, am an amateur in AI, I am unsure how to move forward.

P.S: This could really help some keras fans (like me :P) who have A trained CNN in keras. They could just make a .pb file out of it and insert it as a fixed feature detector/extractor in the Tensorflow Object detection API. Voila!!! Now Keras is integrated with the Tensorflow's object detection API

I have found a hack to integrate keras with tensorflow object detection API. This hack works if you have trained the keras classification model with tensorflow backend. If above is the case you can extend the classification model to a object detection model by first converting the keras checkpoint to a tensorflow checkpoint then in the object detection API write new feature extractor layers using tf.keras (Keras is now part of core tensorflow starting from version 1.4). you can simply copy paste your layer definitions and pass the path to the converted checkpoint in the config file and it will work.


Problems that only humans will ever be able to solve With the increasing complexity of reCAPTCHA, I wondered about the existence of some problem, that only a human will ever be able to solve (or that AI won't be able to solve as long as it doesn't reproduce exactly the human brain).

For instance, the distorted text used to be only possible to solve by humans. Although...

  The computer now got the [distorted text] test right 99.8%, even in the most challenging situations.

It seems also obvious that the distorted text can't be used for real human detection anymore.

I'd also like to know whether an algorithm can be employed for the creation of such a problem (as for the distorted text), or if the originality of a human brain is necessarily needed.

Informally, AI-complete problems are the most difficult problems for an AI. The concept is not mathematically defined yet, as e.g. NP-complete problems. However, intuitively, these are the problems that require a human-level or general intelligence to be solved.

Real natural language understanding is believed to be an AI-complete problem (this is also discussed in the paper Making AI Meaningful Again by Jobst Landgrebe and Barry Smith, 2019). There are a lot more AI-complete problems. For example, problems that involve emotions.


Are decision tree learning algorithms deterministic? Are decision tree learning algorithms deterministic? Given a fixed dataset, do they always produce a tree with the same structure?

What about the random forest?

Are decision tree learning algorithms deterministic? Given a fixed dataset, do they always produce a tree with the same structure?

Generally, yes. Most decision tree learners, like the common ID3 and C4.5/C5.0 algorithms, are deterministic. At each step, the learners consider all possible features that have not yet been used to split the data, and find the splits that maximize some function (e.g. information gain). There is no randomness (or pseudo-randomness) in this process.
The exceptions to this would be if you used randomness to break ties (rather than, say, using the index of each feature, as is common), but this would be an unusual modification.
What about the random forest?
Like the name suggests, random forests do make use of randomness, or at least, pseudo-randomness. If we're only concerned about whether or not the algorithm is deterministic in the usual sense of the word (at least, within computer science), the answer is no.
If you start the same random forest learning algorithm, with the same datasets, at two different times (or, using two different seeds for your pseudorandom number generator), you will get two different forests. This is because the algorithm selects random subsets of the features and/or datapoints to learn on, and, if different seeds are used, the subsets will be different each time.


How does $\mathbb{E}$ suddenly change to $\mathbb{E}_{\pi'}$ in this equation? In Sutton-Barto's book on page 63 (81 of the pdf):
$$\mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_t=s,A_t=\pi'(s)] = \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_{t} = s]$$

How does $\mathbb{E}$ suddenly change to $\mathbb{E}_{\pi'}$ and the $A_t = \pi'(s)$ term disappears?

Also, in general, in the conditional expectation, which distribution do we compute the expectation with respect to? From what I have seen, in $\mathbb{E}[X \mid Y]$, we always calculate the expected value over distribution $X$.

Also, in general, in the conditional expectation, which distribution do we compute the expectation with respect to? From what I have seen, in $\mathbb{E}[X|Y]$, we always calculate the expected value over distribution $X$.

No, for $\mathbb{E}[X|Y]$ we take expectation of $X$ with respect to the conditional distribution $X|Y$, i.e.

$$\mathbb{E}[X|Y] = \int_\mathbb{R} x p(x|y)dx\;;$$

where $p(x|y)$ is the density function of the conditional distribution. If your random variables are discrete then replace the integral with a summation. Also note that $\mathbb{E}[X|Y]$ is still a random variable in $Y$.

  How does $\mathbb{E}$ suddenly change to $\mathbb{E}_{\pi '}$ and the $A_t = \pi '(s)$ term disappears?

This is because in this instance $\pi '(s)$ is a  deterministic policy, i.e. in state $s$ the policy will take action $b$ with probability 1 and all other actions with probability 0. NB: this is the convention used in Sutton and Barto to denote a deterministic policy.

Without loss of generality, assume that $\pi'(s) = b$. The implication of this is that in the first expectation we have
$$\mathbb{E}[R_{t+1} + \gamma v(S_{t+1}) | S_t = s, A_t = \pi'(s) = b] = \sum_{s',r}p(s',r|s,a=b)(r + \gamma v(s'))\;,$$
and in the second expectation we have
$$\mathbb{E}_{\pi'}[R_{t+1} + \gamma v(S_{t+1}) | S_t = s] = \sum_a\pi'(a|s)\sum_{s',r}p(s',r|s,a)(r + \gamma v(s'))\;;$$
However, we know that $\pi'(a|s) = 0 \; \forall a \neq b$, so this sum over $a$ would equal 0 for all $a$ except when $a=b$, in which case we know that $\pi'(b|s) = 1$, and so the expectation becomes

$$\mathbb{E}_{\pi'}[R_{t+1} + \gamma v(S_{t+1}) | S_t = s] = \sum_{s',r}p(s',r|s,a=b)(r + \gamma v(s'))\;;$$

and so we have equality of the two expectations.


Can Facebook's LASER be used like BERT? Can Facebook's LASER be fine-tuned like BERT for Question Answering tasks or Sentiment Analysis?
From my understanding, they created an embedding that allows for similar words in different languages to be close to each other. I just don't understand if this can be used for fine-tuning tasks like BERT can.

LASER creates multilingual contextualized word embeddings, what you do with them is up to you. You can use this as a feature extraction and add whatever you want to the end of the network.
I believe the implementation by facebook does not let you change the weights of the LASER model itself, they are froozen to the best of my knowledge.
So, yes, you can use LASER for downstream tasks, but you cannot change the actual weights of the LASER model.


Is this model overfitted or not? I am training a neural network and plot model accuracy and model loss.
I am a little confused about overfitting. Is my model overfitted or not? how can I interpret it

EDIT: here is a sample of my input data, I have a binary image classification

Overfitting nearly always occurs to some degree when fitting to limited data sets, and neural networks are very prone to it. However neither of your graphs show a major problem with overfitting - that is usually obvious when epoch counts increase, the results on training data continue to improve whilst the results on cross validation get progressively worse. Your validation results do not do that, and appear to remain stable.

It is usually pragmatic to accept that there will be at least some difference between measurements on the training set and cross validation or test sets. The primary goal is usually to get the best measurements in test that you can. With that in mind, you are usually only interested in how much you are overfitting if it implies you could improve performance by using techniques to reduce overfitting e.g. various forms of regularisation.

Without knowing your data set or known good results, it is hard to tell whether the difference you are seeing between test and train in accuracy could be improved. Your accuracy graph shows a train accuracy close to 100% and a validation accuracy close to ~96%. It looks a bit like MNIST results, and if I saw that result on MNIST I would suspect something was wrong, and it might be fixed by looking at regularisation (but it might also be something esle). However, that's only because I know that 99.7% accuracy is possible in test - on other problems I might be very happy with 96% accuracy.

The loss graph is not very useful, since the scale has completely lost any difference there might be between training and validation. You should probably re-scale it to show detail close to 0 loss, and ignore the earlier large loss values.


How to make machine learning model that reports ambiguity of the input? Suppose I want to build a neural network regression model that takes one input and return one output.

Here's the training data:

0.1 => 0.1
0.2 => 0.2
0.1 => -0.1

You will see that there are 2 inputs 0.1 that matches to different output values 0.1 and -0.1. So what will happen with most machine learning models is that they will predict the average when 0.1 is fed to the model. E.g. the output of 0.1 will be (0.1 + (-0.1))/2 = 0.

But this 0 as an average answer is an incorrect answer. I want the model to be telling me that the input is ambiguous/insufficient to infer the output. Ideally, the model would report it as a form of confidence.

How do I report predictability confidence from the input?

The application that I find very useful in many areas is that I could then later ask the model to show me inputs that are easy to predict and inputs that are ambiguous. This would make me able to collect the data that are making sense.

One way I know is to train the model then check the error on each training data, if it's high then it probably means that the input is ambiguous. But if you know any other papers or better techniques, I would be appreciated to know that!

Another specific way to do this if one uses a neural network for this. Use a dropout a layer in your network and instead of scaling the activations at test time, one can sample the activations (just like in training-time) and predict multiple times for a given input, then look at distribution of your outputs. Intuitively this would add "probabilistic, bayesian effect" to you neural network.
I think this method was first proposed in Dropout as a Bayesian Approximation:
Representing Model Uncertainty in Deep Learning which is called Monte Carlo Dropout.


What amount of ressources is involved in building an image recognition system? I would like to have an order of magnitude of ressources required to build an image recognition system.
Let say you want to build a startup company which main product will have to distinguish 20 different kinds of objects (bottle, dogs, car, flowers...). Images are already tagged.

How many images are needed as a learning set ? 1k, 10k, 100k, 1
million ?
What kind of hardware and how long will the learning process take ?
How many developers, how much time ?
Does it changes a lot if the number of target output is reduced to two kinds, or increased to one thousands ?

A link to a real life paper would be perfect. Thank you

One answer is infinite amount of time because it can always be better.
Another answer is:

10k for training set
A PC with a GPU (3~4k USD), google colab (10 USD per month), or other cloud service (probably more expensive than colab)
One developer, 1 day lol
Two kinds is easier than multiple kinds
There is no paper that seeks to answer your question the way you put it. I wouldn't even recommend a paper. In fact, for you I'd recommend an AutoML tutorial. Check these. * no offence if I've misjudged your knowledge/skill level.
Here's a paper anyway :) https://paperswithcode.com/lib/torchvision/alexnet

To conclude, please be aware that your question is super open ended, and my answer is bad (but good enough for now maybe), but a good answer doesn't really exist. It's always going to be context dependent. For instance, you never said whether you need 90% or 99% accuracy.


Why can a fully convolutional network accept images of any size? On this article, it says that:

  The UNET was developed by Olaf Ronneberger et al. for Bio Medical Image Segmentation. The architecture contains two paths. First path is the contraction path (also called as the encoder) which is used to capture the context in the image. The encoder is just a traditional stack of convolutional and max pooling layers. The second path is the symmetric expanding path (also called as the decoder) which is used to enable precise localization using transposed convolutions. Thus it is an end-to-end fully convolutional network (FCN), i.e. it only contains Convolutional layers and does not contain any Dense layer because of which it can accept image of any size.

What I don't understand is how an FCN can accept images of any size, while an ordinary object detector, such as YOLO with a dense layer at the very end, cannot accept images of any size.

So, why can a fully convolutional network accept images of any size?

The reason is that when using a convolutional layer, you select the size of the filter kernels, which are independent of the image/layer input size (provided that images smaller than the kernels are padded appropriately).

When using a dense layer, you specify the size of the layer itself and the resulting weight matrix is a function of both the size of the dense layer and the upstream layer. This is because each neuron in the upstream layer makes a connection to each neuron in the dense layer. So, if you have 50 neurons in the upstream layer and 20 neurons in the dense layer, then the weight matrix has $50 \times 20=1000$ values. Those weights are what get determined during the training phase, and so those layer sizes are fixed.

Now, the output of a CNN layer is a number of images/tensors (specified by the number of filters chosen), whose size is determined by the kernel size and any padding option chosen. If those are fed into a dense layer, then that fixes the the size that those images can be (because of the reason given in the previous paragraph).

On the other hand, if no dense layer is used in the whole network, then the input to the first CNN layer can be any size because the weights are just the individual parameters of the filter kernels, and the filter kernels remain the same size regardless of the input tensor size.


Why is the convolution layer called Conv2D? When I build a convolution layer for image processing, the filter parameters should have 3 dimensions, (filter_length, filter_width, color_depth) is that correct?
Why does this convolution layer call Conv2D? Where does the 2 come from?

A 2D convolution is a convolution where the kernel has the same depth as the input, so, in theory, you do not need to specify the depth of the kernel, if you know the depth of the input.
I don't know which library you are referring to (although you tagged your post with TensorFlow and Keras), but, in TensorFlow, you only need to specify the width and height of the kernel in the Conv2D class, given that the depth is automatically calculated from the depth of the input.
Thus, the $2$ comes from the fact that you slide across two dimensions (i.e. width and height).
On the other hand, in a 3D convolution, the depth of the kernel does not necessarily have the same depth of the input, so, in that case, you also slide across the depth. In TensorFlow, you need to specify the width, height and depth of the Conv3D class.


Any explanation why multiple linear layers work better than a single linear layer in practice? It is a well-known math fact that composition of linear/affine transformations is still linear/affine. For a naive example,
$\textbf{A}_1\textbf{A}_2\textbf{x}$ is simply $\textbf{A}\textbf{x}$ where $\textbf{A}=\textbf{A}_1\textbf{A}_2$
Any one knows why in practice multiple linear layers tend to work better, even though it is mathematically equivalent to a single linear layer? Any reference is appreciated!

The key is that the layers of neurons in neural networks are not affine transformations. All commonly used neurons have some kind of non-linearity. The simplest of these is the Rectified Linear Unit (ReLU), which takes the form $y = x$ when $x > 0$ and $y = 0$ for all other values, where $x$ is a weighted sum of the inputs to the neuron.


What is the difference between graph convolution in the spatial vs spectral domain? I've been reading different papers regarding graph convolution and it seems that they come into two flavors: spatial and spectral. From what I can see the main difference between the two approaches is that for spatial you're directly multiplying the adjacency matrix with the signal where for the spectral version you're using the Laplacian matrix. Am I missing something or are there any other differences that I am not aware of?

Spectral Convolution
In a spectral graph convolution, we perform an Eigen decomposition of the Laplacian Matrix of the graph. This Eigen decomposition helps us in understanding the underlying structure of the graph with which we can identify clusters/sub-groups of this graph. This is done in the Fourier space.
An analogy is PCA where we understand the spread of the data by performing an Eigen Decomposition of the feature matrix. The only difference between these two methods is with respect to the Eigen values. Smaller Eigen values explain the structure of the data better in Spectral Convolution whereas it's the opposite in PCA.
ChebNet, GCN are some commonly used Deep learning architectures that use Spectral Convolution

Spatial Convolution
Spatial Convolution works on local neighbourhood of nodes and understands the properties of a node based on its k local neighbours. Unlike Spectral Convolution which takes a lot of time to compute, Spatial Convolutions are simple and have produced state of the art results on graph classification tasks. GraphSage is a good example for Spatial Convolution.

Additional References: https://towardsdatascience.com/tutorial-on-graph-neural-networks-for-computer-vision-and-beyond-part-2-be6d71d70f49

https://towardsdatascience.com/graph-convolutional-networks-for-geometric-deep-learning-1faf17dee008


Is the distribution of state-action pairs from sample based planning accurate for small experience sets? From the David Silver's lecture 8: Integrating Learning and Planning - based on Sutton and Barto - he talks about using sample-based planning to use our model to take a sample of a state and then use model-free planning, such as Monte Carlo, etc, to run the trajectory and observe the reward. He goes on to say that this effectively gives us infinite data from only a few actual experiences.

However, if we only experience a handful of true state-action-rewards and then start sampling to learn more then we will surely end up with a skewed result, e.g., If I have 5 experiences but then create 10000 samples (as he says, infinite data). I am aware that as the experience set grows the Central Limit Theorem will come into play and the distribution of experience will more accurately represent the true environment's state-actions-rewards distribution but before this happens is sampled based planning still useful?

I am aware that as the experience set grows the Central Limit Theorem
  will come into play and the distribution of experience will more
  accurately represent the true environment's state-actions-rewards
  distribution

I believe here you mean the Law of Large Numbers which states that for a large enough sample ($n \rightarrow \infty$) the sample mean will converge to the true mean. The central limit theorem (CLT) states that if you take the sum/mean of a set of independent random variables then the distribution of this new RV will be approximately normal as $n \rightarrow \infty$.

  before this happens is sampled based planning still useful

As you mentioned, if you had only 5 full episodes of experience to choose from, then this likely would not represent true underlying distributions and so the approximations would not be good -- of course, this will depend on how complex your MDP is, if you had a trivially simple one then it may be enough to represent it well. As David Silver says in his lecture, then one of the disadvantages of planning with a model is that you introduce another set of uncertainty, mainly from when you approximate the properties of the model.


An example of a unique value function which is associated with multiple optimal policies In the 4th paragraph of
http://www.incompleteideas.net/book/ebook/node37.html
it is mentioned:

  Whereas the optimal value functions for states and state-action pairs are unique for a given MDP, there can be many optimal policies

Could you please give me a simple example that shows different optimal policies considering a unique value function?

Consider a very simple grid-world, consisting of 4 cells, where an agent starts in the bottom-left corner, has actions to move North/East/South/West, and receives a reward $R = 1$ for reaching the top-right corner, which is a terminal state. We'll name the four cells $NW$, $NE$, $SW$ and $SE$ (for north-west, north-east, south-west and south-east). We'll take a discount factor $\gamma = 0.9$.

The initial position is $SW$, and the goal is $NE$, which an optimal policy should reach as quickly as possible. However, there are two optimal policies for the starting state $SW$: we can either go north first, and then east (i.e., $SW \rightarrow NW \rightarrow NE$), or we can go east first, and then north (i.e., $SW \rightarrow SE \rightarrow NE$). Both of those policies are optimal, both reach the goal state in two steps and receive a return of $\gamma \times 1 = 0.9$, but they are clearly different policies, they choose different actions in for the initial state.

Note that my language was slightly informal above when talking about "policies for the starting state". Formally, I should have said that there are two optimal policies that select different actions in the starting state (and the same actions in all other states).


How to transform inputs and extract useful outputs in a Neural Network? So I've been trying to understand neural networks ever since I came across Adam Geitgey's blog on machine learning. I've read as much as I can on the subject (that I can grasp) and believe I understand all the broad concepts and some of the workings (despite being very weak in maths), neurons, synapses, weights, cost functions, backpropagation etc. However, I've not been able to figure out how to translate real world problems into a neural network solution.

Case in point, Adam Geitgey gives as an example usage, a house price prediction system where given a data set containing No. of bedrooms, Sq. feet, Neighborhood and Sale price you can train a neural network to be able to predict the price of a house. However he stops short of actually implementing a possible solution in code. The closest he gets, by way of an example, is basic a function demonstrating how you'd implement weights:

def estimate_house_sales_price(num_of_bedrooms, sqft, neighborhood):
  price = 0

  # a little pinch of this
  price += num_of_bedrooms * 1.0

  # and a big pinch of that
  price += sqft * 1.0

  # maybe a handful of this
  price += neighborhood * 1.0

  # and finally, just a little extra salt for good measure
  price += 1.0

  return price

Other resources seem to focus more heavily on the maths and the only basic code example I could find that I understand (i.e. that isn't some all singing, all dancing image classification codebase) is an implementation that trains a neural network to be an XOR gate that deals only in 1's and 0's.

So there's a gap in my knowledge that I just can't seem to bridge. If we return to the house price prediction problem, hows does one make the data suitable for feeding into a neural network? For example:

No. of bedrooms: 3
Sq. feet: 2000
Neighborhood: Normaltown
Sale price: $250,000

Can you just feed 3 and 2000 directly into the neural network because they are numbers? Or do you need to transform them into something else? Similarly what about the Normaltown value, that's a string, how do you go about translating it into a value a neural network can understand? Can you just pick a number, like an index, so long as it's consistent throughout the data?

Most of the neural network examples I've seen the numbers passing between layers are either 0 to 1 or -1 to 1. So at the end of processing, how do you transform the output value to something usable like $185,000?

I know the house price prediction example probably isn't a particularly useful problem given that it's been massively oversimplified to just three data points. But I just feel that if I could get over this hurdle and write an extremely basic app that trains using pseudo real-life data and spits out a pseudo real-life answer than I'll have broken the back of it and be able to kick on and delve further into machine learning.

This is a good question which I wrestled with myself when first trying to code an ANN.

Below is a good general-purpose solution, and it's the one I implemented in my code for trying to predict well-behaved numerical data. If your data is not well-behaved (i.e. fraught with outliers) then you may need to do more work normalizing the inputs and outputs. Some of the more advanced methods are described here.

Note: I will assume that you are using f(x) = tanh(x) as your activation function. If you aren't, you should still be able to reason through how to normalize your data after reading this.

How to prepare the input data:

The basic idea is that you want a significant variation in each input parameter to be reflected by a significant variation in the activation of the neuron those inputs are being fed into. By looking at a plot of the derivative of the tanh(x) actiavtion function, you'll see that the region of significant slope is within a distance of one or two from the origin. This means that whether the input to the activation function is 2000 or 3000 (values of x for which the derivative is negligibly small), the output of the activation will be almost identical...so your neuron's state will be independent of the difference between 2000 and 3000, and your network will never produce any predictive power from values in that range.

So if you want to input the square footage of the house into a neuron, you need to normalize the square footage so that the network can tell the difference between 2000 and 3000. One way to do this so that all of the significant variations in your data are 'noticed' by the neuron is to z-score-normalize the inputs.

Gather all of your square footage values (from your training set) and calculate the mean and standard deviation. Store the mean and standard deviation---you'll need this information to normalize new square footage values when testing.
Normalize the vector of square footage values by subtracting the mean and then dividing the result by the standard deviation (all operations element-wise of course). Subtracting the mean centers your data at the origin, and dividing by the standard deviation makes sure most of it is between -1 and 1, where the neuron's output is most sensitive to its input. This is called z-score normalization because each input value is replaced by its z-score.
Do the above for each input variable.

Now, when you put each input value through a neuron, the output of the neuron is an activation between -1 and 1 (look at the image of tanh(x)). Since this is already in the 'sensitive' range of the activation function, you don't need to worry about altering the output of the input-layer neurons before sending them to the first hidden layer. Just give any hidden layer neurons the outputs of the previous layer directly---they will be able to handle them just fine.

When you reach the last layer (the output neuron(s)), what you get is again another activation between -1 and 1. You have to convert this back into a value for the house in question, whether that value will be used as a prediction in a test set or to calculate error during training. However you do this, you just have to be consistent and use the same de-normalization procedure in training and testing. One way to think about it is: when the output neuron(s) returns 1, that means the network is returning the maximum possible house value as its prediction. What should the highest value the network can estimate be? The right approach here simply depends on your application. This is what I did:

Calculate the mean of [the/each] output variable and store it.
Calculate the maximum deviation of the output variable from the mean.
Python: MaxDev = max([abs(DataPoint-numpy.mean(TrainingData)) for DataPoint in TrainingData])
When the network returns output(s) between -1 and 1, multiply the output by MaxDev and add it to the mean.

Two basic quick checks you can do to see if your normalization-renormalization scheme is suitable (these are necessary, but perhaps not sufficient conditions):

If all the input values are average (e.g. average no. of bedrooms, average sq.feet, etc), is the network's output equal to the average of the output variable (e.g. house value) as well? (It should be.)
If all the input values are unusually high/low, is the network's output unusually high/low as well? (This only works if all the inputs are positively related to the output...if some of them are inversely related related, you will have to think a bit more).

Observe that the scheme presented here satisfies these two conditions.

Notice that this scheme would allow your network to only predict house values inside the range of house values in your training data set. Depending on the application, this behavior can be desirable or undesirable.

For example: you may want to make it impossible for your network to predict negative house values. Think about how you would do this. De-normalize the output so that -1 is mapped to 0.

If you want to set no limit on the values your network can predict, then you can run the network's output through a function that maps the [-1,1] range to all real numbers...like arctanh(x)! As long as you do this during training your network will adjust its weights to accommodate this.

I hope this was helpful. Let me know if you have further questions. My ANN module is in Python, by the way, so I might have language-specific advice.


How to deal with large (or NaN) neural network's weights? My weights go from being between 0 and 1 at initialisation to exploding into the tens of thousands in the next iteration. In the 3rd iteration they become so large that only arrays of nan values are displayed.

How can I go about fixing this?

Is it to do with the unstable nature of the sigmoid function, or is one of my equations incorrect during backpropagation which makes my gradients explode?

Thanks for any advice

import numpy as np
from numpy import exp
import matplotlib.pyplot as plt
import h5py

# LOAD DATASET
MNIST_data = h5py.File('data/MNISTdata.hdf5', 'r')
x_train = np.float32(MNIST_data['x_train'][:])
y_train = np.int32(np.array(MNIST_data['y_train'][:,0]))
x_test = np.float32(MNIST_data['x_test'][:])
y_test = np.int32(np.array(MNIST_data['y_test'][:,0]))
MNIST_data.close()

##############################################################################
# PARAMETERS
number_of_digits = 10 # number of outputs
nx = x_test.shape[1] # number of inputs ... 784 --> 28*28
ny = number_of_digits
m_train = x_train.shape[0]
m_test = x_test.shape[0]
Nh = 30 # number of hidden layer nodes
alpha = 0.001
iterations = 3
##############################################################################
# ONE HOT ENCODER - encoding y data into 'one hot encoded'
lr = np.arange(number_of_digits)
y_train_one_hot = np.zeros((m_train, number_of_digits))
y_test_one_hot = np.zeros((m_test, number_of_digits))
for i in range(len(y_train_one_hot)):
  y_train_one_hot[i,:] = (lr==y_train[i].astype(np.int))
for i in range(len(y_test_one_hot)):
  y_test_one_hot[i,:] = (lr==y_test[i].astype(np.int))

# VISUALISE SOME DATA
for i in range(5):
  img = x_train[i].reshape((28,28))
  plt.imshow(img, cmap='Greys')
  plt.show()

y_train = np.array([y_train]).T
y_test = np.array([y_test]).T
##############################################################################
# INITIALISE WEIGHTS & BIASES
params = { "W1": np.random.rand(nx, Nh),
           "b1": np.zeros((1, Nh)),
           "W2": np.random.rand(Nh, ny),
           "b2": np.zeros((1, ny))
          }

# TRAINING
# activation function
def sigmoid(z):
  return 1/(1+exp(-z))

# derivative of activation function
def sigmoid_der(z):
  return z*(1-z)

# softamx function
def softmax(z):
  return 1/sum(exp(z)) * exp(z)

# softmax derivative is alike to sigmoid
def softmax_der(z):
  return sigmoid_der(z)

def cross_entropy_error(v,y):
  return -np.log(v[y])

# forward propagation
def forward_prop(X, y, params):
  outs = {}
  outs['A0'] = X
  outs['Z1'] = np.matmul(outs['A0'], params['W1']) + params['b1']
  outs['A1'] = sigmoid(outs['Z1'])
  outs['Z2'] = np.matmul(outs['A1'], params['W2']) + params['b2']
  outs['A2'] = softmax(outs['Z2'])

  outs['error'] = cross_entropy_error(outs['A2'], y)
  return outs

# back propagation
def back_prop(X, y, params, outs):
  grads = {}
  Eo = (y - outs['A2']) * softmax_der(outs['Z2'])
  Eh = np.matmul(Eo, params['W2'].T) * sigmoid_der(outs['Z1'])
  dW2 = np.matmul(Eo.T, outs['A1']).T
  dW1 = np.matmul(Eh.T, X).T
  db2 = np.sum(Eo,0)
  db1 = np.sum(Eh,0)

  grads['dW2'] = dW2
  grads['dW1'] = dW1
  grads['db2'] = db2
  grads['db1'] = db1
#  print('dW2:',grads['dW2'])
  return grads

# optimise weights and biases
def optimise(X,y,params,grads):
  params['W2'] -= alpha * grads['dW2']
  params['W1'] -= alpha * grads['dW1']
  params['b2'] -= alpha * grads['db2']
  params['b1'] -= alpha * grads['db1']
  return

# main
for epoch in range(iterations):
  print(epoch)
  outs = forward_prop(x_train, y_train, params)
  grads = back_prop(x_train, y_train, params, outs)
  optimise(x_train,y_train,params,grads)
  loss = 1/ny * np.sum(outs['error'])
  print(loss)

```

This problem is called exploding gradients, resulting in an unstable network that at best cannot learn from the training data and at worst results in NaN weight values that can no longer be updated.

One way to assure it is exploding gradients, is if loss is unstable and not improving, or if loss shows NaN value during training.

Apart from the usual gradient clipping and weights regularization that are recommended, I think the problem with your network is the architecture.

30 is an abnormally high number of nodes for 2 layer perceptron model. Try increasing number of layers and reducing nodes per layer. - This is under the assumption that you're experimenting with MLP's, because for the problem above, convolutional neural networks seem like an obvious way to go. If unexplored - definitely check out CNN's  for digit recognition, two layer models will surely work better there.

Hope this helped!


Zilberstein's "LP-dominate" pruning explained? How does in the (famous Zilberstein) PR(uning) algorithm below the LP-dominate function get started: the first time it's called, D=∅ and the linear program deteriorates (i.e. no constraint equations)?

procedure POINTWISE-DOMINATE(w, U)
...
3. return false
procedure LP-DOMINATE(w, U)
4. solve the following linear program variables: d, b(s) ∀s ∈ S
      maximize d
      subject to the constraints
        b · (w − u) ≥ d, ∀u ∈ U
        sum(b) = 1
5. if d ≥ 0 then return b
6. else return nil
procedure BEST(b, U )
...
12. return w
procedure PR(W)
13. D ← ∅
14. while W = ∅
15.   w ← any element in W
16.   if POINTWISE-DOMINATE(w, D) = true
17.      W ← W − {w}
18.   else
19.      b ← LP-DOMINATE(w, D)
20.      if b = nil then
21.         W ← W − {w}
22.      else
23.         w ← BEST(b, W)
24.         D ← D ∪ {w}
25.         W ← W − {w}
26. return D

I think I found the solution. When in PR(W), D=∅, the weight is:

b[i] = 0          for { i | w[i]<max(w) },

and

b[i] = 1.0/max(w) for { i | w[i]==max(w) }.


Will AI develop in hybrid ways? There are two general ways in which AI can interact with humans:

Implants of AI devices into human bodies or brains — Such implantation has already begun with health monitors and could grow to include cognitive access to general purpose digital computing.  Once such is accepted, it is possible that humans will emerge with behavior largely defined by its implants therefore qualify as a hybrid AI systems.
Embedding of AI systems into business or government organizations — It is possibliites that the behavior of organizations using AI systems for business decisioning will eventually be defined in behavior largely by those systems and would therefore qualify as a hybrid AI systems.

How probable are these possibilities, considering human history and current cultural trends in technology acceptance?

These two are not "general" ways in which AI interact with humans , especially not the first one. In fact most of the applications we have complex AI algorithms deeply rooted in them , your search engine , smart-reply , translating systems all employ some kind of "AI" or to be more specific "machine-learning" algorithms in them. and many organizations are already using business-intelligence systems which you can call "AI" in some sense.


What is the "dropout" technique? What purpose does the "dropout" method serve and how does it improve the overall performance of the neural network?

Dropout means that every individual data point is only used to fit a random subset of the neurons. This is done to make the neural network more like an ensemble model.

That is, just as a random forest is averaging together the results of many individual decision trees, you can see a neural network trained using dropout as averaging together the results of many individual neural networks (with 'results' understood to mean activations at every layer, rather than just the output layer).


How to do speech recognition on a single word I would provide a sound signal of about 2-3 seconds to my neural network. I have trained my network with a single word, like if I speak "Hello" the network may tell if "Hello" is spoken or not, but some other word like "World" is spoken, it will say "Hello" is not spoken. I just want classification of sound if its a specific command or word.
What is the best way to do this, I am not a that much advanced in DNN, I only know about NN and CNN, I want to know if there is some research paper or tutorial, or need some explanation about the work.

If you have fixed length speech data you can detect the content using only CNN. You can see that problem as a binary classification (1 if the spoken word is correct, 0 otherwise).

But first, you need to make the input length is fixed. For example, you use 2 seconds as the fixed length. If the recorded speech is more than 2 seconds, you need to crop it, and if the recorded speech is less than 2 seconds you can pad it with 0 values.

Next, You can either use raw data (time-domain) or transform your data using some features extractors method (FFT, MFCC, or MFSC). Then, use CNN as you use it to classify the image. You can assume the graphic of the sound wave as a 2D image.

But, If your data have a variety of length, you can combine CNN to detect each phoneme then combine it as a sequence using RNN or HMM. You can read this method also in the mentioned papers.


Is it fair to compare AlphaGo with a Human player? A human player plays limited games compared to a system that undergoes millions of iterations. Is it really fair to compare AlphaGo with the world #1 player when we know experience increases with the increase in number of games played?

Is it fair to compare AlphaGo with a Human player?

Depends on the purpose of the comparison.

If we are comparing ability to win a game of Go, then yes.

If we are comparing learning ability, then maybe. It depends on the task. AlphaGo and systems like it are capable of learning only in well-described limited domains. There may be an analogy with sensory learning (it might even be possible in theory to take a small piece of brain tissue and run an algorithm similar to AlphaGo's learning process on it).

In general, the approach used by AlphaGo and other reinforcement learning successes is "trial-and-error plus function approximation". It seems analogous to perception and motor skills, such as object recognition or riding a bike, as opposed to reasoning skills and games as humans play them, which goes through many more cognitive and conscious layers that have no real analog in a RL system like AlphaGo.

  A human player plays limited games compared to a system that undergoes millions of iterations

This is an advantage of a machine to learn this kind of task. It would equally apply in other simulated environments with simple rules. If your goal is to have the most skilled and optimal navigation of such a domain, the implication now is that you would not train a human expert through years of study, but to write the simulator and train an AlphaGo-like machine.

This is no different a comparison than deciding cars and roads are better solutions to long distance travel for the general population than walking or horses and carts. It doesn't matter what underlies the advantage of one over the other, the assessment is cost/benefit, which resolves to a single comparable number.

It would, however, be wrong to assess AlphaGo as a better general-purpose learning engine than a human. The fact that humans do not have to work fully through millions of simulations in full detail is important. It means that something about how humans learn is still not covered by learning machines. Some of these things are understood and being discussed - such as the ability to focus intuitively on important aspects of what to learn, the ability to reason about the environment, learning analogously or transfer learning from other domains.


Is there a clustering algorithm that can make n clusters and the n+1 "others" cluster? As far as I know all clustering algorithms assume that all delivered data points have to find its cluster.
My question is, is there an algorithm that could focus only on n clusters (number stated by user) and try to dismiss the rest of the points that (according to algorithm) do not belong to n clusters, like in the picture shown below? Where we know that there are for example 2 classes that we need to cluster (red and green) and the rest (blue) we do not need in any cluster and therefore algorithm does not try to assign them to any cluster?
For example if we would have 1 000 pictures of animals, of which 200 are dogs, 200 are cats and the rest are all other animals known to men and we want to make 1 cluster for cats, 1 for dogs and maybe another for collectively all others that do not match dogs or cats.

So, I've prepared some data that resembles your sketch:
n , u = np.random.normal , np.random.uniform
x = np.concatenate([ n(1.0,0.2,100), n(3.0,0.3,100), u(0,10.0,100)])
y = np.concatenate([ n(7.0,0.4,100), n(5.0,0.3,100), u(0,10.0,100)])
# lets shuffle it a bit
idx = np.arange(x.shape[0])
np.random.shuffle(idx)
data = np.array([x,y])[:,idx]

And then I just tried using sklearn.mixture.GaussianMixture with n+1 = 3 components and default parameters:
gmm = GaussianMixture(n_components=3).fit(data.T)
cls = gmm.predict_proba(data.T).argmax(axis=1)

# Plotting
color = [['r','k','g'][i] for i in cls]
scatter(data[0],data[1],c=color, marker='.')
scatter(gmm.means_[:,0],gmm.means_[:,1],c='b',marker='o')

That seems awfully lot like the thing you've sketched.


Decision tree classifier I have seen weka j48 classifier, I want to build a classifier similar to it but I don't know how to go about it.
Can anyone advice me on how to create a classifier algorithm for decision tree?

There are a number of open source implementations of the C4.5 algorithm invented by Ross Quinlan:

Java: https://github.com/scottjulian/C4.5
Python: https://github.com/dpkravi/DecisionTreeClassifier
C++: https://github.com/zgyao/DecisionTree_C4.5
JavaScript: https://github.com/miguelmota/C4.5


How to determine if Q-learning has converged in practice？ I am using Q-learning and SARSA to solve a problem. The agent learns to go from the start to the goal without falling in the holes.
At each state, I can choose the action corresponding to the maximum Q value at the state (the greedy action that the agent would take). And all the actions connect some states together. I think that would show me a road from start to goal, which means the result converges.
But some others think that as long as the agent learns how to reach the goal, the result converges. Sometimes the success rate is very high but we cannot get the road from Q table. I don't know which one means the agent is trained totally and what the converged result means.

A typical and practical way to measure the convergence to some solution (so not necessarily the optimal one!) of any numerical iterative algorithm (such as RL algorithms) is to check if the current solution has not changed (much) with respect to the previous one. In your case, the solutions are value functions, so you could check if your algorithm has converged to some value function e.g. as follows
$$
c(q_t, q_{t-1}, \epsilon) =
\begin{cases}
1,  &\text{if } |q_t(s, a) - q_{t-1}(s, a)| < \epsilon, \forall s \in S, a \in A \\
0, & \text{otherwise}
\end{cases},
\tag{1}\label{1}
$$
where

$c$ is the "convergence" function (aka termination condition) that returns $1$ (true) if your RL algorithm has converged to some small enough neighbourhood of value functions (where those value functions are "indistinguishable"), and $0$ otherwise
$q_t$ is the value function at iteration $t$
$\epsilon$ is a threshold (aka precision or tolerance) value, which is a hyper-parameter that you can set depending on your "tolerance" (hence the name); this value is typically something like $10^{-6}$

Of course, this requires that you keep track of two value functions.
You can also define your "convergence" function $c$ in \ref{1} differently. For example, rather than using the absolute value, you could use the relative error, i.e. $\left|\frac{q_t(s, a) - q_{t-1}(s, a)}{q_t(s, a)} \right|$. Moreover, given that RL algorithms are exploratory (i.e. stochastic) algorithms, the value function may not change (much) from one iteration to the other, but, in the next one, it could significantly change because of your exploratory/behavioural actions, so you may also want to take into account more iterations, i.e. after e.g. $N > 1$ iterations, if the value function does not change much, then you could say (maybe probabilistically) that your RL algorithm has converged to some small neighbourhood of value functions in the space of value functions.
Note that these approaches do not guarantee that your RL algorithm has converged to the global optimal value function, but to some locally optimal value function (or, more precisely, small neighborhood of value functions). Q-learning is guaranteed to converge to the optimal value function in the tabular setting (your setting), but this is in the limit; in practice, it is more difficult to know if Q-learning has converged to an optimal or near-optimal value function.
Maybe you can also have a look at episodic returns of the policy derived from your final value function, but without upper and lower bounds on the optimal returns, you don't know much about the global optimality of your policy/value function.
Yes, you can check if the policy makes the agent reach the goal, but many policies could do that job, i.e. that does not say that the policy is the best (or optimal) one, i.e. that's a necessary (provided the goal is reachable and the reward function models your actual goal) but not sufficient condition (for optimality). The optimality here is usually a function of the return (given that is what you are usually trying to optimize).


What are pros and cons of Bi-LSTM as compared to LSTM? What are the pros and cons of LSTM vs Bi-LSTM in language modelling? What was the need to introduce Bi-LSTM?

I would say that the logic behind the introduction was more empirical than technical. The only difference between LSTM and Bi-LSTM is the possibility for Bi-LSTM to leverage future context chunks to learn better representations of single words. There is no special training step or units added, the idea is just to read a sentence forward and backward to capture more information.

And as trivial as the idea sounds, it works, in fact in the original paper the authors managed to achieve state-of-the-art scores in three tagging tasks, namely part-of-speech tagging, chunking and named entity recognition.

Even though it must be said that these scores were not dramatically higher compared to other models, and also the complete architecture included a Conditional Random Field on top of the Bi-LSTM.

Probably the most important aspect to stress out is that the authors performed two interesting comparison tests: one using random embedding initialisation and another one using only words (unigrams) as input features. Under these two test conditions Bi_LSTM (with CRF on top) outperformed significantly all other architectures, proving that Bi-LSTM representations are more robust than representation learned by other models.

I would like also to make a side note regarding human reading. It make sense to consider unidirectional sequence models as the most reasonable to emulate human reading, because we experience reading as a movement of the eyes that goes from one direction to the opposite. But the reality is that saccades (really rapid unconscious eye movement) and other eye movements play an enormous rule in reading. Which means that also we humans do continuously look to past and future words as well in order to understand the purpose of a word or sentence we're processing. Of course in our case these movements are directed by implicit knowledge and habits that allow us to direct our attention only to important words/parts (for example we barely read conjunctions) and it is interesting to notice that now state-of-the-art models based on transformers try to learn exactly this, where to pay attention rather than single probabilities for each word in a vocabulary.


Understanding the "unroling" step in the proof of the policy gradient theorem In the proof of the policy gradient theorem in the RL book of Sutton and Barto (that I shamelessly paste here):

there is the "unrolling" step that is supposed to be immediately clear

With just elementary calculus and re-arranging of terms

Well, it's not. :) Can someone explain this step in more detail?
How exactly is $Pr(s \rightarrow x, k, \pi)$ deduced by "unrolling"?

The unrolling step is due to the fact you end up with an equation that you can keep expanding indefinitely.
Note that we start with calculating $\nabla v_\pi(s)$ and arrive at
$$\nabla v_\pi(s) = \sum_a\left[ \nabla \pi(a|s) q_\pi(s,a) + \pi(a|s) \sum_{s'}p(s'|s,a) \nabla v_\pi (s') \right]\;,$$
which contains a term for $\nabla v_\pi(s')$. This is a recursive relationship, similar to the bellman equation, so we can substitute in a term for $\nabla v_\pi(s')$ which will be a term similar just with $\nabla v_\pi(s'')$. As I mentioned, we can do this indefinitely which leads us to
$$\nabla v_\pi(s) = \sum_{x \in \mathcal{S}} \sum_{k=0}^\infty \mathbb{P}(s\rightarrow x, k, \pi) \sum_a \nabla \pi(a|x) q_\pi(x,a)\;.$$
We need the term $\sum_{x \in \mathcal{S}} \sum_{k=0}^\infty \mathbb{P}(s\rightarrow x, k, \pi)$ because we want to take an average over the state space, however due to unrolling there are many different $s_t$'s that we need to average over (this comes from the $s',s'',s''',...$ in the unrolling) so we also need to add the probability state of transitioning from state $s$ to $x$ in $k$ time steps, where we sum over an infinite horizon due to the repeated unrolling.
If you are wondering what happens to the terms $\pi(a|s)$ and $p(s'|s,a)$ terms and why they are not explicitly shown in this final form, it is because this is exactly what the $\mathbb{P}(s\rightarrow x, k, \pi)$ represents. The average over all possible states accounts for the $p(s'|s,a)$ and the fact that we follow policy $\pi$ in the probability statement accounts for the $\pi(a|s)$.


Bayesian Perceptron: How is it compatible to Bayes Theorem? I found a very interesting paper on the internet that tries to apply Bayesian inference with a gradient-free online-learning approach: [Bayesian Perceptron: Bayesian Perceptron: Towards fully Bayesian Neural Networks.
I would love to understand this work, but, unfortunately, I am reaching my limits with my Bayesian knowledge. Let us assume that we have the weights $\mathcal{w}$ of our model and observed the data $\mathcal{D}$. Using the Bayes rule, we obtain the posterior according to $$p(\mathcal{w}|D)=\frac{p(D|\mathcal{w})p(\mathcal{w})}{p(D)}$$.
In words: we update our prior belief over our weights by multiplying the prior with the likelihood and divide everything by the evidence. In order to calculate the true posterior, we would need to calculate the evidence by marginalizing over (intergrating out) our unknown parameters. This gives the integral $$p(D) = \int p(D|\mathbf{w})p(\mathbf{w})dw$$.
So far so good. Now I refer to the paper mentioned above. Here, the approach is presented exemplarily on a neuron whose weighted sum is called $a$, which is then given to the activation function $f(.)$. Moreover it is assumed that $\mathbf{w}\sim N (\mu_w, \mathbf{C}_w)$. Because of the linearity, it can be exploited that also $\mathbf{a}\sim N (\mu_a, \mathbf{C}_a)$.
What I am confused about now is formula (14), which seems to show the compute the true posterior:
$$p(w) = \int p(a, w|D_i)da = \int p(w|a, D_i)p(a|D_i)da$$
How is this formula of the posterior compatible with the Bayes Theorem? Where is the evidence, likelihood and prior?

Thanks for asking the question. I'm the author of the paper.
The key point is that the weights $w$ cannot be updated directly with the new data as $w$ is not directly related with the output $y$ (see equation (1)). First, it is necessary to update $a$ first, which actually is directly related with $y$ via the activation function, i.e., $y = f(a)$ with $f(.)$ being the activation function (see equation (2)). Hence, you will find the Bayes rule in equation (16) with all the quantities you are missing (likelihood, prior, etc.). This equation is used to calculate the posterior $p(a|D_i)$. This posterior is then used to update the weights by plugging $p(a|D_i)$ into equation (14). The solution of equation (14), i.e., the posterior mean and covariance matrix of $w$, is provided by means of equation (22). Algorithm 2 provides a summary of all necessary calculations.
I hope this answers your question. BTW: we have extended this algorithm to a full neural network. The scientific paper describing the whole procedure is currently under review.


How to "combine" two images for CNN input (classification task)? For a classification task (I'm showing a pair of exactly two images to a CNN that should answer with 0 -> fake pair or 1 -> real pair) I am struggling to figure out how to design the input.

At the moment the network's architecture looks like this:

image-1                       image-2
   |                             |
conv layer                    conv layer
   |                             |
   _______________ _______________
                  |
            flattened vector
                  |
          fully-connected layer
                  |
           reshape to 2D image
                  |
              conv layer
                  |
              conv layer
                  |
              conv layer
                  |
            flattened vector
                  |
                output

The conv layers have a 2x2 stride, thus halfing the images' dimensions. I would have used the first fully-connected layer as the first layer, but then the size of it doesn't fit in my GPU's VRAM. Thus, I have the first conv layers halfing the size of the images first, then combining the information with a fully-connected layer and then doing the actual classification with conv layers for the combined image information.

My very first idea was to simply add the information up, like (image-1 + image-2) / 2...but this is not a good idea, since it heavily mixes up image information.

The next try was to concatenate the images to have one single image of size 400x100 instead of two 200x100 images. However, the results of this approach were quite unstable. I think because in the center of the big, concatenated image convolutions would convolve information of both images (right border of image-1 / left border of image-2), which again mixes up image information in not really senseful way.

My last approach was the current architecture, simply leaving the combination of image-1 and image-2 up to one fully-connected layer. This works - kind of (the results show a nice convergence, but could be better).

What is a reasonable, "state-of-the-art" way to combine two images for a CNN's input?

I clearly can not simply increase the batch size and fit the images there, since the pairs are related to each other and this relationship would get lost if I simply feed just one image at a time and increase the batch size.

You can combine the image output using concatenation. Please refer to this paper:

http://ivpl.eecs.northwestern.edu/sites/default/files/07444187.pdf

You can have a look at the Figure 2. And if you are using caffe, there is a layer called Concat layer. You can use it for your purpose.

I am not fully clear about what you want to do. But like you said, if you want to pass the image values from the first layer to some layers. Try reading about skip architectures.

If you want to use this network as real/fake finder, you can take the difference between two images and convert it to classification problem.

Hope it helps.


CNN Pooling layers unhelpful when location important? I'm trying to use a CNN to analyse statistical images. These images are not 'natural' images (cats, dogs, etc) but images generated by visualising a dataset. The idea is that these datasets hopefully contain patterns in them that can be used as part of a classification problem.

Most CNN examples I've seen have one of more pooling layers, and the explaination I've seen for them is to reduce the number of training elements, but also to allow for some locational independance of an element (e.g. I know this is an eye, and can appear anywhere in the image).

In my case location is important and I want my CNN to be aware of that. ie. the presence of a pattern at a specific location in the image means something very specific compared to if that feature or pattern appears elsewhere.

At the moment my network looks like this (taken from an example somewhere):

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 196, 178, 32)      896
_________________________________________________________________
activation_1 (Activation)    (None, 196, 178, 32)      0
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 98, 89, 32)        0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 96, 87, 32)        9248
_________________________________________________________________
activation_2 (Activation)    (None, 96, 87, 32)        0
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 48, 43, 32)        0
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 46, 41, 64)        18496
_________________________________________________________________
activation_3 (Activation)    (None, 46, 41, 64)        0
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 23, 20, 64)        0
_________________________________________________________________
flatten_1 (Flatten)          (None, 29440)             0
_________________________________________________________________
dense_1 (Dense)              (None, 32)                942112
_________________________________________________________________
activation_4 (Activation)    (None, 32)                0
_________________________________________________________________
dropout_1 (Dropout)          (None, 32)                0
_________________________________________________________________
dense_2 (Dense)              (None, 3)                 99
_________________________________________________________________
activation_5 (Activation)    (None, 3)                 0
=================================================================
Total params: 970,851
Trainable params: 970,851
Non-trainable params: 0
_________________________________________________________________

The 'images' I'm training on are 180 x 180 x 3 pixels and each channel contains a different set of raw data.

What strategies are there to improve my CNN to deal with this? I have tried simply removing some of the pooling layers, but that greatly increased memory and training time and didn't seem to really help.

Pooling doesn't completely remove information about location of features within your image. If you don't want to use pooling but want to reduce the size of your neural net layers you should try a value of stride greater than 1 in your convolutional layers. e.g:

keras.layers.Conv2D(filters, kernel_size, strides=(1, 1), padding='valid', data_format=None, dilation_rate=(1, 1), activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None)


Why is A* optimal if the heuristic function is admissible? A heuristic is admissible if it never overestimates the true cost to reach the goal node from $n$. If a heuristic is consistent, then the heuristic value of $n$ is never greater than the cost of its successor, $n'$, plus the successor's heuristic value.

Why is A*, using tree or graph searches, optimal, if it uses an admissible heuristic?

This is well covered in the corresponding chapter of Russell & Norvig (chapter 3.5, pages 93 to 99 (Third Edition)). Check that out for more details.
First, let's review the definitions:

Your definitions of admissible and consistent are correct.
An admissible heuristic is basically just "optimistic". It never overestimates a distance.
A consistent heuristic is one where your prior beliefs about the distances between states are self-consistent. That is, you don't think that it costs 5 from B to the goal, 2 from A to B, and yet 20 from A to the goal. You are allowed to be overly optimistic though. So you could believe that it's 5 from B to the goal, 2 from A to B, and 4 from A to the goal.
A tree search is a general search strategy for searching problems that have a tree structure: that is, it's never possible to "double back" to an earlier state from a later state. This models certain types of games, for instance, like Tic-Tac-Toe. The tree search does not remember which states it has already visited, only the "fringe" of states it hasn't visited yet.
A graph search is a general search strategy for searching graph-structured problems, where it's possible to double back to an earlier state, like in chess (e.g. both players can just move their kings back and forth). To avoid these loops, the graph search also keeps track of the states that it has processed.
For more on tree vs. graph search, see this answer.

Okay, now let's talk through the intuition behind the proofs.
We first want to show that

If $h(n)$ is admissible, A* using tree search is optimal.

The usual proof is by contradiction.

Assume that A* with tree search and an admissible heuristic was not optimal.

Being non-optimal means that the first complete path from the start to the goal discovered by A* (call this $q$) will be longer than some other path $p$, which A* explored up to some state $s$, but no further.

Since the heuristic is admissible, the estimated cost of reaching the goal from $s$ must be smaller than the true cost.

By 3, and the fact that we know how much it costs to reach $s$ along $p$, the estimated total cost of $p$, and thus the cost to expand $s$ must be smaller than the true cost of $p$.

Since the true cost of $p$ is smaller than the cost of $q$ (by 2), the estimated cost to expand $s$ must be smaller than the true cost of $q$.

A* always picks the path with the most promising total cost to expand next, and the cost of expanding the goal state is given by the total path length required to reach it.

5 and 6 form a contradiction, so our assumption in 1 must have been incorrect. Therefore A* must be optimal.

The graph search proof uses a very similar idea, but accounts for the fact that you might loop back around to earlier states.


Do I need to denormalise results in linear regression? I have learned so far how to linear regression with one or multiple features. So far, so good, everything seems to work fine, at least for my first simple examples.

However, I now need to normalise my features for training. I'm doing this by calculating the mean and the standard deviation per feature, and then calculate the normalised feature by subtracting the mean, taking the absolute value, and dividing by the standard deviation. Again, so far, so good, the results of my tensors which I use for training look good.

I understand why I need to normalise input data, and I also understand why one can do it like this (I know that there are other ways as well, e.g. to map values to a 0-1 interval).

Now I was wondering about two things:

First, after having trained my network, when I want to make a prediction for a specific input – do I need to normalise this as well, or do I take use the un-normalised data? Does it make a difference? My gut feeling says, I should normalise it, as it should make a difference, but I'm not sure. What should I do here, and why?
Second, either way, I get a result. Now I was wondering whether I need to denormalise this? I mean, it should make a difference, shouldn't it? If so, how? How do I get from the normalised result value to a denormalised one? Do I just need to reverse the calculation with mean and standard deviation, to get the actual value?

It would be great if someone could shed some light on this.

Okay, I figured it out by myself, simply by trial and error:

When you normalize your training and test data, you also need to normalize the input you want to have a prediction for. You will also need to denormalize the result, to get a reasonable prediction.

Specifically, in my case, this means:

First I calculated the mean and the standard deviation, and then subtracted the mean and divided by standard deviation, to normalize my training and test data.
So to normalize the input I want to have a prediction for, I also subtract the mean and divide by the standard deviation.
Finally, when I get a result, I multiple by the standard deviation, and add the mean.

This way, I get reasonable results


What exactly are the differences between semantic and lexical-semantic networks? What exactly are the differences between semantic and lexical-semantic networks?

According to Wikipedia  and
Semantic Networks
 ,By John F. Sowa
This is an updated version of an article in the Encyclopedia of Artificial Intelligence, Wiley, 1987, second edition, 1992.

  A semantic network is used when one has knowledge that is best
  understood as a set of concepts that are related to one another.

From Wikipedia:

  Most semantic networks are cognitively based. They also consist of arcs and nodes which can be organized into a taxonomic hierarchy. Semantic networks contributed ideas of spreading activation, inheritance, and nodes as proto-objects.

An example of semantic network is WordNet. It groups English words into synsets. Some semantic relations between these synsets are meronymy, hyponymy, etc.

On the other hand, Lexical semantic:

  ...explores whether the meaning of a lexical unit is established by looking at its neighborhood in the semantic net (words it occurs with in natural sentences), or whether the meaning is already locally contained in the lexical unit.


Can a brain be intelligent without a body? The dialog context

Turing proposed at the end of the description of his famous test, "Will the interrogator decide wrongly as often when the game is played like this as he does when the game is played between a man and a woman? These questions replace our original, 'Can machines think?'"1

Turing effectively challenged the 1641 statement of René Descartes in his Discourse on Method and Meditations on First Philosophy:

  "It never happens that [an automaton] arranges its speech in various ways, in order to reply appropriately to everything that may be said in its presence, as even the lowest type of man can do."

Descartes and Turing, when discussing automatons achieving human abilities, shared a single context through which they perceived intelligence.  Those that have been either the actor or the administrator in actual Turing Tests understand the context: Dialog.

Other contexts2

The context of the dialog is distinct from other contexts such as writing a textbook, running a business, or raising children. If you apply the principle of comparing machine and human intelligence to automated vehicles (e.g. self-driving cars), an entirely different context becomes immediately apparent.

Question

Can a brain be intelligent without a body? More generally, does intelligence require a context?

References

[1] Chapter 1 ("Imitation Game") of Computing Machinery and Intelligence, 1951.

[2] Multiple Intelligences Theory

Can a brain be intelligent without a body?

No. Don't forget that the main function of the brain is to provide homeostasis between the body and the environment. Without the body, the utility of the brain is no longer relevant.

Alternatively, why consider intelligence only in the brain? How far does our body extend? Embodied cognitive science asks us to consider our entire body and its
 surrounding extensions utility on the intelligent faculty; The fact that our thumbs stick out in a direction different to our other four fingers allows us to grab things in a effortless way is itself, intelligent.

From this understanding then, the segregation between the intelligent faculty and the "non-intelligent" seems to be murky at best. We'd might as well not consider it unless there is better motivation.

  Does intelligence require a context?

Pragmatically speaking, yes. Intelligent behaviour is typically understood as being goal-directed and intentional. Intentionality implies some sort of agency. It being an agent, implies some sort of agent/environment relationship, which implies an environment to act as context.

On the other hand, Karl Friston notes in his review that a general principle that the brain (and thus one would imagine, intelligent behaviour) entails is reduction of thermodynamic free energy while maintaining homeostasis with its environment.

This hints at a promise to describe intelligent behaviour purely in terms of thermodynamic processes, but interpretively what this means in a general language is still unclear.

Also, keep in mind that there is no guarantee that we'd recognize a agent as intelligent if its construction is radically different from ours.


Equivalence of the $Q(s,a)$ and $V(s)$ under optimality conditions? Are the state-action values and the state value function equivalent for a given policy? I would assume so as the value function is defined as $V(s)=\sum_a \pi(a|s)Q_{\pi}(s,a)$. If we are operating a greedy policy and hence acting optimally, doesn't this mean that in fact the policy is deterministic and then $\pi(a|s)$ is $1$ for the optimal action and $0$ for all others? Would this then lead to an equivalence between the two?
Here is my work to formulate some form of proof where I start with the idea that a policy is defined to be better than a current policy if for all states then $Q_{\pi}(S,\pi^∗(s))\geq Vπ_{\pi}(s)$ :
I iteratively apply the optimal policy to each time step until I eventually get to a fully optimal time step of rewards
$$Vπ_{\pi}(s)≤Q_{\pi}(S,\pi^∗(s))$$
$$=Eπ[R_{t+1}+\gamma V_{\pi}(St+1)|St=s]$$
$$\leq E[Rt+1+\gamma Q_{\pi}(S_{t+1},\pi^∗(S_{t+1})|S_t=s]$$
$$\leq E[Rt+1+\gamma Rt+2+\gamma 2Q \pi^*(S_{t+2},\pi^∗(S_{t+2})|S_t=s]$$
$$\leq E[R_{t+1}+\gamma R_{t+2}+....|S_t=s]$$
$$=V\pi^∗(s)$$
I would say that our final two lines are in fact inequalities, and for me this makes intuitive sense in that if we are always taking a deterministic greedy action our value function and Q function are the same. As detailed here, for a given policy and state we have that $V(s)=\sum_a \pi(a|s)Q_{\pi}(s,a)$ and if the policy is optimal and hence greedy then $\pi(a|s)$ is deterministic.

In general they are not the same and that should be clear as to why -- mathematically you are conditioning on an extra random variable being known in the state-action value function. You have the correct relationship between them, but I think your understanding of the two may be slightly off. The state-action value function is a function of both $s$ and $a$ whereas the value function is a function of just $s$. As you have noted, the value function is equal to the expected value of the state-action value function, with the expectation taken over the action distribution induced by the policy.
However, if you have a deterministic policy, then two are equivalent only if you evaluate the state-action value function at the action that the deterministic policy gives. That is, $v_\pi(s) = Q(s, \pi(s))$. This is because, as you say, for a given state the policy will assign probability one to a certain action and 0 to all other actions in the action space and so the equation I just wrote is what the expectation in your question reduces to.
Now, you might think 'well the policy is deterministic so we are always going to choose this action, so my theory was correct'. This is not true. The definition of the state-action value function is the value of the expected future (possibly discounted) returns given that we are in state $s$ and given that we have taken action $a$. This means that we have already chosen our action and we are not choosing it according to the policy -- the policy will only choose all future actions.
You also say that "If we are operating a greedy policy and hence acting optimally" which is not true. Just because we act greedily it does not imply we are acting optimally. Value functions are functions of a policy. If the policy is completely random then you would get certain value and state-action value functions which you could act greedily to but this would not be optimal.


How do I know how changes in the weights are changing the reward in Reinforcement Learning I already know the basics of the basic of Machine Learning. E.g.: Backpropagation, Convolution, etc.

First of let me explain Reinforcement learning to make sure I grasped the concept correctly.

In Reinforcement learning a random-initialized network will first "play"/"do" a sequence of moves in an environment. (In this case a Game). After that, it will receive a reward $r$. Furthermore a q-Value gets defined by the engineer/hooby coder. This reward times the q-Value $q$ to the power of the  position $n$ of the action will be feeded back using BP.

So how do I know how slight chances in $\vec{w}$ are changing $rq^n$?

You have the concept slightly wrong.

This part is mostly correct:

  In Reinforcement learning a random-initialized network will first "play"/"do" a sequence of moves in an environment. (In this case a Game). After that, it will receive a reward r.

Technically neural networks are not required in RL, and it is really worth studying some simple systems that don't need them. It will make everything much clearer.

A reward $r$ can be received on every time step. However, some environments will only have a single reward at the end for success or failure for a whole episode - e.g. an instance of a game like chess where a player wins or loses.

This part is where things go a bit off track:

  Furthermore a q-Value gets defined by the [developer]. This reward times the q-Value q to the power of the position n of the action will be [fed] back using BP.

Q values are one type of data that can be calculated for an agent acting in a Markov Decision Process. They are also called "action values" and they are not usually defined by a developer. The q value, if correct should return the expected future sum of rewards from following a current policy. One way of writing this is:

$$q(s,a) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty}\gamma^k R_{t+k+1}| S_t=s, A_t=a]$$

In natural language, the q value for state s and action a is the expected value (when following the given policy) of the discounted sum of rewards, starting from the given state and action. The discount factor, $\gamma$ can take any value from $0$ up to $1$, but only strictly episodic problems (which always terminate) should use the value $1$

A developer does not get to define that (except they might get to choose reward system and value of $\gamma$). Instead, they need to implement something that estimates the value of $q(s,a)$ based on what the agent has experienced. There are a few different algorithms that can do this. A popular one is called Q learning.

Regarding "[fed] back using BP", this is correct if you are using a neural network. Typically in DQN (Q learning with neural networks), this just consists of creating a small sample of training data from recent experience and training the neural network almost identically to supervised learning.

  So how do I know how slight chances in $\vec{w}$ are changing $rq^n$?

Definitely don't use $rq^n$ - there is no purpose to that quantity in RL. Instead for value-based RL, you are mostly interested in your estimate for Q value. This might be written $\hat{q}(s,a,\vec{w}) \approx q(s,a)$.

However, in general your question stands. If you have implemented a neural network to learn q values, how do you know if it is working?

There are actually two parts to this problem:

How do you know whether the agent is getting better at it's task?
How do you know whether the q values are getting more accurate?

What you need to do is measure, and maybe plot some relevant quantities.

For the first question, you would typically plot the total reward that the agent gets each episode. This will be noisy, so it is a good idea to smooth it out by taking some kind of moving average (e.g. average total reward over last 100 episodes).

For the second question, it is normal to plot some loss function of the network, just like supervised learning. Typically this is Mean Squared Error loss, as the network is learning a regression to predict q values given $s$ and $a$.  You can compare observed sums of discounted reward (aka "return" or "utility") with the earlier predicted ones, and take the error function. You need to get some measure of a "true" value of q - usually a noisy sample taken during training or testing, and measure loss. For MSE that might be

$$J(\vec{w}) = \frac{1}{2|D|}\sum_{(s,a) \in D}(\hat{q}(s,a, \vec{w}) - q(s,a))^2$$

Where $D$ is some dataset you have put together of $s,a$ and $q(s,a)$ measurements to test with. If this looks familiar to you from supervised learning MSE loss, then that's correct - it is essentially the same thing, just different how you go about collecting the data.

You may expect the loss function for $\hat{q}$ in Q learning to be somewhat unstable as the agent learns. That's because in Q learning, the policy is updating at the same time as the estimates are improving. Which makes the estimates out-of-date. However, it should still be possible to see a reduction in error as learning progresses. If it becomes stable at a relatively low value compared to initially, then the agent has probably learned all that it can - although sometimes new discoveries by the agent can open up more improvements, even late in training, and throw the error function out again.

Note that a low value of the error function does not mean you have an optimal agent. It means that the value function estimate is good for how the agent is currently behaving. In turn that means the agent cannot make further improvements without new and different experience.


Do we also need to model a probability distribution for the decoder of a VAE? I'm working on understanding VAEs, mostly through video lectures of Stanford cs231n, in particular lecture 13 tackles on this topic and I think I have a good theoretical grasp.

However, when looking at actual code of implementations, such as this code  from this blog of VAEs I see some differences which I can't quite understand.

Please take a look at this VAE architecture visualization from the class, specifically the decoder part. From the way it is presented here I understand that the decoder network outputs mean and covariance for the data distribution. To get an actual output (i.e. image) we need to sample from the distribution that is parametrized by mean and covariance - the outputs of the decoder.

Now if you look at the code from the Keras blog VAE implementation, you will see that there is no such thing. A decoder takes in a sample from latent space and directly maps its input (sampled z) to an output (e.g. image), not to parameters of a distribution from which an output is to be sampled.

Am I missing something or does this implementation not correspond to the one presented in the lecture? I've been trying to make sense of it for quite some time now but still can't seem to understand the discrepancy.

Thanks @nbro for pointing this out.

The pictorial architecture in the slides uses the Gaussian loss, which when coupled with Maximum Likelihood Estimation gives the squared error loss (not to remove any tractability issues). The main reason we do the encoder Gaussian trick is to force the latent variable $z$ to be normal so that we can apply $KL$ $Divergence$ to optimise an otherwise intractable integral. You can get a better intuition and reasoning in this video.

The pictorial architecture is basically taking the Gaussian loss so that the final loss becomes the squared error loss effectively. Also the loss term used in your blog link is exactly the same loss term used in the original paper, but the blog is using CE loss (it is the more common loss used for classification). I am not sure how they are using the CE loss though, since it is only valid for $0$ and $1$ values and AFAIK MNIST data-set has grayscale images.

I am not exactly sure how they implement the randomness of the Gaussian loss in the decoder structure, but in the simplest of cases they just take the MSE

Check out this blog on VAE's (where they have taken the mean $\Sigma$ which they have abbreviated as mean, I have not checked their implementation detail to know what they exactly mean by that) and also this answer on Data Science on implementation of VAE's4 (both of which gives a more general form of loss). Also for the exact Mathematics check out Appendix C of the original paper.


How can I reduce the GPU memory usage with large images? I am trying to train a CNN-LSTM model. The size of my images is 640x640. I have a GTX 1080 ti 11GB. I am using Keras with the TensorFlow backend.

Here is the model.

img_input_1 = Input(shape=(1, n_width, n_height, n_channels))
conv_1 = TimeDistributed(Conv2D(96, (11,11), activation='relu', padding='same'))(img_input_1)
pool_1 = TimeDistributed(MaxPooling2D((3,3)))(conv_1)
conv_2 = TimeDistributed(Conv2D(128, (11,11), activation='relu', padding='same'))(pool_1)
flat_1 = TimeDistributed(Flatten())(conv_2)
dense_1 = TimeDistributed(Dense(4096, activation='relu'))(flat_1)
drop_1 = TimeDistributed(Dropout(0.5))(dense_1)
lstm_1 = LSTM(17, activation='linear')(drop_1)
dense_2 = Dense(4096, activation='relu')(lstm_1)
dense_output_2 = Dense(1, activation='sigmoid')(dense_2)
model = Model(inputs=img_input_1, outputs=dense_output_2)

op = optimizers.Adam(lr=0.00001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.001)

model.compile(loss='mean_absolute_error', optimizer=op, metrics=['accuracy'])

model.fit(X, Y, epochs=3, batch_size=1)

Right now, using this model, I can only use the training data when the images are resized to 60x60, any larger and I run out of GPU memory.

I want to use the largest possible size as I want to retain as much discriminatory information as possible. (The labels will be mouse screen coordinates between 0 - 640).

Among many others, I found this question: How to handle images of large sizes in CNN?

Though I am not sure how I can "restrict your CNN" or "stream your data in each epoch" or if these would help.

How can I reduce the amount of memory used so I can increase the image sizes?

Is it possible to sacrifice training time/computation speed in favor of higher resolution data whilst retaining model effectiveness?

Note: the above model is not final, just a basic outlay.

As the other links suggest, you basically have four options:

"restrict your CNN". This means making your model smaller and simpler, possibly by inserting a pooling layer at the front, or reducing the total number of layers. From a memory perspective, this isn't likely to produce really large gains though.
"stream your data in each epoch". By default, the entire training set will be stored on the GPU. This is a good idea, because the bus connecting the GPU and the RAM has extremely high latency. It takes a very long time to start sending data across the bus. However, most systems have much more space in the RAM than in the video memory on a GPU. Instead of storing all the training data in the GPU, you could store it in main memory, and then manually move over just the batch of data you want to use for a given update. After computing the update, you could free the memory assigned to the batch. I am not sure how to do this in Keras. In the past, I have done this by writing a custom CUDA kernel.
"Use less data". If you train on a random subset of the training data, you can keep your images at high quality, but your model will probably overfit. If you train on downsampled images, your model may not be able to discriminate between them well. However, both of these options are easy to do.
"Get more memory". If you buy a video card with more video RAM, you can train more complex models. This is why scientific-grade cards have much larger memories than gaming cards. The Tesla V100 has 32GB of memory, and 16GB cards are common, whereas even the most advanced cards for gaming have only 11GB.


Can abstractive summarization be achieved using neural networks? Text summarization is a long-standing research problem that was "ignited" by Luhn in 1958. However, a half century later, we still came nowhere close  to solving this problem (abstractive summarization). The reason for this might be because researchers are resorting to statistical (and sometimes linguistic) methods to find & extract the most salient parts of the text.

Is summarization problem solvable using AI (neural networks to be precise)?

The ability to re-frame summarization as a problem for ANN is rather dependent on what kind of output you're looking for: you mentioned 'salient parts of the text'.

One possibly is to use a deep learning approach that first chunks together words that belong in the same phrase as a single 'feature'.

Another possibility is to identify both key words and relations between them. Here is some previous work on using neural nets for relational learning.


Is it possible for a NN to reach the same results as CNNs? Can a normal neural network work as good as a convolutional network? If yes, how much more time and neurons would it need compared to a CNN?

NNs won't be able to reach the performance of CNNs, in general.

By a Neural Network, I am assuming you're pointing to 'vanilla' vector based neural networks.

Let's take the MNIST dataset, it performs almost similar in both NNs and CNNs; the reason being digits of almost same size, and similar spatial drawings. To put it easy, all the digits roughly take up the same area as the other 60k. If you were to zoom in our zoom out these digits, the trained NN may not perform well.

Why?

Plain NNs lack the ability to extract 'position independent' features. A cat is a cat to your eye, no matter whether you saw it in the center of an image, or left corner. CNNs use 'filters' to extract 'templates' of cat. Hence CNNs can 'localize' what it is searching for.

You could say, NNs look at the 'whole' data to make a sense, CNNs look at the 'special' features at certain parts to make a sense.

This is also a reason why CNNs are popular. NNs are suited for their own applications. Every input node in a vectorized Neural Networks represent a feature, and that feature is tied to a purpose throughout training, i.e. its position is fixed throughout the training.


Can artificial intelligence create the next wonder material? The theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages.

Can artificial intelligence create the next wonder material?

Artificial intelligence has already gotten to the point where machines are as good as humans at many tasks, and even outperform humans at several more. The general goal of AI is to improve machines to be as good or better than humans at a wide variety of tasks. There's no theoretical reason why materials science is any different; we may well someday have a machine that can out-think all human materials scientists. There's not much reason to suppose that humans will always reign supreme at any particular task (especially objective ones like science), so yes, it's entirely possible that the next "wonder material" will be invented by a machine.


Trained a regression network and getting EXACT same result on validation set, on every epoch I trained this network from this github.

The training went well, and returns nice results for new, unseen images.

On training, the loss changed (decreased), thus I must assume the weights changed as well.

On training, I saved a snapshot of the net every epoch.

When trying to run a validation set through each epoch's snapshot, I get the exact same results on every epoch.

How can this be possible? What's causing this?

The answer is I was too tired to see I was using the same model for validation every epoch, instead of that epoch's model.

Write good code, and this won't happen to you.


How to stop DQN Q function from increasing during learning? Following the DQN algorithm with experience replay:
Store transition $\left(\phi_{t}, a_{t}, r_{t}, \phi_{t+1}\right)$ in $D$ Sample random minibatch of transitions $\left(\phi_{j}, a_{j}, r_{j}, \phi_{j+1}\right)$ from $D$ Set
$$y_{j}=\left\{\begin{array}{cc}r_{j} & \text { if episode terminates at j+1} \\ r_{j}+\gamma \max _{d^{\prime}} \hat{Q}\left(\phi_{j+1}, a^{\prime} ; \theta^{-}\right) & \text {otherwise }\end{array}\right.$$
Perform a gradient descent step on $\left(y_{j}-Q\left(\phi, a_{j} ; \theta\right)\right)^{2}$ with respect to the network parameters $\theta$.

We calculate the $loss=(Q(s,a)-(r+Q(s+1,a)))^2$.

Assume I have positive but changing rewards. Meaning, $r>0$.

Thus, since the rewards are positive, by calculating the loss, I notice that almost always $Q(s)< Q(s+1)+r$.
Therefore, the network learns to always increase the $Q$ function , and eventually, the $Q$ function is higher in same states in later learning steps.
How can I stabilize the learning process?

You can use discount factor gamma less then one.
You can use finite time horizon - only for states which are no farther away then T time steps reward propagate back
You can use sum of rewords averaged over time for Q

All of those are legitimate approaches.


AI applications of the Fibonacci series I have been looking at Fibonacci series, the golden ratio and its uses in nature, like how flowers and animals grow based on the series.

I was wondering whether we could use the Fibonacci series and the golden ratio in any way in AI, especially in evolutionary algorithms. Any ideas or insights?

Is this research material? If so where can we start?

The use of the Golden Ratio is an interesting suggestion which has intrigued many lovers of the mathematical beauty represented in nature and in AI. The problem lies in the foundations of the AI applications. For example, in designing algorithms for recognizing naturally occurring phenomena, such as face recognition or human body movements (See  https://www.intechopen.com/books/machine-learning-and-biometrics/a-human-body-mathematical-model-biometric-using-golden-ratio-a-new-algorithm) it is suitable. However, for non natural occurrences, the ratio is limited since the data is usually random or chaotic. However, in order to create a master algorithm for the future which encompasses all the best of the current AI algorithms, the use of mathematical concepts such as the golden ration and fractals will be vital. Watch this space...