Which layer in a CNN consumes more training time: convolution layers or fully connected layers? In a convolutional neural network, which layer consumes more training time: convolution layers or fully connected layers?

We can take AlexNet architecture to understand this. I want to see the time breakup of the training process. I want a relative time comparison so we can take any constant GPU configuration.

NOTE: I did these calculations speculatively, so some errors might have crept in. Please inform of any such errors so I can correct it.

In general in any CNN the maximum time of training goes in the Back-Propagation of errors in the Fully Connected Layer (depends on the image size). Also the maximum memory is also occupied by them. Here is a slide from Stanford about VGG Net parameters:

Clearly you can see the fully connected layers contribute to about 90% of the parameters. So the maximum memory is occupied by them.

As far as training time goes, it somewhat depends on the size (pixels*pixels) of the image being used. In FC layers it is straightforward that the number of derivatives you have to calculate is equal to the number of parameters. As far as the convolutional layer goes, let's see an example,let us take the case of 2nd layer:
It has got 64 filters of $(3*3*3)$ dimensions to be updated. The error is propagated from the 3rd layer. Each channel in the 3rd layer propagates it error to its corresponding $(3*3*3)$ filter. Thus $224*224$ pixels will contribute to about $224*224*(3*3*3)$ weight updations. And since there are $64$ such $224*224$ channels we get the total number of calculations to be performed as $64*224*224*(3*3*3) \approx 87*10^6 $ calculations.

Now let us take the last layer of $56*56*256$. It will pass it gradients to the previous layer. Each $56*56$ pixel will update a $(3*3*256)$ filter. And since there are 256 such $56*56$ the total calculations required would be $256 * 56 * 56 * (3*3*256) \approx 1850 *10^6$ calculations.

So number of calculations in a convolutional layer really depends on the number of filters and the size of the picture. In general I  have used the following formulae to calculate the number of updations required for filters in a layer, also I have considered $stride = 1$ since it will be the worst case:

$channels_{output} * (pixelOutput_{height} * pixelOutput_{width}) * (filter_{height} * filter_{width} * channels_{input})$

Thanks to fast GPU's we are easily able to handle these huge calculations. But in FC layers the entire matrix needs to be loaded which causes memory problems which is generally not the case of convolutional layers, so training of convolutional layers is still easy. Also all these have to be loaded in the GPU memory itself and not the RAM of the CPU.

Also here is the parameter chart of AlexNet:

And here is a performance comparison of various CNN architectures:

I suggest you check out the CS231n Lecture 9 by Stanford University for better understanding of the nooks and crannies of CNN architectures.


Is time/space estimation of possible actions required for creating AGI? Given infinite resources/time one could create AGIs by writing code to simulate infinite worlds. In some of the worlds AGIs would be created. Detecting them would be another issue.

Since we don't have infinite resources the most probable way to create an AGI is to write some bootstrapping code which would reduce the resources/time to reasonable values.

  In that AGI code (that would make it reasonable to create with finite
  resources/time) is it required to have a part that deals with
  time/space estimation of possible actions taken? Or should that be outside of the code and be something the AGI discovers by itself after it starts running?

Any example of projects targeting AGI that are using time/space estimation might be useful for reaching a conclusion.

Clarification, by time/space I mean time/space complexity analysis for algorithms, see: Measures of resource usage and Analysis of algorithms

The question formulation might lead people to think that the time/space estimation can only apply to some class of actions called algorithms. To clarify my mistake, I mean the estimation to apply to any action plan.

Imagine you are an AGI and you have to make a choice between different set of actions to pursue your goals. If you had 2 goals and one of them used less space and less time then you would always pick it over the other algorithm. So time/space estimation is very useful since intelligence is about efficiency.

There is at least 1 exception though, imagine in the example before that the goal of the AGI is to pick the set of actions that leads to the most expensive time/space set of actions (or any non-minimal time/space cost) then obviously because of the goal constraint you would pick the most time/space expensive set of actions. In most other cases though, you would just pick the most time/space efficient algorithm.

Time - probably yes; Space - probably no

Every single thing could be time dependent but there is no such thing on which time depends except considering space (black holes, bending of space time), Now we can argue that it can compare memories of present to its previous memories relatively and then call this relativeness time but even there the AGI is putting a time stamp when calling something previous.
Now lets consider how time could be dependent upon space. In this case we can see things getting compared to how much light travels like how the SI unit of time is defined, but that is a really complex observation for a discrete system and it has to go through a lot of time dependent phenomena to reach the level to make the observation.

Space - We can reduce this question by saying that if it could learn the concept of 1-D it could learn about space itself. 1-D space can be compared to a number line, assuming the agi knows the concept of numbers it can compare the differences in location with a number line .And if agi knows the concept of time  it can learn space estimation by always assigning velocity to objects.
The difference between time and space is, time is always moving and there is nothing to compare to.

P.S. Sorry for bad english, english is not my first language.


Reinforcement learning with action consisting of two discrete values I'm new to reinforcement learning. I have a problem where an action is composed of an order (rod with a required length) and an item from a warehouse (an existing rod with a certain length, which will be cut to the desired length and the remainder put back to the warehouse).
I imagine my state as two lists of a defined size: orders and warehouse, and my action as an index from the first list and an index from the second list. However, I have only worked with environments where it was only possible to pick single action and I'm not sure how to deal with two indexes. I'm not sure how DQN architecture should look like to give me such action.
Can anyone validate my general idea and help me find a solution? Or maybe just point me to some papers where similar problems are described?

You would still be picking a single action. Your action space is now $\mathcal{A} = \mathcal{O} \times \mathcal{I}$ where I've chosen $\mathcal{O}$ to be the set of possible orders from your problem and $\mathcal{I}$ to be the set of possible items.
Provided both of these sets are finite, then you should still be able to approach this problem with DQN. Theoretically, this should be easy to see, as any element from $\mathcal{A}$ is still a single element it just happens that this element is now a tuple.
From a programming point of view, let's consider the simple example of cartpole, where the possible actions are left and right. Your $Q$-function obviously won't know the meanings of 'left' and 'right', you just assign it to an element of a vector, i.e. your $Q$-function would output a vector in $\mathbb{R}^2$ with e.g. the first element corresponding to the score for 'left' and the second element corresponding to the score for 'right'. This is still the case in your problem formulation, you will just have a $Q$-function that outputs a vector in $\mathbb{R}^d$ where $d = |\mathcal{A}|$ - you would just have to make sure you know which element corresponds to which action.
Also, there is the possibility that this approach could leave you with a large dimensional vector output, which I imagine would probably mean you'd need more simulations to properly explore the action space.
Hope this helps.


LSTM network doesn't converge, what should be changed? I'm testing out TensorFlow LSTM layer text generation task, not classification task; but something is wrong with my code, it doesn't converge. What changes should be done?

Source code:

import tensorflow as tf;

# t=0  t=1    t=2  t=3
#[the, brown, fox, is,     quick]
#   0  1      2    3       4
#[the, red,   fox, jumps,  high]
#   0  5      2    6       7

#t0 x=[[the],  [the]]
#   y=[[brown],[red]]
#t1 ...
#t2
#t3
bsize = 2;
times = 4;

#data
x = [];
y = [];
#t0        the:     the:
x.append([[0/6],   [0/6]]); #normalise: x divided by 6 (max x)
#          brown:   red:
y.append([[1/7],   [5/7]]); #normalise: y divided by 7 (max y)
#t1
x.append([[1/6],   [5/6]]);
y.append([[2/7],   [2/7]]);
#t2
x.append([[2/6],   [2/6]]);
y.append([[3/7],   [6/7]]);
#t3
x.append([[3/6],   [6/6]]);
y.append([[4/7],   [7/7]]);

#model
inputs  = tf.placeholder(tf.float32,[times,bsize,1]) #4,2,1
exps    = tf.placeholder(tf.float32,[times,bsize,1]);

layer1  = tf.keras.layers.LSTMCell(20)
hids1,_ = tf.nn.static_rnn(layer1,tf.split(inputs,times),dtype=tf.float32);

w2      = tf.Variable(tf.random_uniform([20,1],-1,1));
b2      = tf.Variable(tf.random_uniform([   1],-1,1));
outs    = tf.sigmoid(tf.matmul(hids1,w2) + b2);

loss  = tf.reduce_sum(tf.square(exps-outs))
optim = tf.train.GradientDescentOptimizer(1e-1)
train = optim.minimize(loss)

#train
s    = tf.Session();
init = tf.global_variables_initializer();
s.run(init)

feed = {inputs:x, exps:y}
for i in range(10000):
  if i%1000==0:
    lossval = s.run(loss,feed)
    print("loss:",lossval)
  #end if
  s.run(train,feed)
#end for

lastloss = s.run(loss,feed)
print("loss:",lastloss,"(last)");
#eof

Output showing loss values (a little different every run):

loss: 3.020703
loss: 1.8259083
loss: 1.812584
loss: 1.8101325
loss: 1.8081319
loss: 1.8070083
loss: 1.8065354
loss: 1.8063282
loss: 1.8062303
loss: 1.8061805
loss: 1.8061543 (last)

Colab link:
https://colab.research.google.com/drive/1TsHjmucuynCPOgKuo4a0hiM8B8UaOWQo

writing here my suggestion, because i haven't earned the right to comment yet.

Your main "problem" could be your loss function. It converges, this is why your loss value is decreasing. So I suggest to let it maybe train longer.

Alternatively you could change the loss function to fit your need. For example you could use:

loss  = tf.reduce_mean(tf.square(exps-outs))

You will get a smaller loss value which decreases clearly after every output.

I hope this helps :)


The connection between number of layer of DNN and computational complexity of it number of layer of DNN and computational complexity of it are correlated after optimization, but how to estimate it before designing DNN?

The computational complexity of DNNs is based on 3 main factors.

Matrix Multiplication
Non linear transformation
Weight sharing

Matrix multiplication is the fundamental operation when computing the forward and backward passed in DNNs if using back-propagation. As the complexity of matrix multiplications becomes more expensive as the size of matrices become larger, understanding how to construct networks with effectively sized later which balance the time complexity with accuracy becomes imperative.

Nonlinear transformations allow DNNs to learn nonlinear functions. It is a very important aspect and has been studied rigorously. Classically, the functionf(x) = 1(1+exp(x)) was used to squash outputs from linear layers to be a nonlinear output. However this function has recently been replaced in many applications with the linear rectifier unit (ReLU) f(x)=max(x,0). The relu is much much faster to compute and doesn’t seem to affect the end performance substantially or even noticeably in some situations.

Weight sharing is the idea that some weights in a DNN must share the same value. Beyond the theoretical reasons why this is chosen, it also decreases the number of values that must be updated when performing back-propagation. This is the reason why Convolutional NNs are orders of magnitude faster than their non-convolutional counter parts for image recognition tasks.

There are other things to be aware of when trying to analyze the computational complexity of DNNs but they usually relate to one of the 3 items above.

To estimate the complexity, counting the number of matrix multiplications and their matrix sizes, add time for nonlinearities and you should have a pretty good estimate.


How to deal with evolutionary/genetic fitness function that can have both negative and positive values? I am optimising function that can have both positive and negative values in pretty much unknown ranges, might be -100, 30, 0.001, or 4000, or -0.4 and I wonder how I can transform these results so I can use it as a fitness function in evolutionary algorithms and it can be optimised in a way that, for example, it can go from negative to positive along the optimisation process (first generation best chromosome can have -4.3 and best at 1000 generation would have 5.9). Although the main goal would always be to maximise the function.
Adding a constant value like 100 and then treating it simply as positive is not possible because like I said, the function might optimise different ranges of results in different runs for example (-10000 to +400 and in another run from -0.002 to -0.5).
Is there a way to solve this?

If I understand correctly your problem, you always want to maximize some function $h(x)$, which is defined as follows $h: \Gamma \rightarrow \mathbb{R}$, where $\Gamma$ is the space of genotypes. However, at every generation $g$, you don't know exactly $h(i)$ of each individual $i$, i.e., maybe in one generation $g$, all $h(i)$, for $i=1, \dots, N$ (where $N$ is the size of the population), are negative, but in the next generation $g+1$, due to the mutations and crossovers, that may not be the case anymore, so you don't know how to shift $h(i), \forall i$, so that they are all positive and you can compute the probability of being selected, assuming you are using the fitness proportionate selection.
If my intepretation is correct, then, at every generation $g$, you just need to find $w = \operatorname{min}_i h(i)$, then shift all $h(i)$ by adding $|w| + \epsilon$ (where $\epsilon$ can be zero or a very small number) to all $h(i)$, so you would compute the fitness as follows $f(i) = h(i) + |w| + \epsilon$, for all $i$. Here are all the steps to compute the probability of individual $i$ being selected $p(i)$

$w = \operatorname{min}_i h(i)$
$f(i) \leftarrow h(i) + |w| + \epsilon, \forall i$
$p(i) \leftarrow \frac{f(i)}{\sum_{j=1}^{N} f(j)}, \forall i$

This technique is known as windowing, which I have used in the past to solve the exact same problem that you seem to be trying to solve (check it here).


Are there known error bounds for TD(0) with a constant learning rate? Is there any known error bounds for the TD(0) algorithm for the value function after a finite number of iterations?

$$ \Delta_t=\max_{s \in \mathcal{S}}|v_t(s)-v_\pi(s)|$$
$$v_{t+1}(s_t)=v_t(s_t)+\alpha(r+v_t(s_{t+1})-v_t(s_t))$$

The paper "Bias-Variance" Error Bounds for Temporal Difference Updates (2000) by M. Kearns and S. Singh provides error bounds for temporal-difference algorithms, i.e. TD($k$) and TD($\lambda$) (see theorem 1 and theorem 2, respectively). Note that both TD($k$) and TD($\lambda$) include TD($0$) as a special case.


Which algorithms, between ant colony or classical routing algorithms, have a better time complexity for the shortest path problem? Which algorithms, between ant colony or classical routing algorithms, have a better time complexity for the shortest path problem?

In general, can we compare efficiency of these two types of algorithm for the shortest path problem in a graph?

No. In general, you can't find a tight bound for evolutionary algorithms, and it is one of the main difference of these algorithms with the classical algorithms.

You should notice that it does not mean you can't find when the evolutionary algorithms are finished! But, you can't find a tight bound for the algorithms time complexity to reach to the optimal solution or how much that solution is near to the optimal solution (in contrast to the approximation algorithms).


What is the fundamental difference between CNN and RNN? What is the fundamental difference between convolutional neural networks and recurrent neural networks? Where are they applied?

Basically, a CNN saves a set of weights and applies them spatially. For example, in a layer, I could have 32 sets of weights (also called feature maps). Each set of weights is a 3x3 block, meaning I have 3x3x32=288 weights for that layer. If you gave me an input image, for each 3x3 map, I slide it across all the pixels in the image, multiplying the regions together. I repeat this for all 32 feature maps, and pass the outputs on. So, I am learning a few weights that I can apply at a lot of locations.

For an RNN, it is a set of weights applied temporally (through time). An input comes in, and is multiplied by the weight. The networks saves an internal state and puts out some sort of output. Then, the next piece of data comes in, and is multiplied by the weight. However, the internal state that was created from the last piece of data also comes in and is multiplied by a different weight. Those are added and the output comes from an activation applied to the sum, times another weight. The internal state is updated, and the process repeats.

CNN's work really well for computer vision. At the low levels, you often want to find things like vertical and horizontal lines. Those kinds of things are going to be all over the images, so it makes sense to have weights that you can apply anywhere in the images.

RNN's are really good for natural language processing. You can imagine that the next word in a sentence will be highly influenced by the ones that came before it, so it makes sense to carry that internal state forward and have a small set of weights that can apply to any input.

However, there are many more applications. In addition, CNN's have performed well on NLP tasks. There are also more advanced versions of RNN's called LSTM's that you could check out.

For an explanation of CNN's, go to the Stanford CS231n course. Especially check out lecture 5. There are full class videos on YouTube.

For an explanation of RNN's, go here.


What is the difference between imitation learning and classification done by experts? In short, imitation learning means learning from the experts. Suppose I have a dataset with labels based on the actions of experts. I use a simple binary classifier algorithm to assess whether it is good expert action or bad expert action.
How is this binary classification different from imitation learning?
Imitation learning is associated with reinforcement learning, but, in this case, it looks more like a basic classification problem to me.
What is the difference between imitation learning and classification done by experts?
I am getting confused because imitation learning relates to reinforcement learning while classification relates to supervised learning.

Imitation learning is supervised learning applied to the RL setting.

In any general RL algorithm (such as Q-learning), the learning is done on the basis of the reward function. However, consider a scenario where you have available the optimal policy in the form of a table, mapping each state to each action. In this scenario you will not care about the rewards received - rather, you'd simply do a table lookup to decide the optimal action.

This scenario is impractical in most settings because the table for the optimal policy will be too big. However, if you have enough entries from the table, you can use a general function approximator such as a neural network to find the optimal action. Again, you do not need to look at the rewards, but only at the state $\rightarrow$ action mappings. I do not know imitation learning in detail beyond this, but I suspect in the case of discrete actions (such as in Chess, Go), it would be trained with a cross-entropy objective as is typical of classification tasks.

I suspect that the reason this has a different name in the RL setting is because this is different than how the conventional RL algorithms work. Also, much of RL thinking is inspired from everyday interaction / biology. Think of how we learn how to drive, or play sports such as soccer. Typically there is a coach who tells you what actions to take under different conditions, and you imitate those actions.


What is the reasoning behind the number of filters in the convolution layer? Let's assume an extreme case in which the kernel of the convolution layer takes only values 0 or 1. To capture all possible patterns in input of $C$ number of channels, we need $2^{C*K_H*K_W}$ filters, where $(K_H, K_W)$ is the shape of a kernel. So to process a standard RGB image with 3 input channels with 3x3 kernel, we need our layer to output $2^{27}$ channels. Do I correctly conclude that according to this, the standard layers of 64 to 1024 filters are only able to catch a small part of (perhaps) useful patterns?

From mathematical point of view you are correct as are your calculations. To catch all the patterns you need that many filters, but this is where a whole idea of a training comes in. Main objective of the training in the CNNs is to find just a few good patterns from billions possible ones.

So the direct answer to your question is: The standard layers of 64 to 1024 filters are only able to catch a small part of (perhaps) useful patterns, yes but this is assuming no training taking place. If you conducted training on given data with given model, then 64 to 1024 filters could already extract a lot of useful patterns, perhaps more than needed.


Can PHP be considered as a serious programming language for AI? I read some information1 about attempts to build neural networks in the PHP programming language. Personally I think PHP is not the right language to do so at all probably because it's a high-level language, I assume low level language are way more suitable for AI in terms of performance and scalability.

Is there a good/logical reason why you should or shouldn't use PHP as a language to write AI in?

1 http://www.developer.com/lang/php/creating-neural-networks-in-php.html and https://stackoverflow.com/questions/2303357/are-there-any-artificial-intelligence-projects-in-php-out-there

Question on-topicness questionable, but...

The most logical reason why PHP is unsuited for neural networks is that PHP is, well, intended to be used for server side webpages. It can connect to various external resources, such as databases, via native language features. It is very much a glue language, and not a processing language. PHP is also mostly stateless, only allowing you to store state in either clients, file storage or databases.

As such, it's not suitable for this sort of thing - not because PHP is a high level language, but rather because it's so request based and focused towards creating pages to serve to clients.

That won't stop people from trying, though - there are various esoteric programming languages out there in which regular programming would be an insane task or not possible at all - but from a ease of development perspective, making a neural network in PHP makes no sense.


Given an axis-angle rotation vector, how can I find the unit rotation axis and angle? I have a robotics assignment, which I am unable to solve. Given the axis-angle rotation vector $\Theta = (2, 2, 0)$, how can I calculate the unit vector of the rotation axis $k$ and the angle $\theta$?

Given the axis-angle rotation vector $\Theta = (2, 2, 0)$, you can find the unit vector in the same direction by diving by the norm (or length) of $\Theta$, denoted by $\|\Theta\| = \sqrt{2^2 + 2^2 + 0^2} = \sqrt{8} = 2\sqrt{2}$. Therefore, the unit vector in the direction of $\Theta$ is $k= \Theta/\|\Theta\| = (1/\sqrt{2}, 1/\sqrt{2}, 0)$, which should be the axis of rotation that you're looking for. The angle should just be the norm of $\Theta$, that is, $\theta = \|\Theta\| = 2\sqrt{2}$. Note that $k \theta$ gives you your original vector $\Theta$.


Make 9 AIs to replace Supreme Court justices Since the supreme court is always political, why not program 9 AI robots that use different methods to determine whether a law is constitutional and the outcome of cases. How would engineers go about building this? Would it work?

Politics are driven by a combination of things such as culture, ideology, religion, ethics, morality, desire for money, fame, and power, etc.  Although God has one truth, most people don't know this truth, a few do, and the rest either disagree about it or just don't believe in Him. The Constitution was written by man.  It does not tell us what to do in every situation. It is open to interpretation.  A person's interpretation of the Constitution is driven by his morality, culture, and all the things I mentioned above. AI systems are designed by people so they are driven by their biases.


What is the search depth of AlphaGo and AlphaGo Zero? I cannot find reliable sources but someone says it is 40 moves and someone else says it is 50+ moves. I read their papers and they use value function (NN) and policy function to trim the tree, so more layers can be searched while spending less time searching less different positions.
My question is, is the search depth a fixed preset parameter? If so, approximately how much is it back to 2016 (AlphaGo) and 2018 (AlphaGo Zero)?

For easier visualization, I recommend this video:
https://twitter.com/i/status/1257053365424578565
The more detailed article about GO algorithms:
https://deepmind.com/blog/article/alphago-zero-starting-scratch.
With its breadth of $250$ possible moves each turn (go is played on a $19$ by $19$ board, compared to the much smaller $8$ by $8$ chess field) and a typical game depth of $150$ moves, there are about $250^{150}$, or
$10^{360}$.
After $2$ moves in go, it's $130000$ possible combinations.
Decision tree pruning


Is there a proof to explain why XOR cannot be linearly separable? Can someone explain to me with a proof or example why you can't linearly separate XOR (and therefore need a neural network, the context I'm looking at it in)?
I understand why it's not linearly separable if you draw it graphically (e.g. here), but I can't seem to find a formal proof somewhere, I wanted to try and understand it with either an equation or example written down. I'm wondering if one exists (I guess it has to do with contradictions?), but I can't seem to find it? I have seen this, but it's more a reason than a proof.

Before proving that XOR cannot be linearly separable, we first need to prove a  lemma:
Lemma 1
Lemma: If 3 points are collinear and the middle point has a different label than the other two, then these 3 points cannot be linearly separable.

Proof: Let us label the points as point $A$, $B$, and $C$. $A$ and $C$ have the same label, and $B$ has a different label. They are all collinear with line $\mathcal{L}$.
Assume the contradiction, so a line can linearly separate $A$, $B$, and $C$. This means a line must cross between segment $AB$, and segment $BC$ to linearly separate these three points (by definition of linear separability). Let us label the point where the line crosses segment $AB$ and point $Y$, and the point where the line corsses segment $BC$ and point $Z$.
However, since segments $AB$ and $BC$ are collinear to line $\mathcal{L}$, points $Y$ and $Z$ also falls on line $\mathcal{L}$. Since only one unique line can cross 2 points, it must be that the only line that passes segments $AB$ and $BC$ and (therefore separates points $A$, $B$, and $C$) is line $\mathcal{L}$.
However, line $\mathcal{L}$ cannot linearly separate $A$, $B$, and $C$, since line $\mathcal{L}$ also crosses them. Therefore, no line exists can separate $A$, $B$, and $C$.
Main proof

Consider these 4 points that represent a XOR table. Let us label them clock-wise, so the top-left point as $A$, top-right point as $B$, bottom-right point as $C$, and bottom-left point as $D$. So $A$ and $C$ have the same label, and $B$ and $D$ have the same label. We want to show that points $A, B, C$ and $D$ cannot be linearly separable.
Assume the contradiction, and that there is a line that can separate these 4 points.
Imagine a fifth point that lies in the center, and let us label this as point $E$.
Since $E$ lies in the center, the three points $A, E$ and $C$ are collinear. Similarly, since $E$ lies in the center, the three points $B, E$ and $D$ are collinear.
Because we assume a line can linearly separate $A, B, C$ and $D$, then this line must label point $E$ as some label. If $E$ shares the same label as $A$ and $C$, then the points $B, E$ and $D$ will become "collinear points where the middle point has a different label", which by Lemma 1 cannot be linearly separable. Likewise, if $E$ shares the same label as $B$ and $D$, then the points $A, E$ and $C$ will become "collinear points where the middle point has a different label", which by Lemma 1 cannot be linearly separable.
Therefore it is impossible to give a label to $E$ while satisfying linear separability. As a result, our assumption must be false, and the four points $A, B, C$ and $D$ cannot be linearly separable.


Is there a way of path reconstruction using only the history of belief state? Given a history of belief states, is there a common method that backtracks the most likely path of ending up in the current belief state?
I have a Markov model which calculates belief states after every step. The belief state is a representation of the most likely states one could be in. A belief state may look like this:
$$b=[1,0,0,0,0],$$ where I am in the state $s_0$ with 100% certainty.
I can store the belief state history like $b_0, b_1, b_2,...b_n$.
Is there a common way to represent and estimate the most likely sates one has been in?
A naive approach could be to just look for the state with the highest value per belief state and take that as the node along the reverse-path. But I am not confident enough, if that is a common and a good practice as it is not considering the fuzziness, which comes with a belief state. But then again, if I would take all states that are bigger than 0, I might not know which state leads to which state and if that transition is even possible.

The belief state in a POMDP is a distribution over the hidden state given all past actions and observations, i.e., at time $k$, the belief state is $b_k(s_k) \triangleq P(s_k \mid a_{0:k-1}, z_{1:k})$, where $a$ and $z$ are the actions and observations, respectively.
What you are asking about and calling "backtracking" boils down to the question: "given that at time $k$, I know the history of actions $a_{0:k-1}$ and observations $z_{1:k}$, what is the distribution over states at some past time step $t<k$, that is, $P(s_t\mid a_{0:k-1},z_{1:k})$"?
This is commonly known as Bayesian smoothing. You might be familiar with Bayesian filtering, which aims to recover the belief state as defined in the first paragraph - an estimate of the current state given actions and observations up to the current time. Smoothing also uses information gained after time $t$ to estimate the state at time $t$, and hence it is only possible after those actions and observations become known. There is also prediction, where you estimate the distribution over the state in the future.
Smoothing is too broad of a topic to go into detail here, but below is a picture from the book "Bayesian Filtering and Smoothing" by Simo Särkkä that illustrates the concepts. You could start there for further information on the topic.


What are the strategies for computationally heavy environments or long-time waiting environments? I have an environment that is computationally heavy (takes several seconds to get a reward and next state). This limits reinforcement capability, due to poor sampling of the problem. There is any strategy that could be used to address the problem (e.g. If I can use the environment in parallel, then I could use a multi-agent approach)

You could use Ray RLlib. It has support for parallel environments, even over multiple GPUs and compute nodes.


Is it possible to apply the associative property of the convolution operation when it is followed by a non-linearity? The associative property of multidimensional discrete convolution says that:
$$Y=(x \circledast h_1) \circledast h_2=x\circledast(h_1\circledast h_2)$$
where $h_1$ and $h_2$ are the filters and $x$ is the input.
I was able to do exploit this property in Keras with Conv2D: first, I convolve $h_1$ and $h_2$, then I convolve the result with $x$ (i.e. the rightmost part of the equation above).
Up to this point, I don't have any problem, and I also understand that convolution is linear.
The problem is when two Conv2D layers have a non-linear activation function after the convolution. For example, consider the following two operations
$$Y_1=\text{ReLU}(x \circledast h_1)$$
$$Y_2=\text{ReLU}(Y_1\circledast h_2)$$
It is possible to apply the associative property if the first or both layers have a non-linear activation function (in the case above ReLU, but it could be any activation function)? I don't think so. Any idea or related paper or some kind of approach?

Yes, when you have non-linearity it is not possible to combine your convolution steps.
However, you can approximate the two layers network with one layer net according to the Universal Approximation Theorem. You will probably need to use something like a knowledge distillation technique to do it. Note that, the theorem doesn't say anything about the number of neurons required or if the learning techniques that we usually use will work well.
Also, $ReLU(x)$ is a linear mapping when $x \geq 0$, so if your input, weight, and biases are $\geq 0$, the net can be exactly modeled with a single layer network.


How to determine the size of biases? I'm new to the world of machine learning. My question is how can I determine the size of the biases in a neural network (with backpropagation algorithm)? Currently, I have a 2-layer neural network (1 hidden and 1 output layer). Here's the code:

import numpy as np
from matplotlib import pyplot as plt

sigmoid = lambda x : 1 / (1 + np.exp(-x))
dsigmoid = lambda y: y * (1 - sigmoid(y))

# This function performs the given function (func) to the whole numpy array
def mapFunc(array, func) :
    newArray = array.copy()
    for element in np.nditer(newArray, op_flags=['readwrite']) :
        element[...] = func(element)
    return newArray

class NeuralNetwork :

def __init__(self, input_nodes, hidden_nodes, output_nodes) :
    self.input_nodes = input_nodes
    self.hidden_nodes = hidden_nodes
    self.output_nodes = output_nodes

    self.W_ih = np.random.rand(hidden_nodes, input_nodes)
    self.W_ho = np.random.rand(output_nodes, hidden_nodes)

    self.B_ih = np.random.rand(hidden_nodes, 1)
    self.B_ho = np.random.rand(output_nodes, 1)

    self.learningRate = 0.1

def predict(self, inputs) :
    # Calculate hidden's output
    H_output = np.dot(self.W_ih, inputs)
    H_output += self.B_ih
    H_output = mapFunc(H_output, sigmoid) # Activation

    # Calculate output's output
    O_output = np.dot(self.W_ho, H_output)
    O_output += self.B_ho
    O_output = mapFunc(O_output, sigmoid) # Activation

    return O_output

def train(self, inputs, target) :
    # Calculate hidden's output
    H_output = np.dot(self.W_ih, inputs)
    H_output += self.B_ih
    H_output = mapFunc(H_output, sigmoid) # Activation

    # Calculate output's output
    O_output = np.dot(self.W_ho, H_output)
    O_output += self.B_ho
    O_output = mapFunc(O_output, sigmoid) # Activation

    # Calculate output error :
    O_error = O_output - target

    # Calculate output delta
    O_gradient = mapFunc(O_output, dsigmoid)
    O_gradient = np.dot(O_gradient, np.transpose(O_error)) * self.learningRate

    W_ho_delta = np.dot(O_gradient, np.transpose(H_output))

    self.W_ho -= W_ho_delta
    self.B_ho -= O_gradient

    # Calculate hidden error :
    W_ho_t = np.transpose(self.W_ho)
    H_error = np.dot(W_ho_t, O_error)

    # Calculate hidden delta :
    H_gradient = mapFunc(H_output, dsigmoid)
    H_gradient = np.dot(H_gradient, np.transpose(H_error)) * self.learningRate

    W_ih_delta = np.dot(H_gradient, inputs)

    self.W_ih -= W_ih_delta
    self.B_ih += H_gradient

    return O_output

n = NeuralNetwork(2, 2, 1)

inputs = np.matrix([[1], [0], [1], [1], [0], [1], [0], [0]])

input_list = []
input_list.append([[1], [0]])
input_list.append([[0], [1]])
input_list.append([[1], [1]])
input_list.append([[0], [0]])

target = np.matrix([[0], [0], [1], [1]])

outputs = []
for i in range(50000) :
    ind = np.random.randint(len(input_list))
    inp = input_list[ind]
    out = n.train(inp, target[ind]).tolist()
    outputs.append(out[0][0])

print outputs
plt.plot(outputs)

plt.show()

newInput = [[1], [1]]
print (n.predict(newInput))

In the train function, the line self.B_ih += H_gradient throws me an error about their sizes not being equal. I even tried to make the biases only a single number but that didn't help as it gets changed by H_gradient to a matrix. So, is there something wrong in the bias itself or I did some other step(s) wrong?

The biases appear in the activation function and can be seen as node feature. Suppose you have act = sigmoid, X your input layer and W your first hidden layer weight matrix.
Then the output matrix of this hidden layer will be act(X.dot(W)+b).

So your bias should be a vector containing a random value for each node of the current hidden layer.

It will shift randomly the weight and so the activation function. See here (usually it prevent from overfitting).

I didn't read the whole code so I don't know if the problem comes from here, but why don't you learn how to use tensorflow instead of re-write everything ? There are a lot of tutorials in the net, like MNIST.


At what rate could AI theoretically self-improve? Due to recursive self-improvement, AI could lead to an intelligence explosion improving on itself year over year exponentially.

Assuming the proper environment was created to allow an AI to self-improve, how fast would this occur?

With human intervention we might say AI could improve similar to GDP or science growth say 3% per year. But this intervention would be additive to the self-recurring improvements made by the AI.

What is a reasonable annual rate of self-growth and what would the processing power of the AI be after 10 years given an initial value of say 1PFLOP?

Example:

human rate = 3% annually

AI rate = 1% of current processing power annually

Year 1. 1 PFLOP

Year 2. 1 PFLOP + (1 PFLOP * 3%) + (1 PFLOP * 1%) = 1.04 PFLOP

Year 3. 1.04 PFLOP + (1.04 PFLOP * 3%) + (1.04 PFLOP * 1.04%) = 1.082 PFLOP

Year 4. 1.082 PFLOP + (1.082 PFLOP * 3%) + (1.082 PFLOP * 1.082%) = 1.126 PFLOP

Year 5. 1.126 PFLOP + (1.126 PFLOP * 3%) + (1.126 PFLOP * 1.126%) = 1.172 PFLOP

The processing power might still follow the Moore's law, but it is basically an economical law, although it has worked surprisingly well considering the physical limitations of technology. An Artificially Intelligent agent could focus on manipulating the market, research possible hardware improvements, allocate human resources more effectively to the problem, etc.

I would expect some improvement in the AI algorithm design itself, which will not influence the number of operations available, but the number of operations required to complete a task. By using a better heuristics, the AI algorithm will consume less brute force to complete the task, thus being able to complete more tasks given the same number of operations. Sorry if I misunderstood the question. Thanks for the improvement.


Tabular Datasets where Deep neural networks outperforms XGBoost Are there (complex) tabular datasets where deep neural networks (e.g. more than 3 layers) outperform traditional methods such as XGBoost by a large margin?

I'd prefer tabular datasets rather than image datasets, since most image dataset are either too simple that even XGBoost can perform well (e.g. MNIST), or too difficult for XGBoost that its performance is too low (e.g. almost any dataset that is more complex than CIFAR10; please correct me if I'm wrong).

Thank you!

In my opinion, no.
Also images could be interpreted as tabular dataset as well, where certain columns represent different rgb codes of pixels. If you seek to use neural nets opt for image datasets, with large sample size.
Neural networks generally require large sample sizes to perform, and huge dimension inputs to not be outperformed by boosting.


Choosing an AI method to recreate a given binary 2D image If the title wan not very clear, I want a method to take an input image like this,
[[0, 0, 0, 0],
 [1, 1, 1, 0],
 [1, 1, 1, 0],
 [0, 1, 1, 0]]

and output the 2D coordinates of the 1s of the image (So that I can recreate the image)
The application is a robot creating the image using some kind of building blocks, placing one block after the other
I want the output to be sequential because I need to reconstruct the input image pixel by pixel and there are some conditions on the image construction order (e.g. You cannot place a 1 somewhere when it is surrounded by 1s)
The image can change and the number of 1s in the image too.

What is an appropriate AI method to apply in this case?
How should I feed the image to the network? (Will flattening it to 2D affect my need of an output order?)
Should I get the output coordinates one by one or as an ordered 2xN matrix?
If one by one, should I feed the same image for each output or the image without the 1s already filled?

I have tried to apply "NeuroEvolution of Augmenting Topologies" for this using neat-python but was unsuccessful. I am currently looking at RNNs but I am not sure if it is the best choice either.

This sounds like a problem that's best solved with a simple non-AI algorithm. If you just enumerate the coordinates in a regular order (rows, colums, zig-zag, hilbert curve) and emit the coordinates where the image has a '1' you're meeting your requirements. Is there any specific reason you want to use an AI algorithm which is most likely worse than this?


Can two admissable heuristics not dominate each other? I am working on a project for my artificial intelligence class. I was wondering if I have 2 admissible heuristics, A and B, is it possible that A does not dominate B and B does not dominate A?  I am wondering this because I had to prove if each heuristic is admissible and I did that, and then for each admissible heuristic, we have to prove if each one dominates the other or not. I think I have a case that neither dominates the other and I was wondering if maybe I got the admissibility wrong because of that.

This is possible. Admissibility only asserts that the heuristic will never overestimate the true cost. With that being said, it is possible for one heuristic in some cases to do better than another and vice-versa. Think of it as a game of rock paper scissors.

Specifically, you may find that sometimes $h_1 < h_2$ and in other times $h_2 < h_1$, where $h_1$ and $h_2$ are admissible heuristics. Thus, by definition, neither strictly dominates the other.

In fact, there is a way to "combine" the two admissible heuristics to get the best of both using:

$$h_3 = \max(h_1, h_2)$$


Is pre-processing used in deep learning? I'm new to deep learning. I wanted to know: do we use pre-processing in deep learning? Or it is only used in machine learning. I searched for it and its methods on the internet, but I didn't find a suitable answer.

Yes, sure, data pre-processing is also done in deep learning. For example, we often normalize (or scale) the inputs to neural networks. If the inputs are images, we often resize them so that they all have the same dimensions. Of course, the pre-processing step that you apply depends on your data, neural network, and task.
Here or here are two examples of implementations that perform a pre-processing step (normalization in the second case). You can find more explanations and examples here and probably here too.


How to create a good fitness function? In genetic algorithms, a function called "fitness" (or "evaluation") function is used to determine the "fitness" of the chromosomes. Creating a good fitness function is one of the challenging tasks in genetic algorithms. How would you create a good fitness function?

In my experience, the fitness function is a way to define the goal of a genetic algorithm. It provides a way to compare how "good" two solutions are, for example, for mate selection and for deleting "bad" solutions from the population.

The fitness function can also be a way to incorporate constraints, prior knowledge you may have about the shape of the fitness landscape, or the way your crossover/recombination operators will work in that fitness landscape.

For example, the fitness function can include hard constraints like "Genes x,y, and z must all stay on one side of the surface $Ax +By +Cz = k$" by assigning the fitness value at zero if the gene values are on the wrong side of the surface.  However, it's often better in a case like that soften the boundary by assigning a fitness penalty that is zero at the surface and grows larger as the gene values move farther from the surface on the wrong side of the surface.

Different fitness functions can be used for mate selection vs deleting "bad" trial solutions.  For example, "mating fitness" between two potential parents A and B can be a function of how different the two parents are.  By providing a mating advantage to pairs that are significantly different, the population can be forced to remain fairly diverse and thus explore a larger region of solution space, or to avoid converging to local (sub-optimum) fitness maxima.  Meanwhile, the usual kind of fitness will cull the low-fitness individuals from the population and drive evolution toward high fitness.

What is often much more important is the set of variables ("genes") used to represent a trial solution, how the genes are arranged in the "chromosome", and the ways genes from two parents can be combined to form a new trial solution. Since you didn't ask about those things I won't go into detail in this answer, but if you ask in a separate question I will provide a detailed answer.


How would a quantum computer potentially facilitate artificial consciousness, assuming it is possible? How would a quantum computer potentially facilitate artificial consciousness, assuming it is possible?

Well, this is quite deep question. I would recommend you to have a look on the UK physicist Roger Penrose: https://en.wikipedia.org/wiki/Roger_Penrose
He has some books on the theme of consciousness and quantum mechanics. There are however critics of his ideas. One of them is that the quantum effects are too small to direct any considerable influence on the brain.


What techniques would allow a model to be trained to produce subjectively pleasant 'artistic' output? I've only just started looking into Machine Learning, and so far most of the examples I've seen involve starting with a training set of observations, each representing a value for a number of 'features', and using that to train a model - then evaluating the trained model against a second test set of data to allow refinements to the model. The basic operation the model is asked to do is to predict the value of one feature given others.

When talking about art (by which I would include music, poetry, etc), we can no longer assert that things are correct - but perhaps we could isolate a number of subjectively-judged features, such as whether a work is 'mysterious' or 'joyful' to a particular human judge, and then train the model to our judge's taste. However, The problem here would seem to be that having a human being add judgement features to each of our data points sounds like a very slow task, and it might often be impractical to get to the number of 'assessed works' that would allow the model to be trained.

How could the process of training a model to produce art of a certain 'taste' be done in a practicably short space of time?

You will need a large training set to be able to train your machine to produce subjectively pleasant 'artistic' output.

If you can't produce your own dataset with real people, you can try the following:

search for existing datasets on the internet
produce a new dataset using a spider bot that perform mining of artistic output on websites that have voting features
use crowdsourcing


What kind of output should be used for predicting angles in DNNs? I am building a model which predicts angles as output. What are the different kinds of outputs that can be used to predict angles?

For example,

output the angle in radians

cyclic nature of the angles is not captured
output might be outside $\left[-\pi, \pi \right)$

output the sine and the cosine of the angle

outputs might not satisfy $\sin^2 \theta + \cos^2 \theta = 1$

What are the pros and cons of different methods?

As you said, the first option is not very suitable due to the cyclic nature of the angles. However, if you don't mind discretizing the values, you could represent the output as a binary vector.

A variant of the second option seems perfect to me. You may output a 2D vector and use that vector's angle as output. You'll probably need a regularizer for the vector's norm, but that's it.


Is there a training data capacity limit for AlphaZero (Chess)? In AlphaZero, we collect ($s_t, \pi_t, z_t$) tuples from self-play, where $s_t$ is the board state, $\pi_t$ is the policy, and $z_t$ is the reward from winning/losing the game. In other DeepRL off-policy algorithms (I'm assuming here that AlphaZero is off-policy (?)) like DQN, we maintain a memory buffer (say, 1 million samples) and overwrite the buffer with newer samples if it's at capacity. Do we do the same for AlphaZero? Or do we continually add new samples without overwriting older ones? The latter option sounds very memory heavy, but I haven't read anywhere that older samples are overwritten.

AlphaZero is on-policy*, which partially answers your question.
An on-policy algorithm is not the same as an online policy though, it is not required that updates are made on every step. It is simply required that all data used in the update is taken from the same "current" policy.
In practice, AlphaZero buffers results from games played with the current policy to create a dataset used to update its neural networks. That buffer is then emptied after the data has been used.
From the AlphaZero paper:

At the end of the game, the terminal position $s_T$ is
scored according to the rules of the game to compute the game outcome $z: −1$ for a loss, $0$ for a draw, and $+1$ for a win. The neural network parameters $\theta$ are updated so as to minimise the error between the predicted outcome $v_t$ and the game outcome $z$, and to maximise the similarity of the policy vector $p_t$
to the search probabilities $\pi_t$.

This implies only a single game is buffered in this way before running each update and then discarding the dataset generated in that game. Theoretically the same approach could be used with any number of games for each update step (provided the training system has capacity to store more moves).

* AlphaZero is on-policy because the core algorithm requires using a specific policy and then updating it to match an improved version of the same policy discovered using MCTS for planning during play.
It could be possible to construct an off-policy update mechanism using similar MCTS routine. I am not sure why this is not considered, but suspect it would be due to complexity/efficiency of the algorithm compared to the ease of generating new game data.


In the policy gradient equation, is $\pi(a_{t} | s_{t}, \theta)$ a distribution or a function? I am learning about policy gradient methods from the Deep RL Bootcamp by Peter Abbeel and I am a bit stumbled by the math presented. In the lecture, he derives the gradient logarithm likelihood of a trajectory to be

$$\nabla log P(\tau^{i};\theta) = \Sigma_{t=0}\nabla_{\theta}log\pi(a_{t}|s_t, \theta).$$

Is $\pi(a_{t} | s_{t}, \theta)$ a distribution or a function? Because a derivative can only be taken wrt a function. My understanding is that $\pi(a_{t},s_{t}, \theta)$ is usually represented as a distribution of actions over states, since input of a neural network for policy gradient would be the $s_t$ and output would be $\pi(a_t, s_t)$, using model weights $\theta$.

First, the derivative is usually taken with respect to a variable (input) of the function. Hence the notation $\frac{df}{dx}$ for some function $f(x)$.

If you look at your equation more carefully

$$\nabla log P(\tau^{i};\theta) = \Sigma_{t=0}\nabla_{\theta}log\pi(a_{t}|s_t, \theta).$$

You will see that the gradient is taken with respect to $\theta$, which are the parameters (i.e. a vector) e.g. of your neural network, that is, $\nabla_{\theta}$.

In this case, it doesn't really matter whether $\pi$ represents a distribution or not (for some specific value of $\theta$), but you're right that $\pi$ often represents a probability distribution over the possible actions (given a specific state). In any case, $\pi$ is a function of the parameters $\theta$ (i.e. in the case of a distribution, $\pi_{\theta}$ is a family of distributions for all possible values of $\theta$), i.e. if you change $\theta$ the outputs of $\pi$ also change, so you can take the derivative of it with respect to $\theta$.


Why are decision trees and random forests scale invariant? Feature scaling, in general, is an important stage in the data preprocessing pipeline.
Decision Tree and Random Forest algorithms, though, are scale-invariant - i.e. they work fine without feature scaling. Why is that?

Scaling only makes sense when there is something that reacts to that scale. Decision Trees though, just make a cut at a certain number.
Imagine: For a feature that goes from 0 to 100 a cut at 50 may be improving performance. Scaling this down to 0 to 1 making the cut a 0.5 doesn't change a thing.
Now on the other hand NN have some kind of activation function (leaving RELu aside) that react differently to input that is above 1. Here Normalization, putting every feature between 0 and 1 makes sense.


How is local minima possible in gradient descent? Gradient descent works on the equation of mean squared error, which is an equation of a parabola $y=x^2$.

We often say that weight adjustment in a neural network by gradient descent algorithm can hit a local minima and get stuck in there.

How is local minima on the equation of a parabola possible, where the slope is always parabolic?

$g(x) = x^2$ is indeed a parabola and thus has just one optimum.

However, the $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - f(x_i))^2$, where $\boldsymbol{x}$ are the inputs, $\boldsymbol{y}$ the corresponding labels and the function $f$ is the model (e.g. a neural network), is not necessarily a parabola. In general, it is only a parabola if $f$ is a constant function and the sum is over one element.

For example, suppose that $f(x_i) = c, \forall i$, where $c \in \mathbb{R}$. Then $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - c)^2$ will only change as a function of one variable, $\boldsymbol{y}$, as in the case of $g(x) = x^2$, where $g$ is a function of one variable, $x$. In that case, $(y_i - c)^2$ will just be a shifted version (either to the right or left depending on the sign of $c$) of $y_i^2$, so, for simplicity, let's ignore $c$. So, in the case $f$ is a constant function, then $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i y_i^2$, which is a sum of parabolas $y_i^2$, which is called a paraboloid. In this case, the paraboloid corresponding to $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i y_i^2$ will only have one optimum, just like a parabola. Furthermore, if the sum is just over one $y_i$, that is, $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i y_i^2 = y^2$ (where $\boldsymbol{y} = y$), then the MSE becomes a parabola.

In other cases, the MSE might not be a parabola or have just one optimum. For example, suppose that $f(x) = x^2$, $y_i = 1$ ($\forall i$), then $h(x) = (1 - x^2)^2$  looks as follows

which has two minima (and one maximum): at $x=0$ and $x=1$. We can find the two minima of this function $h$ using calculus: $h'(x) = -4x(1 - x^2)$, which becomes zero when $x=0$ and $x=1$.

In this case, we only considered one term of the sum. If we considered the sum of terms of the form of $h$, then we could even have more "complicated" functions.

To conclude, given that $f$ can be arbitrarily complex, then also $\text{MSE}(\boldsymbol{x}, \boldsymbol{y})$, which is a function of $f$, can also become arbitrarily complex and have multiple minima. Given that neural networks can implement arbitrarily complex functions, then $\text{MSE}(\boldsymbol{x}, \boldsymbol{y})$ can easily have multiple minima. Moreover, the function $f$ (e.g. the neural network) changes during the training phase, which might introduce more complexity, in terms of which functions the MSE can be and thus which (and how many) optima it can have.


Can AI come up with scientific theories of past when provided with sufficient data available at that time? I would love to know if an AI model could come up with certain theories of the old like Pythagoras' theorem, Euclid's formulations,
Newton's gravity, Einstein's theories if provided and trained with sufficient amount of observable data available at those period of time. If this is possible can unsolved conjectures be proved by AI? Or even better can AI develop new theories or will it  fail to come up with even basic mathematical operations by itself?

Yes.

Some good examples of this are Lipson's work using evolutionary models, and Wu & Tegmark's work on a theory-based life-long learner, and Iten et al.'s work with deep neural networks. There are many, many, other research papers in this area, and there is a lot more work that is ongoing. The endgame for a lot of this work is the hope that we can synthesize new theories automatically.


When should I use a linear activation instead of ReLU? I have read this post:
How to choose an activation function?.

There is enough literature about activation functions, but when should I use a linear activation instead of ReLU?

What does the author mean with ReLU when I'm dealing with positive values, and a linear function when I'm dealing with general values.?

Is there a more detail answer to this?

The activation function you choose depends on the application you are building/data that you have got to work with. It is hard to recommend one over the other, without taking this into account.

Here is a short-summary of the advantages and disadvantages of some common activation functions:
https://missinglink.ai/guides/neural-network-concepts/7-types-neural-network-activation-functions-right/

  What does the author mean with ReLU when I'm dealing with positive values, and a linear function when I'm dealing with general values.

ReLU is good for inputs > 0, since ReLU = 0 if input < 0(which would kill the neuron, if the gradient is = 0)

To remedy this, you could look into using a Leaky-ReLU instead.
(Which avoids killing the neuron by returning a non-zero value in the cases of input <= 0)


Can someone direct me to a sites and/or videos that can bring an absolute beginner up to speed with AI? To start, I'm not a programmer/computer scientist/et al... - I work in Finance and have, through my job, self-thought myself VBA for excel and outlook and would consider myself as being in the upper half of the population of people who write VBA creating browser apps for scraping (and interpreting site's DOMs in doing so), creating Arrays of multi-dimensional arrays (arrays of 3D arrays - haven't gone for a 4D array as I haven't needed to) and generally focused on making as efficient macros. I've (very) basic understanding of JAVA also (I even created an basic android app!).

By pure accident, learning the above has given me a massive interest in all thing related to IT developments and the massive development which I truly believe will be the development in Human History (for better or worse has yet to be determined!) is AI.

I want to know all that I can about AI so was wondering if you all could assist an absolute noob on the subject. I know nothing of Python (which seems the language of choice for AI) but anything that would help would be great.

I'm not completely naive to the fact the brightest minds in the IT world are are working on this and concepts are going to be difficult for someone like me but I'm willing to try!

Any and all help/suggestions are welcome!

Thanks in advance.

I think the best approach would be to pick up a good book that lays out the fundamentals very well.

Artificial Intelligence: A Modern Approach (AIMA) is a great place to begin with! It is used as a reference textbook for many university level AI courses. It has helped me quite a lot as a student.


How can I train a neural network for image classification when the dataset is small? I need to train a convolutional neural network to classify snake images. The problem is that I have only a small number of images available for some snake types.
So, what is the best approach to train a neural network for image classification using a small data set?

Use Fine Tuning

You can simply use a pre-trained model on ImageNet, as this data set has multiple snakes classes.

Then you can fine tune the model with your own small data set and outputs.
See this for further understanding :
Fine Tuning in Keras

(if you don't use Keras, there are other tutorials on the internet using other Machine Learning framework)

The idea is just removing the last layer (1000 outputs if you use a model pre-trained with ImageNet) and adding a layer of your choice with random weights and a custom number of outputs (number of your classes).

Then your retrain your network, in general we retrained only the last layers (as first layers have more general features).


Does augmenting data changes the distribution of augmented data? When we augment data for training are we also changing the distribution of data and if its a different distribution why do we use it to train a model for original distribution ?

Yes you do change the distribution of your training data if you modify it (for example by augmenting it with rotated versions of images that were in an original training set of images).

This is fine because, typically, our goal of training is not to get a model with high performance on the dataset we happened to collect as training data (e.g. a bunch of natural images). Our goal is to train a model that generalizes well to new data outside of the training data's distribution.

Typically, the training data is only a sample of the distribution we're actually interested in. For example, we'll be interested in making accurate predictions for all natural images in the entire world. That is a distribution that would likely include rotated variants of all images in our training set. So, if we augment our training set by adding such rotated variants, we expect to modify our training data distribution in such a way that it actually gets a little bit closer to the distribution we're interested in (all natural images).


What is a learning agent? What is a learning agent, and how does it work? What are examples of learning agents (e.g., in the field of robotics)?

A learning agent is any entity that, over time, improves its performance (which can be defined in different ways depending on the context) based on the interaction with the environment (or experience).

The human is an example of a learning agent. For example, a human can learn to ride a bicycle, even though, at birth, no human possesses this skill.


Can we use transformers for audio classification tasks? Since transformers are good at processing sequential data, can we also use them for audio classification problems (same as RNNs)?

Yes, Transformers can be used to work with audio data, such as audio processing (audio classification, speaker identification, etc) (Audio ALBERT), speech-to-text (Streaming Automatic Speech Recognition with the Transformer Model), and text-to-speech (Neural Speech Synthesis with Transformer Network).


What is fuzzy logic? I'm new to A.I. and I'd like to know in simple words, what is the fuzzy logic concept? How does it help, and when is it used?

As complexity rises, precise statements lose meaning and meaningful statements lose precision. ( Lofti Zadeh ).

Fuzzy logic deals with reasoning that is approximate rather than fixed and exact. This may make the reasoning more meaningful for a human:

Fuzzy logic is an extension of Boolean logic by Lotfi Zadeh in 1965 based on the
mathematical theory of fuzzy sets, which is a generalization of the classical set theory.
By introducing the notion of degree in the verification of a condition, thus enabling a
condition to be in a state other than true or false, fuzzy logic provides a very valuable
flexibility for reasoning, which makes it possible to take into account inaccuracies and
uncertainties.

One advantage of fuzzy logic in order to formalize human reasoning is that the rules
are set in natural language. For example, here are some rules of conduct that a driver
follows, assuming that he does not want to lose his driver’s licence:

Intuitively, it thus seems that the input variables like in this example are approximately
appreciated by the brain, such as the degree of verification of a condition in fuzzy
logic.

I've written a short introduction to fuzzy logic that goes into a bit more details but should be very accessible.


Which algorithm to use to solve this optimization problem? I have items called 'Resources' from 1 to 7.
I have to use them in different actions identified from 1 to 10.
I can do a maximum of 4 actions each time. This is called 'Operation'.
The use of a resource has a cost of 1 per each 'Operation' even if it is used 4 times.
The following table indicates the resources needed to do the related actions:

|        |            Resources             |
|--------|----------------------------------|
| Action |  1 |  2 |  3 |  4 |  5 |  6 |  7 |
|--------|----------------------------------|
|     1  |  1 |  0 |  1 |  1 |  0 |  0 |  0 |
|     2  |  1 |  1 |  0 |  0 |  1 |  0 |  0 |
|     3  |  1 |  0 |  1 |  0 |  0 |  1 |  0 |
|     4  |  0 |  1 |  0 |  0 |  0 |  0 |  0 |
|     5  |  1 |  0 |  1 |  1 |  0 |  1 |  0 |
|     6  |  1 |  1 |  1 |  0 |  0 |  0 |  0 |
|     7  |  0 |  1 |  0 |  0 |  0 |  0 |  0 |
|     8  |  1 |  0 |  1 |  0 |  1 |  0 |  0 |
|     9  |  0 |  1 |  0 |  1 |  0 |  0 |  0 |
|    10  |  1 |  1 |  1 |  0 |  0 |  0 |  1 |

The objective is to group all the 'Actions' in 'Operations' that minimize the total cost. For example, a group composed by actions {3, 7, 9} needs the resources {1, 2, 3, 4, 6} and therefore has a cost of 5, but a group composed by actions {4, 7, 9} needs the resources {2, 4} and therefore has a cost of 2.

It is needed to get done all the actions the most economically.

Which algorithm can solve this problem?

A possible approach would be starting by grouping the actions into all possible groups.
For 30 actions grouped in a maximum of 6 this would mean:
$$
C^{30}_6 + C^{30}_5 + C^{30}_4 + C^{30}_3 + C^{30}_2 + C^{30}_1 = 768 211
$$
Operations.

Then, I would define the constraints to simplify the problem, as in a Constraint Satisfaction Problem (CSP). The main constraint that you have is that one action can only be represented in one Operation. So, if you choose 1 of the 768ks operations to start with, the second option will be restricted by the condition.

Then I would do some sort of planning and sum the costs until the end doing a depth-first search. Whenever I find a total cost better the previous best, you would update the best value. (Remember that the search ends when between all the operations used, all the actions were explored once.)

While doing the search the costs are added up on the go. You can precompute a vector with the same size as the number of operations where each value corresponds to the total cost of the operation, and then sum it as you are going down the path.

You should also add some heuristic to prevent checking the same operations in different orders. For example doing operating [2,3,4] followed by [7,10], where each number represents an action, is the same as doing: [7,10] followed by [2,3,4].


How to detect a Neural Network will work with the whole dataset? I want to implement a neural network on a big dataset. But training time is long (~1h30 per epoch). I'm still in the development process, so I don't want to wait such long time just to have poor results at the end.

This and this suggest that overfitting the network on a very small dataset (1 ~ 20 samples) and reach a loss near 0 is a good start.

I did it and it works great. However, I am looking for the next step of validating my architecture. I tried to overfit my network over 100 samples, but I can't reach a loss near 0 in reasonable time.

How can I ensure the results given by my NN will be good (or not), without having to train it on the whole dataset ?

You can try to train it on 1% data, then on 2%, 3%, etc... Then plot it and see if increasing data increases the performance and how it is changing. Not sure if that's correct answer, but at least you can iterate this method pretty fast.


Are 1D CNNs really the appropriate model for human activity recognition? This article on human activity recognition states that 1D convolutional neural networks work the best on the classification of human activities using data from the accelerometer. But I think that human activities, like swinging the arm, are sequential actions and they require LSTMs.
So, which one should be more effective for this task: CNNs or LSTMs? In other words, is spatial learning required or sequence learning?

Human activity recognition, for example as done with the Sports-1M Dataset, is a classification task. Given a video, say which activity / action was executed.

So it is a special case about video classification.

  human activities like swinging the arm are sequential actions and they require LSTMs

This is wrong. As a simple example, think of handwriting recognition as done on write-math.com: I receive a sequence of points (x, y) -- the pen-tip --  and classify which symbol is written. So it is a sequence, but still there a plenty of ways to use a simple multi-layer perceptron (MLP) for it. Two big groups of approaches:

Feature-Engineering: Instead of giving an arbitrary number of coordinates, one can normalize it to a fixed amount. This is what is currently done on write-math.com and works pretty well (see my Bachelors thesis)
Space Transformation: Similar to the feature engineering, but more extreme. Instead of looking at (variants of) coordinates, one can look at the image which is there in the end. This way one can apply CNNs.

You can do similar things for movies:

Multiple Classifications: Classify based on single frames and average the results
Feature-Engineering: Normalize the length to a fixed number; interpret the fixed-length video as an image with more channels.
Use CNNs as feature extractors / for dimensionality reduction and LSTM for the sequence classification.

You might be interested in Beyond Short Snippets: Deep Networks for Video Classification

Efficiency vs Effectiveness

Model A is more effective than model B, if its optimizarion criterion is better.

Model A is more efficient than model B, if it uses less resources (e.g. memory) to achieve the same optimization value (e.g. same accuracy / MSE)


Interpreting Keras Yolov3 config file How does one interpret the "min_input_size", "max_input_size" and "anchors" fields in the Yolov3 config file here. In particular, suppose we have the following:

    "min_input_size":       288,

    "max_input_size":       448,

    "anchors":              [55,69, 75,234, 133,240, 136,129, 142,363, 203,290, 228,184, 285,359, 341,260]

Does the min_input_size and max_input_size indicate the maximum number of training images we can have? What do the numbers in the "anchors" field indicate? Are they the coordinates of the anchor boxes? Surprisingly, I have not been able to find a good explanation of many of these  fields within this file.

The min and max input size should be the min and max image size of the input images. The numbers represent pixels in both axis of the image.

The anchors represents the size of the anchor box. Anchor box does not have coordinates, only have size.

Hope I can help you


How to best make use of learning rate scheduling in reinforcement learning? How to best make use of learning rate scheduling in reinforcement learning?

To me, a low learning rate towards the end to fine-tune what you've learned with subtle updates makes sense. But I don't see why over training time this should be linearly brought down. Wouldn't this increase overfitting too, as it promotes an early adopted policy to get further and further finetuned for the rest of the training?
Wouldn't it be better to keep it constant over the entire training so that when the agent finds novel experiences later, it still has a high enough learning rate to update its model?

I also don't really know how these modern deep RL papers do it. The starcraft II paper by DeepMind, and the OpenAI hide and seek paper don't mention learning rate schedules for instance.

Or are there certain RL environments where it's actually best to use something like a linear learning rate schedule?

I have not used learning rate schedules, but I do have experience with adjustable learning rates.
The Keras callback ReduceLROnPlateau is useful for adjusting the learning rate. If you use it to monitor the validation loss versus training loss, you will avoid the danger of overfitting. Also, you can use the ModelCheckpoint callback to save the model with the lowest validation loss and use that to make predictions. The documentation is here.
I look at the validation loss as a deep valley in $N$ space, where $N$ is the number of trainable parameters. As you progress down the valley, it becomes increasingly narrower, so it is best to reduce the learning rate to get further down the valley (closer to the minimum). With the adjustable learning rate, you can start with a larger initial rate that converges faster, then reduces as needed to achieve a minimum loss.
I wrote a custom callback that initially monitors the training loss and adjusts the learning rate based on that until the training accuracy achieves 95%, then it switches to adjusting the learning rate based on validation loss.
I also am experimenting with a slightly different approach to training. On a given epoch, assume the quantity you are monitoring does NOT improve. That means that you have moved to a point in $N$ space (value of the weights) that is NOT as "good" as the point you were at in the previous epoch. So, instead of training from the point you are at in the current epoch, I set the weights back to what they were for the previous (better) epoch, reduce the learning rate then continue training from there. This appears to work rather well.


Why is there an expectation sign in the Bellman equation? In chapter 3.5 of Sutton's book, the value function is defined as:

Can someone give me some clarification about why there is the expectation sign behind the entire equation? Considering that the agent is following a fixed policy $\pi$, why there should be an expectation when the trajectory of the future possible states is fixed (or maybe I am getting it wrong and it's not). In total, if the expectation here has the meaning of averaging over a series of trajectories, what are those trajectories and what are the weights of them when we want to compute the expected value over them according to this Wikipedia definition of the expected values?

There needs to be an $E_{\pi}$ over the infinite discounted return term because of two reasons-

The policy could be stochastic in nature. That is, for any given state $s_t$ at time $t$, the policy $\pi(s_t)$ does not provide a deterministic action $a$, but rather, it provides us with a distribution over the possible next states, that is the action at time $t$, $a_t$ is distributed as-
$$a_t \sim \pi(s_t)$$
Even if the policy $\pi$ being followed by an agent is deterministic, there still needs to be an expectation over the behavior of the underlying stochastic MDP environment. That is, any action $a_t$, in general, only provides us with a distribution over the possible next states of the agent. That is,
$$P(s_{t + 1} = s') = P_{\pi}(s' | s_t) = \sum_{a \in A} T(s,a,s') \times P_{\pi}(a_t = a)$$
Here $T(s, a, s')$ is the transition function for the MDP and the above equation captures the stochasticity arising from both 1 and 2.

As you see the expectation does not have to do with averaging over a collection of trajectories. However, that idea is often used in Monte-Carlo estimation of value functions.

EDIT:
As pointed out in the comments, it is not correct to say that the expectation is not over a collection of trajectories.


Has reinforcement learning been used to prove mathematical theorems? Coq exists, and there are other similar projects out there. Further, Reinforcement Learning has made splashes in the domain of playing games (a la Deepmind & OpenAI and other less well-known efforts).
It seems to me that these two domains deserve to be married such that machine learning agents try to solve mathematical theorems. Does anyone know of any efforts in this area?
I'm a relative novice in both of these domains, but I'm proficient enough at both to take a stab at building a basic theorem solver myself and trying to make a simple agent have a go at solving some basic number theory problems. When I went to look for prior art in the area I was very surprised to find none. I'm coming here as an attempt to broaden my search space.

Artificial Intelligence for Theorem Proving is an active research area as witnessed by the existence of the AITP conference and of many publications on the topic. Some papers are mentioned in this thread: https://coq.discourse.group/t/machine-learning-and-hammers-for-coq/303. I haven't read these papers myself, so I cannot point you to a paper using reinforcement learning specifically, but given the important activity in this domain, I would be very surprised if it hadn't been attempted.


Are these two versions of back-propagation equivalent? Just for fun, I am trying to develop a neural network.

Now, for backpropagation I saw two techniques.

The first one is used here and in many other places too.

What it does is:

It computes the error for each output neuron.
It backpropagates it into the network (calculating an error for each inner neuron).
It updates the weights with the formula: $\Delta w_{l, m, n}=k \cdot E_{l+1, n} \cdot N_{l, m}$, where

$\Delta w_{l, m, n}$ is the change in weight,
$k$ is the learning speed,
$E_{l+1, n}$ is the error of the neuron receiving the input from the synapse, and
$ N_{l, m}$ is the output sent on the synapse.

It repeats for each entry of the dataset, as many times as required.

However, the neural network proposed in this tutorial (also available on GitHub) uses a different technique:

It uses an error function (the other method does have an error function, but it does not use it for training).
It has another function which can compute the final error starting from the weights.
It minimizes that function (through gradient descent).

Now, which method should be used?

I think the first one is the most used one (because I saw different examples using it), but does it work as well?

In particular, I don't know:

Isn't it more subject to local minimums (since it doesn't use quadratic functions)?
Since the variation of each weight is influenced by the output value of its output neuron, don't entries of the dataset which just happen to produce higher values in the neurons (not just the output ones) influence the weights more than other entries?

Now, I do prefer the first technique, because I find it simpler to implement and easier to think about.

Though, if it does have the problems I mentioned (which I hope it doesn't), is there any actual reason to use it over the second method?

The two examples present essentially the same operation:

In both cases, the network is trained with gradient descent using the backpropagated squared error computed at the output.
Both examples use the logistic function for node activation (the derivative of the logistic function $s$ is $s(1 - s)$. This derivative is obviously very easy to compute, and this is part of the reason why it was so widely used (these days the ReLU activation function is more popular, especially with convolutional networks).
The first method also uses momentum.

The main difference I can see is that in the first case backpropagation is iterative while in the second example it is performed in batch mode.

The last video in the series by Welch Labs introduces a quasi-Newtonian method which offers the advantage of finding the minimum of the cost function by computing the Hessian (matrix of second-order derivatives of the error with respect to the weights). However, this feels like comparing apples and oranges - the vanilla gradient descent does not use second-order information.


In deep learning, is it possible to use discontinuous activation functions? In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)?
(My guess: for example, ReLU is non-differentiable at a single point, but it still has a well-defined derivative. If an activation function has a jump discontinuity, then its derivative is supposed to have a delta function at that point. However, the backpropagation process is incapable of considering that delta function, so the convex optimization process will have some problem?)

Even the first artificial neural network - Rosenblatt's perceptron [1] had a discontinuous activation function. That network is in introductory chapters of many textbooks about AI. For example, Michael Negnevitsky. Artificial intelligence: a guide to intelligent systems. Second Edition shows how to train such networks on pages 170-174.

Error backpropagation algorithm can be modified to accommodate discontinuous activation functions. The details are in paper [2]. That paper points out a possible application: training a neural network on micro-controllers. As the multiplication of the output of the previous layer $x_j$ by the weigth $w_{ij}$ is exspensive, the author suggested to approximate it with a left shift by $n$ bits (multiplication by $2^n$) for the corresponding $n$ in which case the activation function is discontinuous (a staircase).

An example of a neural network with discontinuous activation functions applied to a restoration of degraded images is in Ref. [3]. Applications of recurrent neural networks with discontinuous activation functions to convex optimization problems are in Ref. [4]. Probably more examples can be found in the literature.

References

Rosenblatt, F. The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev. 1958 Nov; 65(6):386-408. PMID: 13602029 DOI: 10.1037/h0042519
Findlay, D.A. Training networks with discontinuous activation functions. 1989 First IEE International Conference on Artificial Neural Networks, (Conf. Publ. No. 313), London, UK, 1989, pp. 361-363.
Ferreira, L. V.; Kaszkurewicz, E.; Bhaya, A. Image restoration using L1-norm regularization and a gradient-based neural network with discontinuous activation functions. 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, 2008, pp. 2512-2519.  DOI: 10.1109/IJCNN.2008.4634149
Liu, Q.; Wang, J. Recurrent Neural Networks with Discontinuous Activation Functions for Convex Optimization. Integration of Swarm Intelligence and Artificial Neural Network, pp. 95-119 (2011) DOI: 10.1142/9789814280150_0004


Calculating Parameter value Using Gradient Descent for Linear Regression Model Consider the following data with one input (x) and one output (y):
  (x=1, y=2)
  (x=2, y=1)
  (x=3, y=2)
  Apply linear regression on this data, using the hypothesis $h_Θ(x) = Θ_0 + Θ_1 x$, where $Θ_0$ and $Θ_1$ represent the parameters to be learned. Considering the initial values $Θ_0$= 1.0, and $Θ_1$ = 0.0, and learning rate 0.1, what will be the values of $Θ_0$ and $Θ_1$ after the first three iterations of Gradient Descent

From least squares method I took the derivative with respect to $Θ_0$ and $Θ_1$ and plugged in the initial values to get the slope/intercept and multiplied it by the learning rate 0.1 to get the step size.The step size was used to calculate the new $Θ_0$ and $Θ_1$ values.

I am getting $Θ_0$ as 1.7821 when following the above. Please let me know if the approach followed and the solution correct or there is a better way to solve

X = np.array([1,2,3])
Y = np.array([2,1,2])

params = np.array([1, 0])

def loss(y, yhat):
    return ((y - yhat)**2).mean()

def model(x):
    return params[0] + params[1]*x

def loss_grad(y, yhat, x):
    return np.array([(2*(yhat-y)).mean(), (2*(yhat-y)*x).mean()])

lr = .1
for _ in range(3):
    yhat = model(X)
    l = loss(Y, yhat)
    g = loss_grad(Y, yhat, X)
    params = params - lr*g
    print(f'thetas are now {params} with new loss of {loss(Y, yhat)}')

outputs

thetas are now [1.13333333 0.26666667] with new loss of 0.6666666666666666
thetas are now [1.13333333 0.23111111] with new loss of 0.2696296296296296
thetas are now [1.14755556 0.22874074] with new loss of 0.262887242798354

I double checked this with keras too, but in numpy i explicitly wrote the gradients, i reccomend double checking your gradient or arithmetic


How do I optimize the number of filters in a convolution layer? I’m trying to figure out how to write an optimal convolutional neural network with respect to maximizing and minimizing filters in a convolution 2D layer. This is my thinking and I’m not sure if it's correct.

If we have a dataset of 32x32 images, we could start with a Conv2D layer, filter of 3x3 and stride of 1x1. Therefore the maximum times this filter would be able to fit into the 32 x 32 images would be 30 times e.g. newImageX * newImageY

newImageX = (imageX – filterX + 1)
newImageY = (imageY – filterY + 1)

Am I right in thinking that because there are only newImageX * newImageY patterns in the 32 x 32 image, that the maximum amount of filters should be newImageX * newImageY, and any more would be redundant?

So, the following code is the maximum possible filters given that we have 3x3 filter, 1x1 stride and 32x32 images?

Conv2D((30*30), kernel=(3,3), stride=(1,1), input_shape=(32,32,1))

Is there any reason to go above 30*30 filters, and are there any reasons to go below this number, assuming that kernel, stride and input_shape remain the same?

If you knew you were looking for one specific filter, would you have to use the maximum amount of filters to ensure that the one you were looking for was present, or can you include it another way?

Am I right in thinking that because there are only newImageX * newImageY patterns in the 32 x 32 image, that the maximum amount of filters should be newImageX * newImageY, and any more would be redundant?

Your assumption is wrong. If you have a $32 \times 32$ images (so consider only grayscale images), then you have $256^{32 \times 32}$ possible patterns (i.e. a huge number of possible patterns!), given that, for each pixel, you can have $256$ values, and there are $32\times 32$ pixels. Of course, for a human, many of these patterns may be very similar to each other (or even indistinguishable), but the way you measure similarity, in this case, is another hard problem. Hence also the difficulty in choosing the number of filters.


How is the probability transition matrix populated in the Markov process (chain) for a board game? Following on from my other (answered) question:

With regards to the Markov process (chain), if an environment is a board game and its states are the various position the game pieces may be in, how would the transition probability matrix be initialised? How would it be (if it is?) updated?

Transition probability matrix cannot be initialized. Your game world has some rules. Since we do not know these rules we can approximate them. To do this, we should run the game over and over and collect the outcome of each action.

Here is a toy example: assume we control a plane. We want the plane to fly forward, but sometimes there may be wind in a random direction. So the plane won't end up in the desired position. Instead, it has some probability to fly left, right and forward. After the data is collected, we can estimate this probabilities.

Another example is estimating n-gram model: assume we want to train a model that will suggest a word in search field. We would analyze a corpus and calculate all 2 word sequences (for bigram model). Then, if a user starts typing I the model would suggest the word am, because I am occurs more frequently then I did, for example, and hence, the transition probability from I to am is greater.

Here is good book on Language Modeling

But your question suggest me that you want something like reinforcement learning. Check out this video wich is a part of a great lectures series on RL by David Silver


What do the state features of KukaGymEnv represent? I trying to use DDPG augmented with Hindsight Experience Replay (HER) on pybullet's KukaGymEnv.
To formulate the feature vector for the goal state, I need to know what the features of the state of the environment represent. To be precise, a typical state vector of KukaGymEnv is an object of the numpy.ndarray class with a shape of (9,).
What do each of these 8 elements represent, and how can I formulate the goal state vector for this environment? I tried going through the source code of the KukaGymEnv, but was unable to understand anything useful.

Here's an incomplete answer, but it may help.
Your state is read by the function getExtendedObservation(). This function makes two things : it calls the function getObservation() from this source code, gets a state, and extend this state with three components :

relative x,y position and euler angle of block in gripper space

But what are the 5 first components returned by getObservation()? From what I read, there are positions, then euler angles describing the orientation. But that would make 6 + 3 = 9 features, so there is either only 2 positions, or only 2 euler angles. You may know kuka better than me and know the answer of this one :).
So, to sum up :
state = [X, Y, (Z, ) , Alpha, Gamma, (Beta, ), gripX, gripY, gripAlpha]

(Either Z or Beta is absent)


What is meant by "known" in "A knowledge-base exhibits complete knowledge if and only if, for every $P$, $P$ or $\neg P$ is known"? I have a question as to what it means for a knowledge-base to be consistent and complete. I've been looking into non-monotonic logic and different formalisms for it from the book "Knowledge Representation and Reasoning" by Hector Levesque and Ronald J. Brachman, but something is confusing me.
They say:

We say a KB exhibits consistent knowledge iff there is no sentence $P$ such that both $P$ and $\neg P$ are known. This is the same as requiring the KB to be satisfiable. We also say that a KB exhibits complete knowledge iff for every $P$ (within its vocabulary) $P$ or $\neq P$ is known

They then seem to suggest that by "known" they mean "entailed". They say

In general, of course, knowledge can be incomplete. For example, suppose KB consists of a single sentence ($P$ or $Q$). Then KB does not entail either $P$ or $\neg P$, and so exhibits incomplete knowledge.

But when dealing with sets of sentences, I usually see these terms as being defined w.r.t. derivability and not entailment.
So my question is, what exactly do these authors mean by "known" in the above quotes?
Edit: this post helped clarify things.

It seems that they are stating that a knowledge base is consistent if and only if it never asserts the truth of both the truth and the negation of a particular P. In other words, a knowledge base is consistent if it never contradicts itself. Their definition allows incomplete knowledge bases to be considered consistent; by their definition, an empty knowledge base is still considered a consistent one.


How to handle multiple types of decisions? In lots of games there are multiple phases or decision points that are not similar yet seem to have a dependency on one another when taking the perspective of the overall strategy of the player. A couple examples I thought up:

In a simple draw poker, you can have a strategy for discarding cards and a strategy for betting. They may not be mutually exclusive if you know your opponents betting will change with the number of cards you draw.
In Cribbage there are two phases, Discard to crib and the Play. The Play phase is definitely dependent on which cards are discarded in the discard phase. So it seems knowledge of Play strategy would be needed to make the Discard decision.

The intent is to learn how to set up an unsupervised learning algorithm to play a game with multiple types of decision making. Doesn't matter the game. I'm at a loss at the highest level in what ML models to learn to use for this scenario. I don't think a single NN would work because of the different decision types.

My question is how are these dependencies handled in ML? What are some known algorithms/models that can handle this?

I'm at a loss on what to even search for so feel free to dump some terminology and keywords on me. =)

Your intuition is correct, neural networks are a no go (except see bottom).

It seems like you'd want to look into the ML sub-field called Reinforcement Learning.
In a nutshell, RL offers a set of methods to learn what is the best action to take in a given situation.

More formally, in RL settings the algorithm learns from experience by observing a reward ($R$) associated to an action ($a$).

RL problems can be conceptualized as Markov Decision Processes (MDPs). From Sutton & Barto:

  MDPs are a classical formalization of sequential decision making, where actions influence not just immediate rewards, but also subsequent situations, or states, and through those future rewards.

MDP can be faced with several methods, mainly Monte Carlo Methods and Temporal-Difference Learning.

In short, both use the Bellman equation in different ways. A key component of the eq. is the discounting term ( $\gamma$ ). Values for rewards associated with future states are discounted (multiplied) by $\gamma<1$ in order to modulate the importance of future rewards.

Since you're concerned with strategy, another way to influence the agent's learning wrt. the environmental rewards is the adoption of an epsilon value $\epsilon<1$. When defining an $\epsilon$-greedy policy, the agent will choose the action with highest value $Q$ only with a probability equal to $1-\epsilon$. This enables the agent to balance exploitation of previous experience with the exploration of the environment. Very useful in cases where an immediate low reward compromises the achievement of a much higher reward later during the episode.

There's many other ways to influence the behaviour of your learner of choice to respond differently to its environment. For a complete view, I recommend Sutton and Burton who offer their book in pdf format for free at the given link.

Edit: almost forgot. Neural networks can be useful when combined with more classical RL methods, say Q-learning (a type of Temporal-difference learning), taking on the name of deep Q-learning.
Alpha go also uses a combination of Monte Carlo methods combined with 2 NNs for Value evaluation and policy evaluation Wikipedia.
I wouldn't necessarily venture in this domain without first having a clear overview of classical RL methods.

Hope this helps.


Is continuous learning possible with a deep convolutional neural network, without changing its topology? In general, is continuous learning possible with a deep convolutional neural network, without changing its topology?
In my case, I want to use a convolutional neural network as a classifier of heartbeat types. The ECG signal is split, and a color image is created using feature extraction. These photos (the inputs) are fed into a deep CNN, but they must be labeled by someone first.
Are there ways to implement continuous learning in a deep neural network for image recognition? Does such an implementation make sense if the labels have to be specially prepared in advance?

In general, is continuous learning possible with a deep convolutional neural network, without changing its topology?

Your intuition that it is possible to perform incremental (aka continual, continuous or lifelong) learning by changing the NN's topology is correct. However, dynamically adapting the NN's topology is just one approach to continual learning (a specific example of this approach is DEN). So, there are other approaches, such as

regularization approaches (examples are LwF, EWC and SI)

ensemble approaches (examples are Learn++, FEL and PathNet)

rehearsal approaches (an example is iCaRL)

For more details about these and other approaches (and problems related to continual learning and catastrophic forgetting in neural networks), take a look at this very nice review of continual learning approaches in neural networks. You should also check this answer.

Are there ways to implement continuous learning in a deep neural network for image recognition?

Yes. Many of the approaches focus on image recognition and classification, and often the experiments are performed on MNIST or similar datasets (e.g. see this paper).

Does such an implementation make sense if the labels have to be specially prepared in advance?

Yes, you can prepare your dataset in advance, and then later train incrementally (in fact, in the experiments I have seen in some of these papers, they usually do this to simulate the continual learning scenario), but I am not sure about the optimality of this approach. Maybe with batch learning (i.e. the usual offline learning where you train on all data), you would achieve higher performance.