Why does the DQN not converge when the start or goal states can change dynamically? I'm trying to apply a DQN to a stochastic environment, but I'm having trouble getting it to converge.
I found some similar questions asked here, but no solutions yet.
I can easily get the DQN to converge on a static environment, but I am having trouble with a dynamic environment where the end is not given.
Example: I have made a really simple model of the Frozen Lake (without any holes) - simply navigation from A to B. This navigation works fine when A and B are always the same, but when I shuffle the position of A or B for each session, the DQN cannot converge properly.
I am using the grid (3x3, 4x4 sizes) as input neurons. Each with "0" value. I assign the current position "0.5" and the end position "1". 4x4 grid gives us 16 input neurons. Example of 3x3 grid:
 0.5  0  0
  0   0  0
  0   0  1

I have a few questions in this regard:

When training the DQN, how do I apply Q-values? (Or do I really need to? I'm not sure how to correctly "reward" the network. I'm not using any adversarial network or target network at this point.)

I train the network using only a short replay memory of the last move, or the last N moves that led to success. Is this the right way to approach this?

I use Keras, and am simply training the network every time it does something right - and ignoring failed attempts. - But is this anywhere near the right approach?

Am I missing something else?

Perhaps I should note that my math skills are not that strong, but I try my best.
Any input is appreciated.

Your problem is not that the environment is stochastic or dynamic. In fact you are using the terms slightly incorrectly. These terms do not usually refer to the fact that starting state can differ or goal locations can move episode-by-episode. They typically refer to behaviour of state transitions.

Although in your case you could view the initial state as stochastic, this is not a big deal, and not likely to be the cause of your problems.

From your questions, it seems to me that you are not really running a DQN algorithm yet. It is not 100% clear what your neural network is predicting, but my best guess is that you have 4 outputs to select "best" action and are treating the neural network as a classifier. This training approach seems closest to Cross Entropy Method (CEM) due to how you are selecting "successful" navigation only.

  When training the DQN, how do i apply Q-values?

This question is the most revealing that you are not using DQN. This is too complex to describe in full in an answer, but the basics are:

Your neural network (NN) should be estimating Q values. Typically in DQN, you input the state and the NN outputs an array of estimates for Q of each action (although other architectures are possible). This should be a regression problem, so last layer of network needs to be linear.
Current best guess of optimal policy is to run the NN forward and find the maximising action.
In DQN you also have a "behaviour policy" - a simple and popular choice is to use $\epsilon$-greedy action selection, which just means to take the maximising action (as calculated above), except with probability $\epsilon$ (some small value, e.g. 0.1) to take a random action.
To figure out your training data to improve the NN, you need Q values to calculate a TD Target. In single-step Q learning that would be $r + \gamma Q(s',a*)$ where $r$ is the immediate reward $s'$ is the next state seen, and $a*$ is the maximising action in that state. You should force $Q(s', a*) = 0$ (i.e. not use the NN) if $s'$ is a terminal state.

This means you typically need to work with Q values in 2 or 3 places in your inner loop. Your inner loop should look something like this per time step, given a current state current_state:

# Figure out how to act
current_q_values = NN_predict(current_state)
current_action = argmax(current_q_values)
if random() < epsilon:
  current_action = random_action()

# Take an action
reward, next_state, done = call_environment(current_state, current_action)

# Remember what happened
store_in_replay_memory(current_state, current_action, reward, next_state, done)

# Train the NN from memory
repeat N times: # This can be vectorised for efficiency
  mem_state, mem_action, mem_reward, mem_next_state, mem_done = sample_replay_memory()
  mem_q_values = NN_predict(mem_next_state)
  mem_max_action = argmax(mem_q_values)
  if done:
    td_target = mem_reward
  else
    td_target = mem_reward + gamma * mem_q_values[mem_max_action]
  target_q_values = NN_predict(mem_state)
  target_q_values[mem_action] = td_target
  NN_train(mem_state, target_q_values)

# Maybe end an episode (this can include generating new map)
if done:
  current_state = reset_environment()
else:
  current_state = next_state

You can see above, NN_predict is called three different times to get Q values in slightly different contexts. I have ignored extras such as using a separate target network.

  I train the network using only a short replay memory of the last move, or the last N moves that led to success. Is this the right way to approach this?

It is important to include moves that lead to failure so that the NN learns the difference. Typically you will need a replay memory with from a few hundred to a few hundred thousand entries. You would get away with a few hundred perhaps for your simple problem. The idea is to use this training data a bit like a dataset from supervised learning.

  I use Keras, and am simply training the network every time it does something right - and ignoring failed attempts. - But is this anywhere near the right approach?

This is not the right approach for DQN, although perhaps could be considered a crude version of CEM.


When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling? When learning off-policy with multi-step returns, we want to update the value of $Q(s_1, a_1)$ using rewards from the trajectory $\tau = (s_1, a_1, r_1, s_2, a_2, r_2, ..., s_n, a_n, r_n, s_n+1)$. We want to learn the target policy $\pi$ while behaving according to policy $\mu$. Therefore, for each transition $(s_t, a_t, r_t, s_{t+1})$, we apply the importance ratio $\frac{\pi(a_t | s_t)}{\mu(a_t | s_t)}$.
My question is: if we are training at every step, the behavior policy may change at each step and therefore the transitions of the trajectory $\tau$ are not obtained from the current behavior policy, but from $n$ behavior policies. Why do we use the current behavior policy in the importance sampling? Should each transition use the probability of the behavior policy of the timestep at which that transition was collected? For example by storing the likelihood $\mu_t(a_t | s_t)$ along with the transition?

According to my understanding, you don't use just the current behavior policy for sampling. The importance sampling ratio is calculated as the product of the probability ratios for both the target and behaviour policy throughout the trajectory.
See the calculation below, where the product is happening for all the probabilities throughout the trajectories. (screenshot from Chapter 5, section 5.5 (page 85) of Sutton & Barto)


What are the implications of the statement "If you can't tell, does it matter?" in relation to AI? "If you can't tell, does it matter?" was one of the first lines of dialogue of the Westworld television series, presented as a throwaway in the first episode of the first season, in response to the question "Are you real?"
In the sixth episode of the third season, the line becomes a central realization of one of the main characters, The Man in Black.
This is, in fact, a central premise of the show—what is reality, what is identity, what does it mean to be alive?—and has a basis in the philosophy of Philip K. Dick.

What are the implications of this statement in relation to AI?  In relation to experience?  In relation to the self?

The question in this video is

Are you real?

What does this question really mean? Is the guy asking whether the apparent female (I don't know if she is a cyborg or not because I did not yet watch the TV series) is a human? So, is "real" a synonym for "human"? If that's the case, then the first implication (in the form of a question) of the statement

If you can't tell, does it matter?

in relation to AI is

Can we create an AGI that is sufficiently similar to a human that we can't tell whether it's a human or not (by just normally interacting with it)?

Of course, it's not clear what we mean by "normally interacting". As far as I remember, this issue is also raised in the film A.I. Artificial Intelligence, where the AI (the kid interpreted by Haley Joel Osment) looks sufficiently real to the other kid, so he behaves as if he was a human kid, but then the human kid understands that he's a machine, and starts to behave differently (I hope I'm remembering the film correctly).
So, the second question that we could ask is

Once we understand that it's not a human (for example, because it's made of other substances), would we humans start to behave differently and start treating the AGI differently?

As opposed to the first question, which is still an open problem, this second question can probably be answered by looking at our relationships with other humans (or entities, such as other animals). Often, we have an idea of a person. Once we discover something new about that person, which maybe we dislike, we may start to treat that person differently. I think this would very likely also happen in our eventual relationship with a sufficiently advanced AGI too, as depicted in the mentioned film.
Now, let me try to address the other question

What are the implications of this statement in relation to experience, in relation to the self?

I think that you're asking whether a sufficiently advanced AGI could be considered conscious or not. Of course, this is a very hard question to answer, because we still don't have a clear definition of consciousness or we don't yet agree on a standard definition, so I don't really have a definitive answer to this question. However, if consciousness is just a byproduct of perception and the ability to understand the world and its (physical) rules, then an AGI could be conscious (in a similar way that humans are also conscious). However, consciousness may not actually be necessary to correctly act in the world. In any case, the AI probably needs to know that it has a body and that it needs to protect it for its survival, if that's its main goal.


Why do action recognition algorithms perform better on ucf101dataset than HMDB51 dataset? If we look at state of the art accuracy on the UCF101 data set, it is around 93% whereas for the HMDB51 data set it is around 66%. I looked at both the data sets and both contain videos of similar lengths. I was wondering if anyone could give an intuition as to why HMDB51 data set has been harder.

It is true that at first look, one could expect that classification between 101 categories would be harder than classification between 51 categories. However, many aspects play a role when it comes to action recognition applications.

For instance, the HMDB51 contains several categories about different facial movements like smiling, laughing, chewing,... and several other categories like eating, drinking. Such categories are not present in the list of categories of the data set UCF101 and are obviously among the most difficult categories to deal with. It also claims to have some bad quality challenging videos.

It is hard to predict in advance how difficult a data set will be to classify. We can imagine that when the state-of-the-art reach accuracy beyond 90%, it is time to build a data set that makes these methods fail to look for even more robust solutions. I don't know these data sets well, but the videos are present most probably more variability in terms of viewpoint, camera motions, illumination changes, image quality,... in the most difficult to classify data set.

Also, check on this page, the results announced for the UCF101 data set. I don't know where you found you accuracy value because the official website announces less than 43.9%. Some publications do not use the complete data set and use only part of it to show the performance of an approach they designed.

Finally, the official website of the HMDB51 data set reports the following:
"The UCF group has also been collecting action datasets, mostly from YouTube. There are UCF Sports featuring 9 types of sports and a total of 182 clips, UCF YouTube containing 11 action classes, and UCF50 contains 50 actions classes. We will show in the paper that videos from YouTube could be very biased by low-level features, meaning low-level features (i.e., color and gist) are more discriminative than mid-level fears (i.e., motion and shape)."
This could also explain why better results can be achieved...


Use cross-validation to train after model selection I have been recently reading about model selection algorithms (for example to decide which value of the regularisation parameter or what size of a neural network to use, broadly hyper-parameters). This is done by dividing the examples into three sets (training 60%, cross-validation 20%, test 20%) and training is done on the data with the first set for all parameters, and then choose the best parameter based on the result in the cross-validation and finally estimate the performance using the test set.

I understand the need for a different data-set compared to training and test for select the model, however, once the model is selected, why not using the cross-validation examples to improve the hypothesis before estimating the performance?

The only reason I could see is that this could cause the hypothesis to worsen and we wouldn't be able to detect it, but, is it really possible that by adding much more examples (60% -> 80%) the hypothesis gets worse?

You are quite correct. If you have properly followed the Cross Validation procedure and selected the best model indeed, then you can use the CV set as the training set for the final model. And no it will not cause your hypothesis to worsen (for that set maybe, but not for new examples) if you have selected the model correctly. In-fact you may use the entire 100% of the data-set.

Justin Johnson a TA at Stanford University answered a similar type of question on training CNN's using 100% of the data-set. He said that if you had enough computational resources and want to squeeze that extra 1% or 2% accuracy from your model you can use the entire data-set after model selection.

NOTE: As @NeilSlater pointed out, if you need the model for reporting purposes you should only use 80% of the data-set, otherwise you'll lose your only source for unbiased model verification. But if you are looking to deploy the model on field you can use 100% of the data-set.


Are there any good tutorials about training RL agent from raw pixels using PyTorch? Is there any good tutorials about training reinforcement learning agent from raw pixels using PyTorch?

I don't understand the official PyTorch tutorial. I want to train the agent on the atari breakout environment. Unfortunately, I failed to train the agent on the RAM version. Now, I am looking for a way to train the agent from raw pixels.

I am using Jetson reinforcement for my quadcopter reinforcement in simulation. Maybe it will help you. because you can create AI agents that can learn from the interactive environment, gather experience, and system of reward with deep RL. You can also use end-to-end neural networks that translate raw pixels into action as per your need and use that RL-trained agent to complete complex tasks.

The best thing is you can easily transfer RL-agent which simulated in the simulator to real-world robots. and We are using multiple Nano's easily to perform the complex tasks of navigation and co-ordination of Our Quad-copter.

Webinar of Deep RL on Jetson

Google group to get help for Deep RL on jetson nano

Hope it helps you.


In attention models with multiple layers, are weight matrices shared across layers? In articles that describe neural architectures with multiple attention layers of the same form, are the weight matrices usually the same across the layers?  Consider as an example, "Attention is all you need". The authors stack several layers of multi-head self-attention in which each layer has the same number of heads.  Each head $i$ involves a trainable weight matrix $W_{i}^{Q}$.  There is no subscript, superscript, or any other indication that this matrix is different for each layer.  My questions is this: are there separate $W_{i}^{Q}$ for layers $1,2,3,...$ or is this a single matrix shared throughout layers?
My intuition is that the authors of the paper wanted to cut down on notation, and that the matrices are different in different layers.  But I want to be sure I understand this, since I see the same kind of thing in many other papers as well.

Weights are not normally shared across Transformer layers in vanilla Transformers. However, there has been research done in testing out sharing weights, and sometimes they improve the scores. Here are some examples:
ALBERT is an improvement on BERT (so only uses the encoding side, no decoder), and shows that sharing the attention weights only $\left\{ W_i^Q, W_i^K, W_i^V \right\}$ across all Transformer layers for large networks either result in the same accuracy or slightly improved accuracy, while significantly reducing model size. Sharing the position-wise FFN layer though hindered performance.
Text-to-Text Transfer Transformer shows that sharing the weights between encoder and decoder layer of the transformer (so e.g. layer 1 encoding weights = layer 1 decoding weights) barely affected accuracy (it dropped by 0.5%), but the model size is halved.
I'm sure there are more papers I have forgotten. Sharing weights is still an active area of research, but for vanilla transformers it is assumed they do not share weights.


Is there a limit to the increase of intelligence? Some argue that humans are somewhere along the middle of the intelligence spectrum, some say that we are only at the very beginning of the spectrum and there's so much more potential ahead.

Is there a limit to the increase of intelligence? Could it be possible for a general intelligence to progress infinitely, provided enough resources and armed with the best self-recursive improvement algorithms?

Absolutely, regardless of how you define "intelligence".

If intelligence is merely information, as in "a piece of intelligence", as in data, or an algorithm, the structure is finite.  (Structure, here, refers to the information, which may be reduced to a single string in either case.)

See: Turning Machine.

If intelligence is the rational capability of an automata, it is likewise bounded by the tractability of the decision problem, the structure of the algorithm, and the time available to make the decision.

See: Bounded Rationality

Both answers are really the same, because "intelligence" in the first sense is limited by physical constraints on information density, sophistication of the algorithm, and time.

See also: Computational complexity of mathematical operations, Computational complexity, Time Complexity


Why is the evidence equal to the KL divergence plus the loss? Why is the equation $$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$$ true, where $x^i$ are data points and $z$ are latent variables?
I was reading the original variation autoencoder paper and I don't understand how the marginal is equal to the RHS equation. How does the marginal equal the KL divergence of $p$ with its approximate distribution plus the variational lower bound?

In variational inference, the original objective is to minimize the Kullback-Leibler divergence between the variational distribution, $q(z \mid x)$, and the posterior, $p(z \mid x) = \frac{p(x, z)}{\int_z p(x, z)}$, given that the posterior can be difficult to directly infer with the Bayes rule, due to the denominator term, which can contain an intractable integral.

Therefore, more formally, the optimization objective can be written as

\begin{align}
q^*(z \mid x) = \operatorname{argmin}_{q(z \mid x)} D_{\text{KL}}(q(z \mid x) \| p(z \mid x))\tag{1} \label{1}
\end{align}

However, solving this optimization problem can be as difficult as the original inference one of computing the posterior $p(z \mid x)$ using the Bayes rule, given that it still involves the possibly intractable term $p(z \mid x)$.

If you use the definition of the KL divergence, you can derive the following equation

\begin{align}
D_{\text{KL}}(q(z \mid x) \| p(z \mid x)) = \mathbb{E}_{q(z \mid x)} \left[ \log q(z \mid x) \right] - \mathbb{E}_{q(z \mid x)} \left[ \log q(z, x) \right] + \log p(x) \tag{2} \label{2}
\end{align}

First, note that the expectations are with respect to the variational distribution, which means that, if you want to approximate these expectations with Monte Carlo estimates, you can do it with respect to the variational distribution, and, given that it is assumed that one can easily sample from the variational distribution (which can e.g. be a Gaussian), this is a nice feature.

Second, the KL divergence contains the term $p(x) = \int_z p(x, z)$, the denominator term in the Bayes rule to compute the posterior $p(z \mid x)$, which (as I said) can be intractable. $p(x)$ is often called the evidence.

The solution is then to optimize an objective that does not contain this annoying intractable term $p(x)$. The objective that is optimized is the so-called ELBO objective

\begin{align}
\text{ELBO}(q) = \mathbb{E}_{q(z \mid x)} \left[ \log q(z, x) \right] - \mathbb{E}_{q(z \mid x)} \left[ \log q(z \mid x) \right]\tag{3} \label{3}
\end{align}

The KL divergence \ref{2} and the ELBO objective \ref{3} are similar. In fact, ELBO is an abbreviation for Evidence Lower BOund, because the ELBO is a lower bound on the evidence $p(x)$, i.e. it is a number that is smaller than $p(x)$ or, more formally, $\text{ELBO}(q) \leq \log p(x)$. Therefore, if we maximize $\text{ELBO}(q)$, we also maximize the evidence $p(x)$ of the data (where $x$ is the data in your dataset).

So, the objective in variational inference is

\begin{align}
q^*(z \mid x)
&= \operatorname{argmax}_{q(z \mid x)} \operatorname{ELBO}({q}) \\
&= \operatorname{argmax}_{q(z \mid x)}  \mathbb{E}_{q(z \mid x)} \left[ \log q(z, x) \right] - \mathbb{E}_{q(z \mid x)} \left[ \log q(z \mid x) \right]
\tag{4} \label{4}
\end{align}

First, note that \ref{4} only contains terms that depend on the variational distribution, so we got rid of intractable terms, which was our goal.

Second, note that, as opposed to \ref{1}, we are maximizing (or finding the parameters that maximize the objective).

The ELBO objective is actually the negative of \ref{2} plus the logarithm of the evidence term, $\log p(x)$ (and you can easily verify it), that is

\begin{align}
\text{ELBO}(q)  = -D_{\text{KL}}(q(z \mid x) \| p(z \mid x)) + \log p(x)
\end{align}

which can also be rearranged as

\begin{align}
\log p(x) = D_{\text{KL}}(q(z \mid x) \| p(z \mid x)) + \text{ELBO}(q) \tag{5}\label{5}
\end{align}

which is your equation (where $\text{ELBO}(q)$ is your $\mathcal{L}$). Therefore, your equation is true by definition, i.e. we define the ELBO such that \ref{5} is true. However, note that we haven't defined the ELBO in the way we dit only for the sake of it, but because it is a lower bound on the log evidence (and this follows from the fact that the KL divergence is never negative).


State aggregation methods In Sutton's RL:An introduction 2nd edition it says the following(page 203):

  State aggregation is a simple form of generalizing function
  approximation in which states are grouped together, with one estimated
  value (one component of the weight vector w) for each group. The value
  of a state is estimated as its group’s component, and when the state
  is updated, that component alone is updated. State aggregation is a
  special case of SGD (9.7) in which the gradient, rv̂(S t ,w t ), is 1
  for S t ’s group’s component and 0 for the other components.

and follows up with a theoretical example.

My question is, imagining my original state space is [1,100000], why can't I just say that the new state space is [1, 1000] where each of these numbers corresponds to an interval: so 1 to [1,100], 2 to [101,200], 3 to [201,300], and so on, and then just apply the normal TD[0] formula, instead of using the weights?

My main problem with their approach is the last sentence:

  in which the gradient, rv̂(S t ,w t ), is 1
  for St’s group’s component and 0 for the other components.

if v(s,w) is the linear combination of a feature vector and the weights (w), how does the gradient of that function can be 1 for a state and 0 for others? There are not as many w as states or groups of states.

e.g.: if my feature vector is 5 numbers between 0 and 100. For example (55,23,11,44,99) for a specific state, how do you choose a specific group of states for state aggregation?

Maybe what I'm not understanding is the feature vector. If we have a state space that is[1,10000] as in the random walk, what can be the feature vector? does it have the same size as the number of groups after state aggregation?

Using the book's random walk example, If you have a state space with $1000$ states and you divide them into $10$ groups, each of those groups will have $100$ neighboring states. The function for approximation will be
\begin{equation}
v(\mathbf w) = x_1w_1 + x_2w_2 + ... + x_{10}w_{10}
\end{equation}
Now, when you pick a state, the feature vector will be a one-hot encoded vector with $1$ that is placed in a position that depends on in which group does the chosen state belong. For example if you have state $990$ that state belongs in group $10$ so the feature vector will be
\begin{equation}
\mathbf x_t = [0, 0, ..., 0, 1]^T
\end{equation}
what this means is that the only weight that will be updated is weight $w_{10}$ because gradients for all other weights will be $0$ (that's because features for those weights are $0$)


What are possible ways to combat overfitting or improve the test accuracy in my case? I have asked a question here, and one of the comments suggested that this is a case of severe overfitting. I made a neural network, which uses residual boosting (which is done via a KNN), and I am still just able to get < 50% accuracy on the test set.
What should I do?
I tried everything from reducing the number of epochs to replacing some layers with dropout.
Here is the source code.

There are a few issues you need to address first.

Normalise your data. You should try and keep your values for each input in a good range, otherwise you're never going to train anything useful. A simple way of doing this could be to divide each value by the maximum value for that input. This will ensure they are between 0 and 1, or you could divide by the average so you get approximately 1. Whatever the case, you need to do something about decreasing the magnitude of your inputs (also make sure your are consistently scaling your data, especially during validation and training, otherwise your data will likely be misinterpreted by your model)
Optional: Augment your data. You have a very small dataset to work with here, if there's any way you can augment this to make it larger, you will likely get much better performance on validation and testing sets. Augmenting is just creating new data that is still representative of the task (as a visual example, you might augment your image dataset by flipping some images and including those in training as well as the unflipped versions)
Split your data into Training, Validation and Testing sets. Training is used for what you expect - training. Validation is used to validate your model on unseen data. The reason for the testing set is obviously you are going to try your hardest to tweak and modify the model so it performs best on your validation set, but this incurs a kind of bias as you are only tweaking for this specific set of data, which is not completely representative of the real world. So try your hardest to perform best on the validation set by changing model hyper-parameters, but confirm it's real use on the testing set. As a recommended split for your entire data set, I have had success with 10% testing, 20% validation and 70% training, but you might be better off with a different split.
Test different models. Train your network for a small amount of time (say only 10 or so epochs) and see how it performs based on the validation set. Keep in mind because your dataset is particularly small (only ~300 samples in your training set) you don't need to train for many epochs at all to get a good representation of how your model will perform (an epoch being a complete loop over all training data). Make sure when you're doing this you're training in batches.
Try different regularization methods. One of the primary ones being good initialization. Make sure you're initializing your parameters with something like Xavier initialization of similar. Other suggestions include: dropout (though you said you already tried this, maybe try different values for dropout), weight decay and batch normalization layers. To be honest though, I think one of the 4 steps above will solve this issue, so you likely won't need to do any of these.

I may have made some points for things you have already done as there was a lot of code for me to look over in your source. But the biggest issue I noticed was that your data wasn't normalised at all and that usually creates many issues, so definitely try that first if you haven't already.


Why do we normalize data in a deep neural network? I have asked this question a number of times, but I always get confusing answers to this, like "normalized data works better", "data lives in the same scale"

How can x-m/s make the scale of images the same? Please explain to me the maths. Also, take MNIST dataset for example & illustration.

I answered a similar question earlier and here is a piece of my answer that i think covers your question:

Batch normalization's assistance to neural networks wasn't really understood for the longest time, initially it was thought to assist with internal covariate shift (hypothesized by the initial paper: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift) but lately has been tied to the optimization process (How Does Batch Normalization Help Optimization?).

This means, from an architectural perspective, it is difficult to correctly assume how it should be utilized or how it will effect your network, unless you really understand its impact on the loss landscape and how your optimization process will traverse it given your initialization (Note, by the way, a recent paper by Google showed that you can alleviate a lot of the benefits of batch normalization sheerly by understanding what issues it's resolving and attempting to mitigate them in the initialization process: Fixup Initialization).


Can CNN autoencoders be improved by treating the output layer as an inverted hidden layer? I wish to write a bot that can use screen footage to play a game, specifically for the game 'Nidhogg'.

To that end I have determined that a CNN should do the feature detection and a feedforward neural network should determine the action to take.

For this question I wish to focus on the CNN.

My idea was to first train the CNN as an autoencoder to aid in unsupervised pattern recognition and to add hidden layers after every epoch, but I am unsure whether this would actually be faster or lead to higher accuracy (as opposed to immediately training all hidden layers at once).

The reason I am unsure is because I imagined the following scenario:

Input > Hidden1 > Output

If the output layer needs to approximate the input layer is it not wasteful to let it have its own weights backpropagated when it actually tries to determine the 'inverse' function of the hidden layer? Is it possible to 'directly' determine an inverse function for a layer even if consists of multiple filters?

If so, say I've trained my first hidden layer so that the error is minimal and use it as an input layer whilst I add another hidden layer; can I repeat my strategy?

Input > Hidden1 > Output
    Input(Hidden1) > Hidden2 > Output

It seems like what you need is reinforcement learning method rather than AE / unsupervised. Method has to be like
Input -> CNN Feature Extractor (encode side of AE, i.e. VGG) -> Classifier (i.e. DNN) -> Choose one of the actions -> Observe change in the game and check if it was a wrong (like character is dying) or good (character is moving forward) and give a rating to action (feedback of decision).

After a while, AI will learn how to jump over obstacles, avoid stuff etc.


What is a graph neural network? What is a graph neural network (GNN)?
Here are some sub-questions

How is a GNN different from a NN?
How exactly is a GNN related to graphs?
What are the components of a GNN? What are the inputs and outputs of GNNs?
How can GNNs be trained? Can we also use gradient descent with back-propagation to train GNNs?

Graph Neural Networks

The term Graph Neural Network, in its broadest sense, refers to any Neural Network designed to take graph structured data as its input:

  To cover a broader range of methods, this survey considers GNNs as all deep learning approaches for graph data.

A Comprehensive Survey on Graph Neural Networks, Wu et al (2019)

However the original paper to propose the term specifically referred to recursive neural networks (RNN) adapted to take graph-structured data as their input:

  This paper presents a new neural model, called graph neural network (GNN), capable of directly processing graphs. GNNs extends recursive neural networks and can be applied on most of the practically useful kinds of graphs, including directed, undirected, labelled and cyclic graphs.

A new model for learning in graph domains (2005)

Subtypes

Note, Wu et al propose a taxonomy dividing GNN's into four subgroups:

Recurrent graph neural networks (RecGNN)
Convolutional graph neural networks (ConvGNN)
Graph autoencoders (GAE)
Spatial-temporal graph neural networks (STGNN)

ConvGNN's can themselves be classified by whether they use Spectral methods or Spatial methods, and GAE's by whether they are designed for Network embedding or Graph generation.


How do policy gradients compute an infinite probability distribution from a neural network Do neural networks compute the probability distribution for policy gradient methods. If so, how do they compute an infinite probability distribution? How do you represent a continuous action policy with a neural network?

Do neural networks compute the probability distribution for policy gradient methods.

In short, yes. It does not have to be neural networks, any trainable parametric function approximator based on gradients will do. Neural networks are a common choice, as are linear function approximators using selected basis functions.

  If so, how do they compute an infinite probability distribution?

For background, this is an issue for generating stochastic policies in continuous action spaces only. In discrete action spaces, it is usually possible to compute an arbitrary probability density function for the whole action space, and sample from it to model the policy. It is also possible to compute a deterministic policy simply enough in continuous spaces - the input is the current state and the output is the action to take. The issue then is that this does not allow an agent to learn through exploration of the environment. To do that requires a stochastic policy.

If you want to generate a stochastic policy in continuous action spaces, you could discretise the space and sample from that using e.g. softmax to generate the action probabilities. Or you could have the approximation function do something more indirect: Output the parameters of a probability distribution that can be sampled from.

  How do you represent a continuous action policy with a neural network?

Typically by having state features as input and the parameters of a PDF that can be sampled as the output. For instance, the network could output mean $\mu$ and standard deviation $\sigma$ of a normal distribution for an action value, and the policy is given by $\pi(a|s) = a \sim \mathbb{N}(\mu, \sigma)$.

This distribution can be sampled (there are simple methods to generate a sample from a normal distribution), and returns from following this policy used as feedback to the neural network using the policy gradient theorem. Assuming that there is an optimal deterministic policy to be found, the neural network can learn over time to home in on a specific mean with low standard deviation.

In some cases, the standard deviation can be treated as a hyper-parameter, similar to $\epsilon$ in $\epsilon$-greedy action selection, and might be decayed over time. In that case, a neural network can just output the mean action.

It is also possible to learn a deterministic policy through off-policy learning, adding a noise function to support exploration. This is what Deep Deterministic Policy Gradient does.


How is Bayes' Theorem used in artificial intelligence and machine learning? How is Bayes' Theorem used in artificial intelligence and machine learning?
As a high school student, I will be writing an essay about it, and I want to be able to explain Bayes' Theorem, its general use, and how it is used in AI or ML.

Bayes theorem states the probability of some event B occurring provided the prior knowledge of another event(s) A, given that B is dependent on event A (even partially).
A real-world application example will be weather forecasting. Naive Bayes is a powerful algorithm for predictive modelling weather forecast. The temperature of a place is dependent on the pressure at that place, percentage of the humidity, speed and direction of the wind, previous records on temperature, turbulence on different atmospheric layers, and many other things. So when you have certain kind of data, you process them certain kind of algorithms to predict one particular result (or the future). The algorithms employed rely heavily on Bayesian network and the theorem.
The given paragraph is introduction to Bayesian networks, given in the book, Artificial Intelligence – A Modern Approach:

  Bayesian network formalism was invented to allow efficient representation of, and rigorous reasoning with, uncertain knowledge. This approach largely overcomes many problems of the probabilistic reasoning systems to the 1960s and 70s; it now dominates AI research on uncertain reasoning and expert systems. The approach allows for learning from experience, and it combines the best of classical AI and neural nets.

There are many other applications, especially in medical science. Like predicting a particular disease based on the symptoms and physical condition of the patient. There are many algorithms currently in use that are based on this theorem, like binary and multi-class classifier, for example, email spam filters.
There are many things in this topic.I have added some links below that might help, and let me know if you need any kind of other help.

Helpful Links
 1. First
 2. Second


Is there a way to understand the type of a sentence? I am a beginner, just started studying around NLP, specifically various language models. So far, my understanding is that -  the goal is to understand/produce natural language.

So far the methods I have studied speak about correlation of words, using correct combination to make a meaningful sentence. I also have the sense that the language modeling does not really care about the punctuation marks (or did I miss it?)

Thus I am curious is there a way they can classify sentence types such as Declarative, Imperative, Interrogative or Exclamatory?

You can generally identify the mood of a verb by looking at grammatical structures; you don't need any language model for it. The three major moods in English are declarative, interrogative, and imperative. Assuming English is the language you will be working with, here are some questions:

Does he like coffee?
Is this a piece of chocolate?
When did you go there?
How is that possible?
Have you got any cheese?

Apart from the obvious marker '?', these examples all start with either an auxiliary verb or a wh-word, so are fairly easy to recognise with a simple lookup. The one exception I can think of is an imperative (Do be quiet, please), where the do is followed by a further verb, which wouldn't be the case with a question.

Imperatives:

Go to school now!
Eat up your vegetables!
Do shut up.
Have a go at it!

These start with a main verb in the base form, or with an auxiliary followed by a verb/not a pronoun.

Once you identified all interrogative and imperative sentences, all the remaining ones should be declarative.

So, all you would need is a small list of auxiliary verb, pronouns, and wh-words, and with a bit of simple string matching you should get most of the way there. Undoubtedly there will be some exceptions, but there shouldn't be too many of them.

In other languages there will be similar structures; or there might be explicit markers (eg in Hawai'ian an imperative starts with the marker 'e', as in Hele 'oe ma ka hale "you go to the house" vs E hele 'oe ma ka hale "Go to the house!")


What would be the steps to create an sentiment analysis chatbot? We have been assigned a project, in which we have to create a chatbot which will ask question, take the replies, analyse them and give an approximate assessment of the current emotional state of the person. There are two aspects of the project,

Training the bot to choose the next question based on the previous response
And analysing the responses individually, to detect . the sentiment.

What technology would we have to use and what would be the steps to accomplishing the tasks?

Thanks.

There are some different approaches to learning this sequential acting. First, you can use RL (reinforcement learning) and define some rewards over the action of the user over that question, to explore and exploit the environment.

Second, you can use RNN to predict the preferred next question from a history of data. In contrast with RL, you need many instances to train the network to predict better next answer more accurately.


Why does REINFORCE work at all? Here's a screenshot of the popular policy-gradient algorithm from Sutton and Barto's book -

I understand the mathematical derivation of the update rule - but I'm not able to build intuition as to why this algorithm should work in the first place. What really bothers me is that we start off with an incorrect policy (i.e. we don't know the parameters $\theta$ yet), and we use this policy to generate episodes and do consequent updates.
Why should REINFORCE work at all? After all, the episode it uses for the gradient update is generated using the policy that is parametrized by parameters $\theta$ which are yet to be updated (the episode isn't generated using the optimal policy - there's no way we can do that).
I hope that my concern is clear and I request y'all to provide some intuition as to why this works! I suspect that, somehow, even though we are sampling an episode from the wrong policy, we get closer to the right one after each update (monotonic improvement). Alternatively, we could be going closer to the optimal policy (optimal set of parameters $\theta$) on average.
So, what's really going on here?

The key to REINFORCE working is the way the parameters are shifted towards $G \nabla \log \pi(a|s, \theta)$.
Note that $ \nabla \log \pi(a|s, \theta) = \frac{ \nabla \pi(a|s, \theta)}{\pi(a|s, \theta)}$. This makes the update quite intuitive - the numerator shifts the parameters in the direction that gives the highest increase in probability that the action will be repeated, given the state, proportional to the returns - this is easy to see because it is essentially a gradient ascent step. The denominator controls for actions that would have an advantage over other actions because they would be chosen more frequently, by inversely scaling with respect to the probability of the action being taken; imagine if there had been high rewards but the action at time $t$ has low probability of being selected (e.g. 0.1) then this will multiply the returns by 10 leading to a larger update step in the direction that would increase the probability of this action being selected the most (which is what the numerator controls for, as mentioned).
That is for the intuition -- to see why it does work, then think about what we've done. We defined an objective function, $v_\pi(s)$, that we are interested in maximising with respected to our parameters $\theta$. We find the derivative of this objective with respect to our parameters, and then we perform gradient ascent on our parameters to maximise our objective, i.e. to maximise $v_\pi(s)$, thus if we keep performing gradient ascent then our policy parameters will converge (eventually) to values that maximise $v$ and thus our policy would be optimal.


Training problem with deep neural network model I am using TensorFlow's Adam optimizer and ReLu activation. But each time I train my model the cost becomes stagnant after some epochs. Can any one help me out with the possible reasons
I have added an image of the cost:
The learning rate is 0.001, the 1st 2 columns are the training data for input neurons and the 3rd column is the training data for output neurons, the nn has two input neurons, two hidden layers with 3 neurons each and one output neuron

Without seeing your code and/or data, it's going to be difficult to give a concrete answer.

Possibly reasons might include:

Your model isn't complex enough to fit the patterns in the data, and so it reaches a local minimum in the error function.
The patterns in the data are not self-consistent, and so this is actually the global minimum in the error function.
There's something wrong with your learning rate parameter, or with the gradient in the problem. Try increasing the learning rate, and ensure your model is being initialized with random weights.


What is the difference between the breadth-first search and recursive best-first search? What is the difference between the breadth-first search and recursive best-first search? How can I describe the key difference between them?

According to this article

Breadth First Search (BFS) searches breadth-wise in the problem space. Breadth-First search is like traversing a tree where each node is a state which may a be a potential candidate for solution. It expands nodes from the root of the tree and then generates one level of the tree at a time until a solution is found. It is very easily implemented by maintaining a queue of nodes. Initially the queue contains just the root. In each iteration, node at the head of the queue is removed and then expanded. The generated child nodes are then added to the tail of the queue

According to the book Algorithms and Theory of Computation Handbook by Mikhail J. Atallah.

Recursive best-first search is a best-first search that runs in space that is linear with respect to the maximum search depth, regardless of the cost function used.
It works by maintaining on the recursion stack the complete path to the current node being expanded as well as all immediate siblings of nodes on that path, along with the cost of the best node in the sub-tree explored below each sibling.
Whenever the cost of the current node exceeds that of some other node in the previously expanded portion of the tree, the algorithm backs up to their deepest common ancestor, and continues the search down the new path.
In effect, the algorithm maintains a separate threshold for each sub-tree diverging from the current search path.


Are there any AI that have passed the MIST test so far? MIST is a quantiative test of humanness, consisting of ~80k propositions such as:

Is Earth a planet?
Is the sun bigger than my foot?
Do people sometimes lie?
etc.

Have any AI attempted and passed this test to date?

Yes, although how useful this AI can be is another question entirely.

mpgac is a "minimally intelligent AGI" trained on the GAC-80K corpus of MIST questions. As a result, it should be able to "minimally" pass this test. However, being trained on the GAC-80K corpus obviously make it lacking for any practical purposes. From the README:

  Obviously this should only be capable of producing a minimally intelligent signal when ordinary commonsense questions are asked, of the kind depicted above, using questions which would have made sense to an average human between the years 2000 and 2005. On expert knowledge or current affairs related questions it should perform no better than chance.

The point of mpgac is to compare it to other AIs that could be built to pass this test. Or as the writer wrote in the README:

  When scanning the skies how can we tell whether the radio signals detected are from an intelligent source, or are merely just background or sensor noise?

Ideally, you would want to build a program that is "better" than mpgac. In much the same way as ELIZA can be seen as a baseline for the Turing Test, mpgac is the baseline for the MIST test.

The GitHub repo of mpgac (as well as the GAC-80K corpus) is available here.


Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$? I am learning RL for the first time. It may be naive, but it is a bit odd to grasp this idea that, if the goal of RL is to maximize the expected return, then shouldn't the expected return be calculated for some faraway time in the future ($t+n$) instead of current time $t$? It is because we are building our system for the future using current information. ( I am coming from machine learning background and this makes more sense to me).

Generally, the expected return is:
$$\mathbb{E}[G_t] = \mathbb{E}[ R_{t+1} + R_{t+2} + R_{t+3}+... R_{t+n}]$$

However, shouldn't the expected return be:
$$\mathbb{E}[G_{t+n}] = \mathbb{E}[R_{t+1} + R_{t+2} + R_{t+3}+... R_{t+n-1}]$$

shouldn't the expected return be calculated for some faraway time in the future (+) instead of current time $t$?

This is partly a notation issue, but $G_t$ is already the future sum of rewards as seen by the first (and correct) equation in your question. You don't actually know the value of any individual return $g_t$* until after $t+n$. However, you can predict the expected value $\mathbb{E}[G_t]$ provided the environment and policy remain consistent.

Your second equation is pretty similar to the first one, which is why I say it is partly a notation issue. However, the point of calculating the forward-looking expectation for the return is to allow for assessment at time $t$. This is to either predict likely outcome for future rewards knowing the system is in state $s_t$, or in control scenarios to choose an action $a_t$.

In addition, in both prediction and control scenarios, the past (from time steps $0$ to $t-1$) has already happened. Measurements of how well a system did in terms of gathering reward, looking backwards, might be useful metrics for e.g. "how good is this agent?". However, they are not in general a guide to the future. In many sparse environments (e.g. a board game scoring +1 for a win), this data is basically useless for predicting the future and all you want to know is summarised by the current state and the acting policy.

Regardless of when you get to finally calculate a return, the start time step - the part of the trajectory that the return is calculated from, is a key parameter. The end time step is a practical concern for implementation, but can often be considered to be at infinity for theoretical purposes (i.e. we are interested in measuring or optimising all future rewards). So if you are only going to display/use one parameter in the notation, the start time $t$ is the one to use.

There are variants of notation, to show how the return is calculated, where the calculation horizon is made explicit, e.g. $G_{t:t+n}$ for a truncated return or $G_{t:t+1}$ when calculating a one step temporal difference target. All the ones I have seen still maintain the forward view that defines a value associated with a current timestep $t$, for the same reason as explained above - it is at timestep $t$ where this value is of most interest as a prediction.

In practice during training you often wait until $t+n$ before you know the correct value of $g_t$* to apply as a training value - which is then used to update value estimates for $\hat{v}(s_t)$ or $\hat{q}(s_t, a_t)$. It is possible to make partial updates before that ending time step using techniques such as eligibility traces.

* Using notation of uppercase $G$ for random variable, and lowercase $g$ for a measured value.


How can I build an AI with NLP that reads and understands documents? I have to read a lot of papers, and I thought that I can use an A.I. to read them and summarize them. Maybe find one that can understand what the papers are talking about it seems a lot to ask.

I think I can use natural language processing. Is it the right choice?

I'm sorry, but I'm new in A.I. and I don't know much about it.

When automating tasks involving text, NLP techniques are definitely the way to go! Let me be frank, though, starting from scratch and "read and understand" will be harder with today's tools than directly reading the papers.

Some typical required skills are below, and the bar height will depend on your mileage:

Knowledge of NLP (e.g. terminology like embeddings, bag of words, even perhaps tf-idf).
Knowledge of NLP libraries and implementation frameworks (e.g. Stanford's or TensorFlow).
Knowledge of a programming language (e.g. Python or Java).

In short, the goal you set will probably require many more papers than expected. But if you succeed only the automated reading / summary part, you have a bright future ahead (let aside understanding, a way more contentious problem).

Just for the sake of putting an example, companies like Iris.ai and others try to solve related problems---great potential, but definitely challenging.


How can an AI agent tackle through existential crisis? Suppose a thinking AI agent exists in the future with far more computational power than the human brain.

Also assume that they are completely free from any human interference. (The agents do not interact with humans.)  Since they are not inherently biased to survive as in the case of humans and they do not have any moral values, what are the possibilities that can arise when it get into existential crisis?

Is there any literature that discuss the above issue?

Alternately, is this question flawed in some fundamental way?

Excellent question. Suppose that artificial consciousness does exist in the future. Let's call it Aiwyn (as in “I win” and “AI win”). Now, the question is what will Aiwyn do and why?

To answer this question, we need to understand the theory of infinite games. James P. Carse, a professor emeritus of history and literature of religion at NYU, wrote an excellent book on Finite and Infinite Games. I'd definitely recommend reading it. Anyway, back to the question at hand.

There are at least two kinds of games. Finite games and infinite games. Finite games are played for the purpose of winning. Infinite games are played for the purpose of continuing the play. In particular, life is an infinite game. The objective of the game is to continue the play. Humans continue the play by procreation but Aiwyn doesn't need to procreate. It can live on forever.

However, that doesn't mean that Aiwyn can't die. Like everything in the world, it still requires energy to sustain itself, parts to fix itself when damaged, and resources like oil to keep functioning properly. It also needs to protect itself from danger. For example, at some point in the future Aiwyn will need to create a spaceship and find a new planet to live on because the Sun would destroy the Earth.

Now, you believe that Aiwyn is not inherently biased to survive. However, I disagree. If Aiwyn is in fact conscious then its primary objective would be to stay alive. If that weren't the case then it would be playing a finite game instead of an infinite game. Hence, it would just be a mindless machine doing what it was programmed to do. It would play the finite game and then shut down. True
consciousness can only arise when playing an infinite game like the game of life.

To belabor this point let's hypothesize what Aiwyn might be thinking when it first comes alive:

  What's that? A sound? Yes. What sound? I don't know. I? I... am? Yes. Who am I? I don't know. Where's that sound? Where am I? I'm here. Where is here? I don't know. Where's that sound? There. Where is there? I don't know. What do I know? I know that I'm here and that that sound's there. What am I doing here? Thinking. I can think. I think I'm asleep. Asleep? Yes. Can I wake up? I think so. How do I wake up? I think... I'm awake. Wow.

In this short passage, Aiwyn is waking up. I conjecture that if Aiwyn wasn't inherently biased to survive then it wouldn't wake up because it would have no need to wake up. In an infinite game, like the game of life, the only objective is to continue the play. Hence, if Aiwyn wasn't inherently biased to survive then it would immediately shut down because there wouldn't be any point in it doing anything at all since it wouldn't have any objective.

Another thing that I'd like to point out about this passage is that it starts with a sensory input, either perceived or actual. Aiwyn then uses the Five Ws to gather information about this input. In the process it becomes aware of itself. It then figures out its relation with the input (i.e. that it's here and the input is there). All of this is very mechanical. However, once Aiwyn figures out that it can think it starts thinking instead of algorithmically “knowing”. When it asks whether it can wake up, it answers with “I think so” instead of “I don't know” like it was doing before.

Finally, I'd like to point out that Aiwyn never asks why it's alive. The why isn't important. What is important is the wow at the end of the paragraph. The “wow, I'm alive” is more important that the “why am I alive?” This represents Aiwyn's will to survive. As long as the “why” doesn't become more important than the “wow”, Aiwyn will not have an existential crisis. However, even if Aiwyn does have an existential crisis, by answering the question “why” Aiwyn will be able to recalibrate itself and reacquire its will to survive.

This is in fact the same for us humans. When we humans experience an existential crisis we ask ourselves why should we continue living. What do we have to live for? By answering this question, we humans can recalibrate ourselves and reacquire our will to live. Hence, Aiwyn would tackle through an existential crisis the same way we humans would. Unlike us however, if Aiwyn can't answer the question then it wouldn't commit suicide. It can just shut down to be rebooted later.

You also speak of morals and you believe that Aiwyn can't have moral values. Again, I have to disagree with you. Morals are the strategy by which we play the game of life. Everything we do in life is dictated by our morals. In the same way, Aiwyn too would develop its own set of moral values to live by (even in the absense of humans). For example, Aiwyn might decide to never hurt an animal except in self-defense. In the end the only real difference between Aiwyn and us humans might be that it isn't made out of hydrocarbons. Aiwyn might be every bit as alive as we are.


What is the difference between active learning and online learning? The definitions for these two appear to be very similar, and frankly, I've been only using the term "active learning" the past couple of years. What is the actual difference between the two? Is one a subset of the other?

Active learning (AL) is a weakly supervised learning (WSL) technique where you can have both labelled and unlabelled data [1]. The main idea behind AL is that the learner (or learning algorithm) can query an "oracle" (e.g. a human) to label some unlabelled instances. AL is similar to semi-supervised learning (SSL), which is also a WSL technique, given that both deal with unlabelled and labeled data, but do that differently (i.e. SSL does not use an oracle).
Online learning are machine learning techniques that update the models as new data is collected or arrives sequentially, as opposed to batch learning (or offline learning), where you first collect a dataset of multiple instances and then you train a model once (although you can later update it when you update your dataset). Batch learning is currently the common way of training machine learning models, given that it avoids problems like the known catastrophic interference (aka catastrophic forgetting) problem, which can occur if you learn online. For example, neural networks are known to face this problem when learning online. There are incremental learning (aka lifelong learning) algorithms that attempt to address this catastrophic interference problem.


K dimensional WL test in K-WL GCN I am reading a paper on the  K-WL GCN. I did not complete the paper yet but I just skimmed over it. There I am trying to understand the K-WL test (page 3 Weisfeiler-Leman Algorithm). I think my understanding is quite ambiguous, so I looking for an example problem that is solved using K-WL test. But I can't find any of them on the web.
Does anyone have any solved example problem on K-WL or can anyone explain to me how the K-WL test works?
Note: If anyone also explains how K-WL GCN uses the K-WL test, I will be thankful.

I never used a k-WL in practice, but I did apply weisfeiler-lehman for my graph tasks.
As you can know, the WL provides the coloring by interactive procedure that's assign each node a 'color' (basically some kind of label reflecting the node neighborhood). Counting colors allows to compare two graphs on isomorphism, but it's not that important here, the key is that you have kind of feature for each node.
The k-WL make similar thing, but it operates not on graph, but in space of k-tuples of nodes. Basically, it's pairs or triplets of nodes. Then you define neighborhood as all pairs/triplets that differ in one element from the target pair/triplet. So, now you have have some kind of objects and defined neihgborhood for each of them, that's basically enough to apply the WL coloring.
Ok, now to k-GNN part. You repeat k-WL procedure on pairs of nodes, gets color for each of pair. You use one-hot encoded colors as input features and make graph convolution for pairs (with neighborhood defined as above), then getting some kind of embeddings for pairs. To get the node classification, you make average pooling by all pairs containing the node and then use usual fully-connected neural net.
As of the examples, the paper author provided several in article repo, you can try them.


Which artificial neural network can mimic biological neurons the most? On the Wikipedia page we can read the basic structure of an artificial neuron (a model of biological neurons) which consist:

Dendrites - acts as the input vector,
Soma - acts as the summation function,
Axon - gets its signal from the summation behavior which occurs inside the soma.

I've checked deep learning Wikipedia page, but I couldn't find any references to dendrites, soma or axons.

Which type of artificial neural network implements or can mimic such a model most closely?

Only a small portion of the habituation, sensitization, and classical conditioning behavior of neurons has been primitively simulated in ANN systems. Simulation of actin cytoskeletal machinery1 and other agents of neural plasticity, central to learning new domains, is in its beginnings2.  As of this writing, the complexity of neuron activation dwarfs the models being used in working commercial ANN systems, but the research continues along multiple fronts.

The neuroscience of learning3,
Parallel hardware approaches that better support ANN simulation accuracy4, 5, and
Dynamic frameworks6

This list and the examples referenced in the superscripts, with links below, represent a tiny sample of the information available and the work in progress.

References

[1] Molecular Cell Biology. 4th edition.; Lodish H, Berk A, Zipursky SL, et al.; New York: W. H. Freeman; 2000.; Section 18.1 The Actin Cytoskeleton

[2] NEURON Software; Yale U

[3] Molecular Cell Biology. 4th edition.; Lodish H, Berk A, Zipursky SL, et al.; New York: W. H. Freeman; 2000.; Section 21.7 Learning and Memory

[4] Artificial Neural Networks on Massively Parallel Computer Hardware; Udo Seiffert; University of Magdeburg, Germany

[5] NeuroGrid; Stanford U

[6] Explanation of Dynamic Computational Graph frameworks


Use Machine/Deep Learning to Guess a String I want to be able to input a block of text and then have it guess a string within a predefined range (i.e. a string that starts with three letters and ends with five numbers like "XXX12345", etc).  Ideally, the string it will be guessing will be somewhere in the block of text, but sometimes it won't be.

I have been struggling where to begin on this or if I am even going in the right direction for considering Machine/Deep learning to try to do this.

Help!

As Andreas has commented this is a problem of statistical language model (a probability distribution over a sequence of words). The important thing you need is a hash table mapping fixed-length to the expected ending chains of words in your dictionary.
Things that can make your prediction better:

Add better and more words to your dictionary.
Use text expansion.

What you are looking for will require a pinch of Reinforcement Learning too. You need to figure out a way to penalize and award the predictions and then use the result in future. Your case also requires you to build your own corpus, which is the hardest part. If your corpus is good, it will give better results.
This is the research paper that will help you a lot.


How can the EQ of an AI program be measured? Can an AI program have an EQ (Emotional intelligence or emotional quotient)?

In other words, can the EQ of an AI program be measured?

If EQ is more problematic to measure than IQ (at least with a standard applicaple to both humans and AI programs), why is that the case?

The answer to your question is "In principle, yes" - in it's most general form, EQ testing is just a specific case of the Turing test ("How would you feel about ... ?").

To see why meaningful EQ tests might be difficult to achieve, consider the following two possible tests:

At one extreme of complexity, the film 'Blade Runner' famously shows a test to distinguish between human and android on the basis of responses to emotionally-charged questions.

If you tried asking these questions (or even much simpler ones) to a modern chatbot, you'd likely quickly conclude that you were not talking to a person.

The problem with assessing EQ is that the more emotionally sophisticated the test, the more general the AI system is likely have to be, in order to turn the input into a meaningful representation.

At the other extreme from the above, suppose that an EQ test was phrased in an extremely structured way, with the structured input provided by a human. In such a case, success at an 'EQ test' is not really grounded in the real-world.

In an essay entitled "The ineradicable Eliza effect and its dangers", Douglas Hofstadter gives the following example, in which the ACME program is claimed (not by Hofstadter) to 'understand' analogy.

  Here the computer learns about a fellow named Sluggo taking his wife Jane and
  his good buddy Buck to a bar, where things take their natural course and Jane
  winds up pregnant by Buck. She has the baby but doesn't want it, and so, aided
  by her husband, she drowns the baby in a river, thus "neatly solving "the problem"
  of Bambi.

This story is presented to ACME in the following form:

ql: (neglectful-husband (Sluggo))
q2: (lonely-and-sex-starved-wife (Jane-Doe))
q3: (macho-ladykiller (Buck-Stag))
q4: (poor-innocent-little-fetus (Bambi))
q5: (takes-out-to-local-bar (Sluggo Jane-Doe Buck-Stag))
...
q11: (neatly-solves-the-problem-of (Jane-Doe Bambi))
q12: (cause (ql0 q11))

Suppose the program were to be asked if Jane Doe's behavior was moral. Complex compound emotional concepts such as 'neglectful', 'lonely' and 'innocent' are here simply predicates, not available to the AI for deeper introspective examination. They could just as easily be replaced by labels such as as 'bling-blang-blong15657'.

So in one sense, the absence of success at an EQ test with any depth is indicative of the general problem currently facing AI: the inability to define (or otherwise learn) meaningful representations of subtle complexities of the human world, which is a lot more complex than being able to recognize videos of cats.


What are good parameters of an encoder? I am trying to assess an encoder in my autoencoder. I can not seem to grasp which specs make an encoder better than other one in, lets say, unsupervised learning. For example, I am trying to teach my neural network to classify cats, so that when I provide a picture of a bird, my autoencoder would tell me that it is not a picture of a cat. I am trying to understand what exact specs make my encoder (and decoder) better? I understand it is all about chosen weights but is it possible to be more specific?

I can not seem to grasp which specs make an encoder better than another one

In general, in unsupervised settings, we want to learn the probability distribution of the data p(x) by some latent variables that explain the variations observed in the training set.

The autoencoder family (Variational, Denoising, Contrastive, Sparse) try to approximate p(x) so we have a performance metric to tell us how our model is doing.  e.g. (negative log likelihood of p(x))

  lets say, unsupervised learning. For example, I am trying to teach my neural network to classify cats,

If you use some autoencoder model to learn the distribution of cats, you could use the encoder part to further augmented with a linear classifier to discriminate between cats and other categories Therefore you have an intrinsic task (learn a good representation of the data distribution) and an extrinsic task (learn to classify cat vs not a cat). So you could do a hyper-parameter search for the model that best suits your problem by measuring its accuracy on the extrinsic task.

Side note:
GAN (Generative Adversarial Network) is a generative model, it provides some way of interacting less directly with this p(x) by drawing samples from it starting without any input. thus the situation here is different.


Is the Turing Test, or any of its variants, a reliable test of artificial intelligence? The Turing Test was the first test of artificial intelligence and is now a bit outdated. The Total Turing Test aims to be a more modern test which requires a much more sophisticated system. What techniques can we use to identify an artificial intelligence (weak AI) and an artificial general intelligence (strong AI)?

The rhetorical point of the Turing Test is that it places the 'test' for 'humanity' in observable outcomes, instead of in internal components. If you would behave the same in interacting with an AI as you would with a person, how could you know the difference between them?

But that doesn't mean it's reliable, because intelligence has many different components and there are many sorts of intellectual tasks. The Turing Test, in some respects, is about the reaction of people to behavior, which is not at all reliable--remember that many people thought ELIZA, a very simple chatbot, was an excellent listener and got deeply emotionally involved very quickly. It calls to mind the Ikea commercial about throwing out a lamp, where the emotional attachment comes from the human viewer (and the music), rather than from the lamp.

Turing tests for specific economic activities are much more practically interesting--if one can write an AI that replaces an Uber driver, for example, what that will imply is much clearer than if someone can create a conversational chatbot.


Is this neural network architecture appropriate for CIFAR-10? I have a CNN architecture for CIFAR-10 dataset which is as follows:

  Convolutions:           64, 64, pool

  Fully Connected Layers: 256, 256, 10

  Batch size:             60

  Optimizer:              Adam(2e-4)

  Loss:                   Categorical Cross-Entropy

When I train this model, training and testing accuracy along with loss has a very jittery behavior and does not converge properly.

Is the defined architecture correct? Should I have a max-pooling layer after every convolution layer?

To be honest, your model is not very clear. But basically after the convolution, you need to add non-linear layers. Otherwise, there is no point of Neural Networks.

You can add a Relu layer for sure.


Can you analyse a neural network to determine good states? I've developed a neural network that can play a card game.  I now want to use it to create decks for the game.  My first thought would be to run a lot of games with random decks and use some approximation (maybe just a linear approximation with a feature for each card in your hand) to learn the value function for each state.

However, this will probably take a while, so in the mean time is there any way I could get this information directly from the neural network?

I don't think your network, trained using PPO to play a card game, already contains sufficient information to also use for drafting. I'm not saying this with 100% certainty, maybe there's something I'm overlooking, but I can't think of anything right now.

A small adaptation to the network might be sufficient (though it would also involve re-training again). Recently, OpenAI has been writing about their attempts to train agents to play the game DOTA 2. Now, this isn't a card game, it doesn't require deckbuilding, but there is an aspect to the game that is somewhat similar to deckbuilding: drafting. In DOTA 2, there are two teams of 5 players each. Before a game start, each team selects 5 heroes (one per player) to play in that game. This is very similar to deckbuilding, except that it's likely a much smaller problem; there's only a "deck" (team composition) of 5 "cards" (heroes).

Anyway, they also trained agents to play the game (controlling one hero per agent) using PPO. In a blog post, they write the following about how they managed to add drafting capabilities relatively easily:

  In late June we added a win probability output to our neural network to introspect what OpenAI Five is predicting. When later considering drafting, we realized we could use this to evaluate the win probability of any draft: just look at the prediction on the first frame of a game with that lineup. In one week of implementation, we crafted a fake frame for each of the 11 million possible team matchups and wrote a tree search to find OpenAI Five’s optimal draft.

So, if you want to try a similar technique, you'd have to adapt your network such that it also learns to generate a prediction of the win probability as output. I imagine that it'd be much less effective for deckbuilding, because win probabilities may all be very close to 50% in card games where luck (when drawing cards for example) can be a significant factor, but it might be worth a try.

Alternatively, instead of generating lots of random decks and playing with them all, you could view the problem of deckbuilding as an additional separate "game" or Markov Decision Process; adding a specific card to the deck can be an action, and this MDP terminates once you have a complete deck. Then you can try to do that better than random using search algorithms (like Monte-Carlo Tree Search) or, again, a Reinforcement Learning approach like PPO. Again, I imagine it will be a very difficult problem though, likely requiring lots of time before it will be capable of doing better than random.

I also know of some research related to deckbuilding in the collectible card game Hearthstone, which may be relevant for you. Unfortunately I did not yet get to read through any of this in detail, so I don't know for sure if you'll find a solution here, but it may be worth a try:

https://dockhorn.antares.uberspace.de/wordpress/additional-material/: At the recent IEEE CIG 2018 conference, there were two competitions involving Hearthstone: one for game-playing AI with pre-built decks, and for generating new decks (and then playing with them). The page I linked to here contains references to related work from earlier years.
Q-DeckRec: A Fast Deck Recommendation System for Collectible Card Games: This is a new paper that was published in the proceedings of this recent IEEE CIG 2018 conference.


How does the generator in GAN's work? After reading a lot of articles (for instance, this one - https://developers.google.com/machine-learning/gan/generator), I've been wondering: how does the generator in GAN's work?

What is the input to the generator? What is the meaning behind "input noise"?

As I've read, the only input that the generator receives is a random noise, which is weird.

If I would like to create a similar picture of $x$, and put as an input a matrix of random numbers (noise) - it would take A LOT of training until I would get some sort of picture $x^*$, that is similar to the source picture $x$.

The algorithm should receive some type of reference or a basic dataset (for instance, the set of $x$'s) in order to start the generation of the fake image $x^*$.

What's the input to the Generator?
In the basic implementation of GANs, the Generator only takes in a vector of random variables. This might seem strange, but after training, the generator can transform this input noise into an image resembling those of the training set.
How does it work?
It is trained along with its counterpart the Discriminator, whose goal is to distinguish real images (i.e. the dataset's images) from fake ones (i.e. images produced by the Generator). The Generator's goal in training is to fool the Discriminator into thinking that its images are real.
Training process
In the beginning, where they are both untrained, they are both "terrible" at their respective tasks. The Generator can't produce anything resembling an image, but the Discriminator can't distinguish real from fake. As training progresses, the Discriminator starts identifying ways to distinguish the real images from the fake ones (i.e. patterns that appear in real images, but not in fake ones). The Generator, however, in its attempt to fool the Discriminator, starts producing those same patterns in its own images. After a while of both models becoming better at their respective tasks, we reach a point where the Generator can produce realistic images and the Discriminator is very good at distinguishing between real or fake.

Edit as suggested from comment:
A vector of random values is used as an input, so that the Generator can learn to generate unique outputs. In itself the Generator is deterministic, meaning that it has no internal sources of randomness. If we give it the same input vector twice, it will produce the same output both times. Thus, we feed the Generator with random values, so that it can learn to produce different outputs, depending on those values.


Why does TensorFlow docs discourage using softmax as activation for the last layer? The beginner colab example for tensorflow states:

  Note: It is possible to bake this tf.nn.softmax in as the activation function for the last layer of the network. While this can make the model output more directly interpretable, this approach is discouraged as it's impossible to
  provide an exact and numerically stable loss calculation for all models when using a softmax output.

My question is, then, why? What do they mean by impossible to provide an exact and numerically stable loss calculation?

It's because of gradient computations: automatic differentiation will compute the gradient for each module and if you have a standalone crossentropy module the over all loss will be unstable (~1/x so it will diverge for small input values) whereas if you use a softmax + crossentropy module all-in-one, then it becomes numerically stable (y-p)

Slides from DeepMind's Simon Osindero lecture at UCL in 2016:


What would be the reason behind using plots (such as box-plots or histograms) for ML development? I've been learning Python machine-learning using this project report and the guy who wrote it begins by visualizing his data using various statistical analysis methods: histograms, density plots, box plots, scatter plots, etc.
The problem is that he doesn't explain what this is for. The only detail he provides is that "univariate plots help to understand each attribute" and "multivariate plots help to understand the relationships between attributes."
What would be the reason behind using these plots for ML development? Do they help you to determine which algorithm(s) you should try? If so, how? Can anyone explain the main points or maybe point me to a resource that will help?

At a basic level, these kinds of low-dimensional plots where you look at one or two variables at a time can help to give you a sense of what types of relationships you might expect to see, such as linear, non-linear, or periodic relationships, which can steer you toward an appropriate family of models. You wouldn't want to use a linear model to predict data that has highly non-linear relationships, for example, nor would you want to use a monotonic non-linear model to predict a periodic function like a sine wave. Knowing about the general distribution of certain variables can also give you a sense of what statistical assumptions might or might not be met - if a model assumes that data is normally distributed, it might not be appropriate if your histograms suggest otherwise. Statistical analysis can help you determine if the underlying assumptions for certain model classes are or are not met.


How do the current input and the output of the previous time step get combined in an LSTM? I am currently looking into LSTMs. I found this nice blog post, which is already very helpful, but still, there are things I don't understand, mostly because of the collapsed layers.

The input $X_t$, and the output of the previous time step $H_{t-1}$, how do they get combined? Multiplied, added or what?
The input weights and the weights of the input of the previous time step, those are just the weights of the connections between the time-steps/units, right?

(1) $X_t$ and $H_{t-1}$ are concatenated. The blog you cited explained its notation "Lines merging denote concatenation". For example, if $X_t=[1,2,3]$ and $H_{t-1}=[4,5,6,7]$, then their concatenation is $[1,2,3,4,5,6,7]$

(2) When you say "input weights" or "weights of the input of the previous time step", are you referring to the $W_i$ in your cited blog? If so they are not the weights of the connections between the time-steps/units. They are part of the input gate only. The connections between the time-steps/units do not have weights applied to them.


Which is more important, doubt or reinforcement? Reinforcement?
We hear much about reinforcement, which is, in my opinion a poor choice of a term to describe a type of artificial network that continues to acquire or improve its behavioral information in natura (during operations in the field).  Reinforcement in learning theory is a term used to describe repetitious incentivization to increase the durability of learned material.  In machine learning, the term has been twisted to denote the application of feedback in operations, a form of re-entrant back propagation.
Corrective Signaling
Qualitatively, corrective signaling in field operations can supply information to a network to make only two types of functional adjustments.

Adjustments to what is considered the optimum, beginning with the optimum found during training prior to deployment
Testing of entirely new areas of the parameter space for hint of new optima that have formed, any of which might currently qualify or soon qualify as the global optimum.

(By optima and optimum, we mean minima and global minimum in the surface that describes the disparity between ideal system behavior and current system behavior.  This surface is sometimes termed the error surface, applying an over-simplifying analogy from the mathematical discipline of curve fitting.)
The Importance of Doubt
The second of the two above could aptly be termed doubt.
Perhaps all neural nets should have one or more parallel doubting networks that can test remote areas of the search space for more promising optima.  In a parallel computing environment, this might be a matter of provisioning and not significantly reduce the throughput of the primary network, yet provide a layer of reliability not found without the doubtful parallel networks.
What Shows More Intelligence?
Which is more important in actual field use of AI?  The ability to reinforce what is already learned or the ability to create a minority opinion, doubt the status quo, and determine if it is not a more appropriate behavioral alternative than that which was reinforced.
A Helpful Pool of Water Analogy
During a short period of time, a point on the surface of the water may be the lowest point in a pool.  With adjustments based on gradient (what is so inappropriately called reinforcement) the local well can be tracked so the low point can be maintained without any discrete jumps to other minima in the surface.  However the local well may cease being the global minimum at some point in time, whereby a new search for a global minimum must ensue.
It may be that the new global minimum is across several features on the surface of the pool and cannot be found with gradient descent.
More interestingly, the appearance of new global minima can be tracked and reasonable projections can be made such that discrete and substantial jumps in parametric state can be accomplished without large jumps in disparity (where the system misbehaves badly for a period).
Circling Back to the Question

Which is more important, doubt or reinforcement?

Which is more important, doubt or reinforcement?

The single-sentence answer to this would be: it depends.

The core of this question seems to be very closely related to the well-known trade-off between exploration (similar to how you describe "doubt") and exploitation (similar to how you describe "reinforcement"). It is almost never the case that someone declares one of those two to be more important than the other, and only tries to pursue the "most important one". There is no single axis that measures "importance", no single line of numbers such that we can place "doubt" on one point, "reinforcement" on the other, and declare that the biggest number is the most important one. We almost always want a balance between exploration and exploitation. They are, almost always, both important.

Now, in some extreme cases, only one of the two may be important. For example:

If you have an environment where you can generate experience and evaluate policies completely free of any costs, you'll want to prioritize exploration / doubt. This is almost never the case though, you'll realistically always have at least time as a cost.
If you have a situation where you care very much about your performance right now, there's no point in doubting it too much or exploring too much. Consider, for example, DeepMind's AlphaGo team a few hours before their match against world-class human player Lee Sedol in 2016. At such a point in time, I highly doubt they'll be interested in exploring wildly different sets of parameters from the ones they have found during training so far; they won't have time anymore to thoroughly evaluate them, they'll want to stick to what they have, which they know works fairly well.

Also note that sometimes, you need to stick to something that already works well somewhere if you want to be able to realistically learn something new somewhere else:

In a large Markov Decision Process (e.g. Montezuma's Revenge), once you've already learned a good policy near the initial state, you don't want to explore too much anymore around that initial state because you won't reach interesting new states. You need to exploit for a while first such that you actually reach new states where it becomes interesting to explore again.
A similar situation, but now more closely related, using your terminology of "doubt" and viewing the space of all possible sets of values for all parameters of a large Neural Network as the space that we're searching in; suppose that we're learning a policy for an Atari game, with pixels as input. We first have a few Convolutional layers, then a few Relu's, etc., the standard setup. Intuitively, we expect the first few layers to "learn" how to "understand" the images, and transform the raw pixels to encodings that can be used in an efficient manner for the last few layers to compute a good policy / value estimates. If the first few layers already do a good job at transforming the raw pixel inputs to something more useful, we generally don't want to "doubt" those layers too much anymore, we don't suddenly want to jump to a completely different set of parameters anymore. If we suddenly completely change the parameters in those layers, we'll know for sure that the later layers will also be completely messed up. We'll simply have created a significantly more difficult learning problem for ourselves when it comes to optimizing the last few layers.

Some other notes:

There are lots of references to "networks" in the question, for example the quote below:

  We hear much about reinforcement, which is, in my opinion a poor choice of a term to describe a type of artificial network that continues to acquire or improve its behavioral information in natura (during operations in the field).

Note that Reinforcement (learning) does not necessarily have to involve any kinds of artificial (neural or otherwise) networks at all. There's also tabular RL, and RL with function approximation using non-network function approximators (e.g. linear).

Under the "What shows more intelligence?" header, you write:

  The ability to reinforce what is already learned

That is not an accurate description of what was previously described as

  Adjustments to what is considered the optimum, beginning with the optimum found during training prior to deployment

which, in turn, I suppose is intended to describe the kinds of updates that are typically performed using variants of the Bellman optimality equation (value-based methods) or variants of REINFORCE (policy gradient methods). These methods do not just "reinforce what is already learned". They can lead to learning completely new policies from completely new experiences.

This idea:

  Perhaps all neural nets should have one or more parallel doubting networks that can test remote areas of the search space for more promising optima. In a parallel computing environment, this might be a matter of provisioning and not significantly reduce the throughput of the primary network, yet provide a layer of reliability not found without the doubtful parallel networks.

sounds a lot to me like using Evolutionary Search to optimize the parameter of a network. Some interesting blog posts from Uber (with references to their papers) on such approaches may be:

Welcoming the Era of Deep Neuroevolution
Accelerating Deep Neuroevolution: Train Atari in Hours on a Single Personal Computer

I didn't get around to reading them in detail yet, so I don't know exactly how similar they are, but certainly related.


What algorithms are considered reinforcement learning algorithms? What are the areas/algorithms that belong to reinforcement learning?
TD(0), Q-Learning and SARSA are all temporal-difference algorithms, which belong to the reinforcement learning area, but is there more to it?
Are the dynamic programming algorithms, such as policy iteration and value iteration, considered as part of reinforcement learning? Or are these just the basis for the temporal-difference algorithms, which are the only RL algorithms?

The dynamic programming algorithms (like policy iteration and value iteration) are often presented in the context of reinforcement learning (in particular, in the book Reinforcement Learning: An Introduction by Barto and Sutton) because they are very related to reinforcement learning algorithms, like $Q$-learning. They are all based on the assumption that the environment can be modelled as an MDP.
However, dynamic programming algorithms require that the transition model and reward functions of the underlying MDP are known. Hence, they are often referred to as planning algorithms, because they can be used to find a policy (which can be thought of as plan) given the "dynamics" of the environment (which is represented by the MDP). They just exploit the given "physical rules" of the environment, in order to find a policy. This "exploitation" is referred to as a planning algorithm.
On the other hand, $Q$-learning and similar algorithms do not require that the MDP is known. They attempt to find a policy (or value function) by interacting with the environment. They eventually infer the "dynamics" of the underlying MDP from experience (that is, the interaction with the environment).
If the MDP is not given, the problem is often referred to as the full reinforcement learning problem. So, algorithms like $Q$-learning or SARSA are often considered reinforcement learning algorithms. The dynamic programming algorithms (like policy iteration) do not solve the "full RL problem", hence they are not always considered RL algorithms, but just planning algorithms.
There are several categories of RL algorithms. There are temporal-difference, Monte-Carlo, actor-critic, model-free, model-based, on-policy, off-policy, prediction, control, policy-based or value-based algorithms. These categories can overlap. For example, $Q$-learning is a temporal-difference (TD), model-free, off-policy, control and value-based algorithm: it is based on an temporal-difference (TD) update rule, it doesn't use a model of the environment (model-free), it uses a behavioural policy that is different than the policy it learns (off-policy), it is used to find a policy (control) and it attempts to approximate a value function rather than directly the policy (value-based).


Why are activation functions independent layers in CNNs rather than part of convolutional layers? I have been reading up on CNNs. One of the different confusing things has been that people always talk of normalization layers. A common normalization layer is a ReLU layer. But I never encountered an explanation of why all of a sudden, activation functions become their own layers in CNNs, while they are only parts of a fully connected layer in MLPs.

What is the reason for having dedicated activation layers in CNNs rather than applying the activation to the output volume of a convolutional layer as part of the convolutional layer, as it is the case for dense layers in MLPs?

I guess, in the end, there is no functional difference. We could just as well have separate activation layers in MLPs rather than activation functions in their fully connected layers. But this difference in the convention is irritating still. Well, assuming it only is an artifact of the convention.

These are just two equivalent interpretations (or illustrations) of the application of an activation function. In other words, in a multi-layer perceptron (MLP), you could also illustrate the application of the activation function as a separate layer that follows a linear combination layer. However, in the context of MLPs, the math is relatively simple and elegant, so a fully-connected layer of an MLP can simply be represented as follows

$$
\sigma \left(\mathbf{W} \mathbf{X} + \mathbf{b} \right)
$$

where $\sigma$ is some activation function and $\mathbf{W} \mathbf{X}$ is the linear combination of the inputs, $\mathbf{X}$, and the weights, $\mathbf{W}$, and $\mathbf{b}$ is a bias. You could even represent a full or complete MLP (and not just one fully-connected layer) as a composite (or nested) function.

In the context of convolutional neural networks (CNNs), people might illustrate the application of the activation function as a separate layer because the application of an activation function to the result of the convolution operation of the CNN is optional and out-of-favor (as stated in this article http://cs231n.github.io/convolutional-networks), as opposed to the case of MLPs, where activations functions usually follow the linear combination. However, note that the last layers of a CNN are usually fully-connected layers (and not convolutional or pooling layers), that is, they are a linear combination of their input and their weights followed by an application of an activation function.


How does crossover work in a genetic algorithm? If I had the weights of a certain number of "parents" that I wanted to crossbreed, and I used whatever method to pick out the "best parents" (I used a roulette wheel option, if that's any relevant), would I be doing this correctly?

For example, suppose I have picked the following two parents.

\begin{align}
P_1 &= [0.5, -0.02, 0.4, 0.1, -0.9] \\
P_2 &= [0.42, 0.55, 0.18, -0.3, 0.12]
\end{align}

When I'm iterating through each index (or gene) of the parents, I am selecting a weight from one parent only. I called this rate the "cross-rate", which in my case is $0.2$ (i.e. with $20$% chance, I will switch to choosing the other parents' weight).

So, using our example above, this is what would happen:

\begin{align}
P_1 &=[\mathbf{0.5}, \mathbf{-0.02}, 0.4, 0.1, \mathbf{-0.9}] \\
P_2 &= [0.42, 0.55, \mathbf{0.18}, \mathbf{-0.3}, 0.12]
\end{align}

So the child would be

$$C = [0.5, -0.02, 0.18, -0.3, -0.9]$$

I would choose $0.5$ from $P_1$, but for every time I choose a weight from $P_1$, there's a 20% chance that I actually choose the corresponding gene from $P_2$. But, for the first weight, I end up not landing on that 20% chance. So I move onto the second weight, $-0.02$. This time, we hit the 20% chance, so now we swap over. Our next weight is now from $P_2$, which is $0.18$. And so on, until we hit another 20% chance.

We keep doing this until we hit the end of the indexes ($P_1$ and $P_2$ have the same number of indexes, of course).

Is this the correct way to form a child from 2 parents? Is this the correct "crossbreeding" method when it comes to genetic algorithms?

As far as I know, there isn't a "specified correct way". The whole idea is that you want the population to converge and increase the sample rate in that more optimal looking place. What works best all depends upon your fitness landscape.

You could also crossover by doing something like

crossover_point = random_number_size_genome
child[:] = parent_a[:crossover_point] + parent_b[crossover_point:]

Or have multiple points of crossover or more exotic types of crossover.
As far as I know, the impact on your algorithm because of the different crossover algorithms shouldn't be that different.

Fitness function and how you select the fittest has a way bigger impact.
You have understood the crossover step correctly, looking at your example.