What is the purpose of hidden nodes in neural network? If I have a set of sensory nodes taking in information and a set of "action nodes" which determine the behavior of my robot, why do I need hidden nodes between them when I can let all sensory nodes affect all action nodes?

(This is in the context of evolving neural network)

A feed forward neural network without hidden nodes can only find linear decision boundaries. However, most of the time you need non-linear decision boundaries. Hence you need hidden nodes with a non-linear activation function. The more hidden nodes you have, the more data you need to find good parameters, but the more complex decision boundaries you can find.


Training a CNN from scratch over COCO dataset I am using Tensorflow Object Detection API for training a CNN from scratch on COCO dataset. I need to use this specific configuration.
There is no pre-trained model on COCO with that configuration and this is the reason why I am training from scratch.

However, after 1 week of training and evaluating each checkpoint generated by the training phase this is how my learning phase appears on Tensorboard:

Thus, my questions are:

does anyone know how many iterations approximately will be necessary? Right now I did more than 500'000 iterations.
How can be possible that after 500'000 the evaluation is 0,8%? I would expected something like 60-70%.
Why does there is a sudden drop after 500k iterations? I thought that the eval was supposed to converge to some limit. (this is what SGD should do)
Is there any 'trick' to speed up the training phase? (ex: increasing the learning rate, etc).

It's hard to know for sure what's gone wrong, but here are some possibilities:

The problem is difficult. The COCO  paper reports that a typical object covers just 4-6% of the image. A randomly initialized model is therefore likely to do extremely poorly, with an expected precision of between 4 and 6% for detecting the object of interest in a frame. You also have 90 classes in your configuration file. It's not clear whether the model has access to the correct label, but if it's also inferring the class, we'd expect initial accuracy somewhere around 0.06%. That's actually about the precision of your starting model.
You're training on mini-batches of size 32. It's not clear to me from your config whether the lower axis of that graph is iterations or epochs. If it's the former, then your model will have seen only 1.5 million examples during the entire training period. For a problem this hard, that's not nearly enough. Indeed, COCO contains only 328k unique examples, so the model would have seen each of them just 5 times (corresponding to 5 epochs). If you've done 500,000 epochs though, then that ought to be enough.
It's possible that the hyper-parameters are not well set. I'm not an expert at training CNNs, but deep networks are notoriously finicky. I have difficulty reading your config file, but your learning rate looks reasonable to me. It also appears to undergo exponential decay over time though, and it's not clear to me that you want to be doing that, or whether the schedule you're using for the decay makes sense. This might be worth reviewing.


When to expand and when to simulate in Monte Carlo Tree Search? In Monte Carlo Tree Search (MCTS), we start at root node $R$. Then we select some leaf node $L$. And we expand $L$ by one or more child nodes and simulate from the child to the end of the game.

When should we expand and when should we simulate in MCTS? Why not expand 2 or 3 levels, and simulate from there, then back-propagate the values to the top?  Should we just expand 1 level? Why not more?

The most common strategy is to simply expand exactly one node per iteration; you can view this as expanding the first node of the Play-Out phase ("simulation" in your image), and not expanding any other nodes of the Play-Out phase. This is also what's done in your image.

That is the most common and probably most simple strategy, but it's certainly not the only one. It is pretty much the minimum you have to expand, but you can expand more if you like. The most common approach of only expanding one node per iteration is basically there because it minimizes the risk of running out of memory; when you only add one new node to the tree per iteration, the tree grows relatively slowly, so you'd have to keep the algorithm running for a really long time before you run out of memory.

If you're not afraid of running out of memory, you can choose to expand as many nodes as you like. For example, my General Video Game AI agent expands every single node encountered during the play-out at once, because in that particular domain we only get to run relatively few iterations anyway so I'm not afraid of running out of memory, even if I do expand lots of nodes.

The backpropagation step can only store results (information) from your iterations in nodes that actually exist, nodes that have been expanded. So, the benefit of expanding more (if possible) is that you'll retain more information from early iterations, your backpropagation step immediately gets to store results in all nodes along the path instead of only the top few nodes. Generally, this is a relatively minor benefit; nodes deep in the tree / deep inside the simulations are quite unlikely to ever get visited more than once anyway, and then this doesn't matter. But expanding more can, in theory, result in slightly more accurate value estimates in nodes closer to the root.


Why don't those developing AI Deepfake detectors use two detectors at once so as to catch deepfakes in one or the other? Why don't those developing AI Deepfake detectors use two differently trained detectors at once that way if the Deepfake was trained to fool one of the detectors the other would catch it and vice-versa?
To be clear this is really a question of can deepfakes be made to fool multiple high-accuracy detectors at the same time. And if so then how many can they fool before they become human detectable from noticeable noise?
I've heard of papers where they injected a certain noise into their deepfake videos which allows them to fool a given detector (https://arxiv.org/abs/2009.09213, https://delaat.net/rp/2019-2020/p74/report.pdf), so I thought well if they simply used two high-accuracy detectors then any pattern of noise used to fool one detector would interfere with the pattern of noise used to fool the other detector.

Because it is possible to fool many different models at once.
See table 2 in this paper, for an example using adversarial perturbations: https://arxiv.org/pdf/1610.08401.pdf
That being said, there is no reason to think that using two detectors at once will not increase chance to detect deepfakes. It will just not resolve the problem completely.


Where do 'random seeds' get used in deep neural networks? I know that when creating neural networks it's standard practice to create a 'random seed' so that you can get producible results in your models. I have a couple of questions regarding this:

Is the seed just something that is used in the 'learning' phase of the network or does it get saved? i.e. is it saved into the model itself and used by others if they decide to implement a model you created?
Does it matter what you choose to be the seed? Should the number have a certain length?
At what step of the creation of a model does this seed get used and how does it get used?

Other information about 'random seeds' would be welcomed! But these are my general questions.

I suppose the most common part where it will be used is in the initialization of weights before training; the best ways currently known to do that involve randomness.

If you use Dropout during training (randomly setting some activation levels to zero to combat overfitting), that also involves randomness, so your seed could also have influence there. Dropout should not be used anymore after training, although it could if you accidentally implement it to be used there. If you don't make that mistake though, your seed here should also only matter during training.

Depending on implementation, I suppose the seed could also have influence on random ordering of input data between epochs / random selection of minibatches from the training dataset during training. Of course, this is very much implementation-dependent. If you end up implement that kind of data processing yourself and don't do it through some framework, you'll also be the one who determines what random seed has influence on that process.

In general (barring any special cases that I'm unaware of), a Neural Network should behave deterministically after training; if you give it the same input, it should provide the same output, your random seed should no longer have influence after training.


How to detect patterns in salary distribution if we are suspecting malicious distribution based on employee's region? We are having suspects in salary distribution in our organisation due to employee's region. The data we have is as the following:

Name
Region
Work Position (4 main positions)
Salary
Gender

What technique should we use in Machine learning to check and detect malicious salary distribution? By using clustering?

A simple initial approach would be to separate it by position and check for each:

Use linear regression: $\hat{salary} = \sum_i \alpha_i * \hat{region}_i + \sum_k \beta_k * \mathbf{1}[\hat{gender}=k]$ and now you have an intuitive measure by looking at $\alpha$'s and $\beta$'s.

2 issues that may arise with this method:

This assumes though that a linear model is a good fit, which it very well might not be
The amount of data points may be small and this will give you point estimates without any sense of confidence.

2 possible respective solutions:

Use a stronger model with less intuitive parameters (like a neural network) and then use common visualization techniques to determine weight of each input (this could be as simple as changing them and looking at the outcome or something more complex that utilizes the gradient), but even more complex models can still suffer from the second problem, which is why id reccommend option 2
Use a Bayesian model, you can make something hierarchical something as simple as doing the above linear regression with a non informative prior and eventually you'd be able to build confidence intervals and get a better sense of the associations (note if youre looking for what to write this in, I recommend JAGS for its ease of use)


How do I update the Q values of a Deep Q Network when exploring? I am trying to implement a Deep Q Network to play Asteroids. Unfortunately, I am not sure how to calculate the Q value exactly, if I am exploring. For example, the agent is exploring for 1 second (otherwise makes no sense; I cannot let it just explore one step). Unfortunately, it makes a mistake at 0.99s, and the reward collapses.

At the moment, I am using the following formula to evaluate or update the Q value:

$$Q_{new,t} = reward + \gamma Q_{max,t+1}$$

But how do I know the max Q value of the next step? I could consider the best Q value the network says, but this is not necessarily true.

You can see the current implementation at the following URL:
https://github.com/SuchtyTV/RLearningBird/blob/master/src/main/java/rlgame/Brain.java.

For tabular Q-learning, the q-values for state s and action a are updated according to

$$
Q(s, a) \gets Q(s, a) + \alpha [(r + max_{a'} Q(s', a')) - Q(s,a)]
$$

where $\alpha$ is the learning rate and $(r + max_{a'} Q(s', a')) - Q(s,a)$ is the difference between the current estimate of the q-value, $Q(s,a)$, and the target, $r + max_{a'} Q(s', a')$.

The target q-value is based on the greedy policy, not the exploratory policy. Q-learning is theoretically guaranteed to converge to the optimal policy for any behavior policy (like $\epsilon$-greedy) that is guaranteed to visit every state and action pair an infinite number of times.  See Section 6.5 of the Sutton and Barto book for more details.

In contrast to Q-learning, the target q-value for SARSA is $r + Q(s', a')$, where $a'$ is chosen from an exploratory behavior policy like $\epsilon$-greedy. For SARSA the learned q-values are dependent on the behavior policy and therefore not guaranteed to converge to the optimal policy. A behavior policy that intentionally acted randomly for multiple consecutive actions, as in your example Asteroids exploratory policy, would likely lead to learning different q-values than would be learned for an $\epsilon$-greedy behavior policy.

Unfortunately Q-learning's theoretical guarantees of convergence to an optimal policy go out the window when nonlinear function approximation is introduced, as is the case for deep neural networks. Nevertheless, in the Deep Q-Networks paper, the q-value function is updated using a target value based on the maximum q-value for the next state. Specifically, if $Q(s, a, w)$ is a q-value function parameterized by weights $w$, then the weights are updated by

$$
w \gets w + \alpha [(r + max_{a'} Q(s', a', w^-)) - Q(s, a, w)] \nabla_w Q(s,a,w)
$$

where $w^-$ are the parameters of the target network used to stabilize training. (See the paper for more details). This update rule is chosen to minimizes the loss function

$$
L(w) = E[(r + max_{a'} Q(s', a', w^-)) - Q(s, a, w)]^2
$$

For your own implementation, it may be helpful to see a code example of the Deep Q-Networks parameter updates.  A tensorflow implementation is available in the function build_train in the OpenAI Baselines DeepQ code.


How does noise affect generalization? Does increasing the noise in data help to improve the learning ability of a network? Does it make any difference or does it depend on the problem being solved? How is it affect the generalization process overall?

Noise in the data, to a reasonable amount, may help the network to generalize better. Sometimes, it has the opposite effect. It partly depends on the kind of noise ("true" vs. artificial).

The AI FAQ on ANN gives a good overview. Excerpt:

  Noise in the actual data is never a good thing, since it limits the accuracy of generalization that can be achieved no matter how extensive the training set is. On the other hand, injecting artificial noise (jitter) into the inputs during training is one of several ways to improve generalization for smooth functions when you have a small training set.

In some field, such as computer vision, it's common to increase the size of the training set by copying some samples and adding some noises or other transformation.


Why coupling coefficients in capsule neural networks can't be learned by back-propagation? The paper Dynamic Routing Between Capsules uses the algorithm called "Dynamic Routing Between Capsules" to determine the coupling coefficients between capsules.

Why it can't be done by backpropagation?

Very interesting paper.

We can see that they have effectively modeled probabilistically the fuzzy existence of an object with various geometric attributes as objects in a scene with a hook to the region of pixels that correspond to its possible existence.  I agree with the authors that this is a much more robust approach than CNN and LSTM and may compete well with the emerging attention based approaches.

Back propagation does seem to be involved, although that is not the focus of the paper.

Notice, "there is top-down feedback," and, "an appropriate parent in the layer above," which are hints to a hierarchical approach, and we can see this approach is not absent of its context in the overall feeding forward of input signalling to output, all of which must be trained as a whole.  But just as with much older control systems, the end-to-end convergence is facilitated by (not replaced by) more local convergence of subsystems with their own control objectives.

The overview is shown in section 4 on page 4.

The entire thing could conceivably be done without capsules at all, as with the aforementioned designs, but this group is apparently showing improved results when stopping short of full convergence on the MNIST.  No comparison is shown with LSTM, but even if LSTM was showing greater results, this direction of research is an excellent one because of the probabilitic way that it approaches an object.

Consider the classic case of waving back at someone who, as it turns out, was waving at someone behind you.  The existence of an object or an action is necessarily probabilitic, and to construct the depth of network required to model all that complexity with ReLU is probably unrealistic as the expectations increase in AI system requirements.


Which simulation platform is used by DeepMind (and others) to handle inverse kinematics musculoskeletal? Which simulation platform is used by DeepMind and others to handle inverse kinematics musculoskeletal simulation, etc., for reinforcement learning simulations and agents?

I thought they use Unity or Unreal but I assume that would be resource-heavy.

DeepMind used MuJoCo (see also the related paper MuJoCo: A physics engine for model-based control) for the simulations, as they stated in section 3.1 of their paper Emergence of Locomotion Behaviours in Rich Environments (2017), which is the paper you should read to know more about their results related to those animations of skeletons that try to walk or jump (but do it weirdly).


Is reinforcement learning possible for chatbots? The reinforcement learning paradigm has the aim to determine the optimal actions for a robot. A typical example is a maze finding robot, but reinforcement learning can also be used for training a robot to play the pong game. The principle is based on a reward function. If the robot is able to solve a problem, he gets a score from the underlying game engine. The score can be positive, if the robot reaches the end of a maze, or it can be negative, if he is colliding with an obstacle. The principle itself is working quite well, that means for simpler applications it is possible to train a robot to play a game with reinforcement learning.

Chatbots are a different category of artificial intelligence. They are working not with actions but with natural language. Person #1 is opening a dialogue with “Hi, I'm Alice”, while person #2 is responding with “Nice to meet you”. What is missing here is an underlying game which is played. There is no reward available for printing out a certain sentence. In some literature the problem of language grounding was discussed seriously, but with an unclear result. It seems, that a classical game for example pong, and a chatbot conversation doesn't have much in common.

Is it possible to combine Reinforcement Learning with chatbot design? The problem is, that a speech-act should be connected to a reward. That means, a well formulated sentence gets +10 points but a weak sentence gets -10 points. How can this be evaluated?

The problem is indeed that the 'rules' of conversations are not as fixed as the rules for games. However, you can make use of  descriptive formalism from Discourse Analysis, called adjacency pairs. These describe regularities between utterances on a local level, for example greeting/reply, which would match your "Hi, I'm Alice" and "Nice to meet you".

You will need to be able to classify utterances by your chat bot according to a set of possible responses, and then you can see if a valid response is produced for any given utterance. If the user asks a question, then a greeting will not be a good answer, but a statement could be, if it was a response to the question. This is leaving aside the content and focuses merely on the formal characteristics of the utterance.

If you want to know more about the topic, have a look at Conversation Analysis, which is the linguistic field dealing with the subject.


How can genetic programming be used to prevent cyberterrorism? I've read on wiki that genetic programming has 'outstanding results' in cyberterrorism prevention.

Further more, this abstract says:

  Using machine-coded linear genomes and a homologous crossover operator in genetic programming, promising results were achieved in detecting malicious intrusions.

I've checked the study, but it's still not clear for me.

How exactly was this detection achieved from the technical perspective?

They treated it as a classification problem. While it's common to use some variety of Neural Nets (NNs) to build classifiers, Genetic Programming (GP) can also be used for this purpose. In contrast to NN classifiers, GP can use a wider range of operations (e.g. if,while,logical statements,arbitrary mathematical functions etc) to perform the classification than weighted arithmetic expressions involving an activation function. Whether or not this is actually of benefit depends on the specific application.

In addition, the abstract implies their algorithm is adaptive (i.e. responds in some fashion to the nature of incoming attacks), which would most easily be achieved by continuing to run the GP program in the background to monitor potential intrusions.


How important is learning to learn for the development of AGI? Some people say that abstract thinking, intuition, common sense, and understanding cause and effect are important to make AGI.

How important is learning to learn for the development of AGI?

Learning to learn (also known as meta-learning) is very important for the development of artificial general intelligence (AGI), given that one of the desirable and fundamental properties of an AGI is the adaptability to different environments and the ability to continually learn, and meta-learning can be used to achieve that.

Meta-learning is thus related to multi-task, continual (or lifelong) and transfer learning. There are several approaches to meta-learning, such as MAML, but there is usually a meta-learner and a learner.

In the paper Building Machines That Learn and Think Like People (2016), Marcin Andrychowicz et al. argue that truly human-like learning and thinking
machines (or AGI) should

build causal models of the world that support explanation and understanding
ground learning in intuitive theories of physics and psychology, to support and enrich the knowledge that is learned
harness compositionality and learning-to-learn to rapidly acquire and generalize knowledge to new tasks and situations

Another person that thinks that meta-learning is important for the development of AGI is Ben Goertzel. See e.g. his article From Here to Human-Level Artificial General Intelligence in Four (Not All That) Simple Steps.


What's the difference between a 1d tensor and a 2d tensor with 1 dimension? I'm doing a TensorFlow tutorial, where they convert an array of the numbers [1,2,3] to a tensor like this:
const xs = tf.tensor2d([1, 2, 3], [3, 1])

The shape is [3,1] because there is one row with 3 numbers.
My question is, why would they use a 2D tensor, isn't this just exactly the same as:
const xs = tf.tensor1d([1, 2, 3])

The required shape of the tensor $T$ depends on the shape of other tensors that are involved in the same operations of that same tensor $T$ and the required/desired shape of the resulting tensor, in the same way that the number of columns of the matrix $M \in \mathbb{R}^{n \times m}$ needs to match the number of rows of the matrix $M' \in \mathbb{R}^{n' \times m'}$ when you perform the matrix multiplication $M M'$, i.e. in order for $M M'$ to be well-defined, $m = n'$.
So, in your case, although the tensors contain the same elements, it might not be possible to use both in the same operations. Without more context/details, I cannot specifically answer why the first tensor was used in the tutorial you're mentioning.


Best architecture to learn image rotation? Given an input image and an angle I want the output to be the image rotated at the given angle.

So I want to train a neural network to do this from scratch.

What sort of archetecture do you think would work for this if I want it to be lossless?

I'm thinking of this archetecture:

256x256 image

--> convolutions to 64x64 image with 4 channels

--> convolutions to 32x32 image with 16 channels and so on

until a 1 pixel image with 256x256 channels.

And then combine this with the input angle, and then a series of deconvolutions back up to 256x256.

Do you think this would work? Could this be trained as a general rotation machine? Or is there a better archetecture?

I would also like to train the same archetecture to do other transforms.

This would likely suffer from the blurry image problem that autoencoders are known to suffer from. See also here.
On the other hand, using GAN's to sharpen your images doesn't seem particularly helpful since you seem to be lookng for a way to rotate general images, not ones of a specific domain. Moreover, there's almost certainly going to be some loss

It seems unreasonable/overkill to use neural networks to do this transformation. I suggest treating the rotation as a linear transformation, mapping certain pixel locations to other pixel locations (in fact, you can make this transformation differentiable with respect to its inputs using some deep learning library's autodifferentiation tools, if that's what you're looking for).
Since rotations are just linear maps that send pixels to other pixels, this will be much more computationally efficient.


Why is the perceptron criterion function differentiable? I'm reading chapter one of the book called Neural Networks and Deep Learning from Aggarwal.
In section 1.2.1.1 of the book, I'm learning about the perceptron. One thing that book says is, if we use the sign function for the following loss function: $\sum_{i=0}^{N}[y_i - \text{sign}(W * X_i)]^2$, that loss function will NOT be differentiable. Therefore, the book suggests us to use, instead of the sign function in the loss function, the perceptron criterion which will be defined as:
$$ L_i = \max(-y_i(W * X_i), 0) $$
The question is: Why is the perceptron criterion function differentiable? Won't we face a discontinuity at zero? Is there anything that I'm missing here?

$\max(-y_i(w x_i), 0)$ is not partial derivable respect $w$ if $w x_i=0$.
Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval of the weights.
Assume $y_i = 1$ and $w x_i < 0$ (that is, an error of type "false negative").
In this case, function $[y_i - \text{sign}(w x_i)]^2 = 4$. Derivative on all interval $w x_i < 0$ is zero, thus, the learning algorithm has no any way to decide if it is better increase or decrease $w$.
In same case, $\max(-y_i(w x_i), 0) = - w x_i$, partial derivative is $-x_i$. The learning algorithm knows that it must increase $w$ value if $x_i>0$, decrease otherwise. This is the real reason this loss function is considered more practical than previous one.
How to solve the problem at $w x_i = 0$ ? simply, if you increase $w$ and the result is an exact $0$, assign to it a very small value, $w=\epsilon$. Similar logic for remainder cases.


Is the pattern recognition capability of CNNs limited to image processing? Can a Convolutional Neural Network be used for pattern recognition in problem domains without image data? For example, by representing abstract data in an image-like format with spatial relations? Would that always be less efficient?
This developer says current development could go further but not if there's a limit outside image recognition.

Convolutional Nets (CNN) rely on mathematical convolution (e.g. 2D or 3D convolutions), which is commonly used for signal processing. Images are a type of signal, and convolution can equally be used on sound, vibrations, etc. So, in principle, CNNs can find applications to any signal, and probably more.

In practice, there exists already work on NLP (as mentioned by Matthew Graves), where some people process text with CNNs rather than recursive networks. Some other works apply to sound processing (no reference here, but I have yet unpublished work ongoing).

Original contents: In answer to the original title question, which has changed now. Perhaps need to delete this one.

Research on adversarial networks (and related) show that even deep networks can easily be fooled, leading them to see a dog (or whatever object) in what appears to be random noise when a human look at it (the article has clear examples).

Another issue is the generalization power of a neural network. Convolutional nets have amazed the world with their capability to generalize way better than other techniques. But if the network is only fed images of cats, it will recognize only cats (and probably see cats everywhere, as by adversarial network results). In other words, even CNs have a hard time generalizing too far beyond what they learned from.

The recognition limit is hard to define precisely. I would simply say that the diversity of the learning data pushes the limit (I assume further detail should lead to more appropriate venue for discussion).


Count number of objects in image using CNN I'm looking for neural network architecture that excel in counting objects. For example, CNN that can output the number of balls (or any other object) in a given image.
I already found articles about crowd counting, I'm looking for articles about different types of objects.

Thank

You can run YOLO, then count the number of occurrences.


How to compute the target for double Q-learning update step? I've already read the original paper about double DQN but I do not find a clear and practical explanation of how the target $y$ is computed, so here's how I interpreted the method (let's say I have 3 possible actions (1,2,3)):

For each experience $e_{j}=(s_{j},a_{j},r_{j},s_{j+1})$ of the mini-batch (consider an experience where $a_{j}=2$) I compute the output through the main network in the state $s_{j+1}$, so I obtain 3 values.

I look which of the three is the highest so: $a^*=arg\max_{a}Q(s_{j+1},a)$, let's say $a^*=1$

I use the target network to compute the value in $a^*=1$ , so  $Q_{target}(s_{j+1},1)$

I use the value at point 3 to substitute the value in the target vector associeted with the known action $a_{j}=2$, so: $Q_{target}(s_{j+1},2)\leftarrow r_{j}+\gamma Q_{target}(s_{j+1},1)$, while $Q_{target}(s_{j+1},1)$ and $Q_{target}(s_{j+1},3)$, which complete the target vector $y$, remain the same.

Is there anything wrong?

$$Y_{t}^{\text {DoubleDQN }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}\right), \boldsymbol{\theta}_{t}^{-}\right)$$
The only difference between the "original" DQN and this one is that you use your $Q_\text{est}$ with the next state to get your action (by choosing the action with the highest Q).
Afterward, you just figure out what the target $Q$ is given that action, by selecting the $Q$ belonging to that action from the target_network (instead of using the argmax a directly on the target Q network).
About the formula

$\theta_{t}^{-}$ above it means frozen weights, so it represents the target Q network.

the other $\theta_{t}$ represents the "learnable weights" so the estimate Q network.


Why does a fully connected layer only accept a fixed input size? I'm studying how SPP (Spatial, Pyramid, Pooling) works. SPP was invented to tackle the fix input image size in CNN. According to the original paper https://arxiv.org/pdf/1406.4729.pdf, the authors say:

  convolutional layers do not require a fixed image size and can
  generate feature maps of any sizes. On the other hand, the
  fully-connected layers need to have fixed size/length input by their
  definition. Hence, the fixed size constraint comes only from the
  fully-connected layers, which exist at a deeper stage of the network.

Why does a fully connected layer only accepts a fixed input size (but convolutional layers don't)? What's the real reason behind this definition?

A convolutional layer is a layer where you slide a kernel or filter (which you can think of as a small square matrix of weights, which need to be learned during the learning phase) over the input. In practice, when you need to slide this kernel, you will often need to specify the "padding" (around the input) and "stride" (with which you convolve the kernel on the input), in order to obtain the desired output (size). So, even if you receive inputs of different sizes, you can change these values, like the padding or the stride, in order to produce a valid output (size). In this sense, I think, we can say that convolutional layers accept inputs of (almost) any size.

The number of feature maps does not depend on the kernel or input (sizes). The number of features maps is determined by the number of different kernels that you will use to slide over the input. If you have $K$ different kernels, then you will have $K$ different feature maps. The number of kernels is often a hyper-parameter, so you can change it (as you please).

A fully connected (FC) layer requires a fixed input size by design. The programmer decides the number of input units (or neurons) that the FC layer will have. This hyper-parameter often does not change during the learning phase. So, yes, FC often accept inputs of fixed size (also because they do not adopt techniques like "padding").


Why does Q-learning converge to the optimal policy, even if the agent acts sub-optimally? In Q-learning, during training, it doesn't matter how the agent selects actions. The algorithm always converges to the optimal policy. Why does this happen? What's the intuition?

Q-learning is an off-policy learning algorithm. We are following the behaviour policy, $b$, which is $\epsilon-$greedy. This behaviour policy need not be an optimal policy rather it is a more explorable policy. But we are learning the target policy, $\pi$, which is argmax of state action value $(Q(s,a))$. This target policy is by definition optimal policy.
From the $\epsilon$-greedy policy improvement theorem we can show that for any $\epsilon$-greedy policy (I think you are referring to this as a non-optimal policy) we are still making progress towards the optimal policy and when $\pi^{'}$ = $\pi$ that is our optimal policy (Rich Sutton's book Chapter-5). Here $\pi^{'}$ is the new policy and $\pi$ is the previous policy.

Think of this diagram, where we are selecting action based on $\epsilon$-greedy policy but still making progress towards the optimal policy $\pi_*$.


What is an identity recurrent neural network? What is an identity recurrent neural network (IRNN)? What is the difference between an IRNN and RNN?

An identity recurrent neural network (IRNN) is a vanilla recurrent neural network (as opposed to e.g. LSTMs) whose recurrent weight matrices are initialized with the identity matrix, the biases are initialized to zero, and the hidden units (or neurons) use the rectified linear unit (ReLU).

An IRNN can be trained more easily using gradient descent (as opposed to a vanilla RNN that is not an IRNN), given that it behaves similarly to an LSTM-based RNN, that is, an IRNN does not suffer (much) from the vanishing gradient problem.

The IRNN achieves a performance similar to LSTM-based RNNs in certain tasks, including the adding problem (a standard problem that is used to examine the power of recurrent models in learning long-term dependencies). In terms of architecture, the vanilla RNNs are much simpler than LSTM-based RNNs, so this is an advantage.

For more details, see the paper A Simple Way to Initialize Recurrent Networks of Rectified Linear Units (2015), by Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton. See also this Keras implementation of the MNIST experiment described in the linked paper.


How to add some data input in a CNN? There is this problem I have encountered, I was trying to classify the pixels from input image into classes, sort of like segmentation, using a encoder-decoder CNN. The “interested” pixels usually locate in the top right corner of the input image, but the input images are too big, which I have to slice them in patches, by doing this, each input patch loses its “which region of the whole picture it’s from” information.

I'm using pytorch, I thought of manually add this patch location info into the input, but then it will be convoluted, which does make sense to me since it's not a part of an image.

I'm new to this, not sure if I'm thinking the whole thing right, how should I manually add this info into the input correctly or if there is some keywords I can do some researches, in order to let the CNN taking position into account? Thank you.

If your interest is positional information, encode it!

This could include learning an embedding for each position and leveraging that in your model. You could also use an approach to hard-encode rather than learn it (kinda like adding sinusoids in the transformer paper Attention is All You Need

an example of a paper that encodes the 2D positional info: Attention Augmented Convolutional Networks


How to use text as an input for a neural network - regression problem? How many likes/claps an article will get I am trying to predict the number of likes an article or a post will get using a NN.
I have a dataframe with ~70,000 rows and 2 columns: "text" (predictor - strings of text) and "likes" (target - continuous int variable). I've been reading on the approaches that are taken in NLP problems, but I feel somewhat lost as to what the input for the NN should look like.
Here is what I did so far:

Text cleaning: removing html tags, stop words, punctuation, etc...
Lower-casing the text column
Tokenization
Lemmatization
Stemming

I assigned the results to a new column, so now I have "clean_text" column with all the above applied to it. However, I'm not sure how to proceed.
In most NLP problems, I have noticed that people use word embeddings, but from what I have understood, it's a method used when attempting to predict the next word in a text. Learning word embeddings creates vectors for words that are similar to each other syntax-wise, and I fail to see how that can be used to derive the weight/impact of each word on the target variable in my case.
In addition, when I tried to generate a word embedding model using the Gensim library, it resulted in more than 50k words, which I think will make it too difficult or even impossible to onehot encode. Even then, I will have to one hot encode each row and then create a padding for all the rows to be of similar length to feed the NN model, but the length of each row in the new column I created "clean_text" varies significantly, so it will result in very big onehot encoded matrices that are kind of redundant.
Am I approaching this completely wrong? and what should I do?

I'll answer in a couple of stages.
I feel somewhat lost as to what the input for the NN should look like.
Your choices boil down to two options, each with their own multitude of variants:

Vector Representation: Your input is a vector of the same size as your vocabulary where the elements represent the tokens in the input example. The most basic version of this is a bag-of-words (BOW) encoding with a 1 for each word that occurs in the input example and a 0 otherwise. Some other variants are (normalized) word counts or TF-IDF values. With this representation padding will not be necessary as each example will be encoded as a vector of the same size as the vocabulary. However, it suffers from a variety of issues: the input is high-dimensional and very sparse making learning difficult (as you note), it does not encode word order, and the individual word representations have little (TF-IDF) to no (BOW, counts) semantic information. It also limits your NN architecture to a feed-forward network, as more "interesting" architectures such a RNNs, CNNs, and transformers assume a matrix-like input, described below.

Matrix Representation: Here your input representation is a matrix with each row being a vector (i.e. embedding) representation of the token at that index in the input example. How you actually get the pretrained embeddings into the model depends on a number of implementation-specific factors, but this stackoverflow question shows how to load embeddings from gensim into PyTorch. Here padding is necessary because the input examples will have variable numbers of tokens. This stackoverflow answer shows how to add zero padding in PyTorch.
This representation will be significantly better than the vector representation as it is relatively low-dimensional and non-sparse, it maintains word order, and using pretrained word-embeddings means your model will have access to semantic information. In fact, this last point leads to your next question.

Learning word embeddings creates vectors for words that are similar to each other syntax-wise, and I fail to see how that can be used to derive the weight/impact of each word on the target variable in my case.
Word embeddings are based on the assumptions of distributional semantics, the core tenet of which is often quoted as "a word is characterized by the company it keeps". That is, the meaning of a word is how it relates to other words. In the context of NLP, models can make better decisions because similar words are treated similarly from the get-go.
For example, say that articles about furry pets get a lot of likes (entirely plausible if you ask me). However, the mentions of furry pets in these articles will be varied, including words like "dog", "cat", "chinchilla", "poodle", "doggo", "good boy", etc. An input representation that treats these mentions as completely distinct (such as BOW) will need to learn individual correlations between each word and the number of likes (that's a lot of learning). A well-trained word embedding, on the other hand, will be able to immediately group these mentions together and learn general correlations between groups of similar words and likes. Fair warning, this is a very imprecise description of why word embeddings work, but I hope it gives you some intuitive understanding.
Finally, since you're doing regression, make sure you choose your objective function accordingly. Mean squared error would be my first try.


Is REINFORCE the same as 'vanilla policy gradient'? I don't know what people mean by 'vanilla policy gradient', but what comes to mind is REINFORCE, which is the simplest policy gradient algorithm I can think of. Is this an accurate statement?

By REINFORCE I mean this surrogate objective

$$ \frac{1}{m} \sum_i \sum_t log(\pi(a_t|s_t)) R_i $$

Where I indices over the $m$ episodes and $t$ over time steps, and $R_i$ is the total reward of the episode. It's also common to replace $R_i$ with something else, like a baselined version $R_i - b$ or use the future return, potentially also with a baseline $G_{it} - b$. However, I think even with these modifications to the multiplicative term, people would still call this 'vanilla policy gradient'. Is that correct?

By vanilla policy gradient, I think what is meant is normally any arbitrary policy gradient for the purposes of formalization(or whatever is trying to be communicated).

For example, let $J(\theta)$ be any policy objective function. Then our policy gradient would be $\begin{equation} \nabla_\theta J(\theta)  \end{equation}$ where the change in our policy is $\Delta \theta = \alpha \nabla_\theta J(\theta)  $ with $\alpha$ being our step size.

This will, of course, vary as notational convention and term definition can change between practitioners.


How do I know that the DQN has learnt an appropriate Q function? Is there any sanity check to know whether the Q functions learnt are appropriate in deep Q networks? I know that the Q values for end states should approximate the terminal reward. However, is it normal that Q values for the non-terminal states have higher values than those of the terminal states?

The reason why I want to know whether Q values learnt are appropriate is because I want to apply the doubly robust estimator for off-policy value evaluation. Using doubly robust requires a good Q value estimate to be learnt for each state.

DQN is famous for doing over-approximation on Q function. However, having over approximated Q does not imply that it does not perform well in the environment. (unless it looks ridiculously high) From my experience, high learning rate usually cause over approximated Q, or mistakes made in the code. Best way to check is to see plot of Q function when running on environment to see if it makes any sense. (e.g. It should go lower for bad states and vice versa)


What are possible functions assigned on decision nodes for decision tree prediction? In Decision Tree or Random Forest, each tree has a collection of decision nodes (in which each node has a threshold value) and a class labels (or regression values).

I know that threshold values are used for comparison with a corresponding feature value. As far as I know, the comparison is performed either "<", ">"  or "==" predicate.
Anything else for the functions taking threshold value and a feature value as inputs??

For a binary split, there are only three possible operations (or arguably only two if you consider one-hot encoding). Any other kind of split would simply not be binary. Almost every tree-based model is restricted to binary splits, due to the combinatorial explosion when considering ternary or even more complex splits.

Of course you could write your own algorithm that uses recursive non-binary splits. Beware though that you'll be facing the same difficulty that has led the vast majority of algorithms to be limited to strictly binary splits.

Have a look at this related question.


Meaning of Actor Output in Actor Critic Reinforcement Learning In actor critic, The equations for calculating the loss in actor critic are an

actor loss (parameterized by $\theta$)

$$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$

and a critic loss (parameterized by $w$)

$$r(s_t,a_t) + \gamma Q_w(s_{t+1}, a_{t+1}) - Q_w(s_{t}, a_t).$$

This is bootstrapping in experience replay:

$$
L_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \left[ \left(r + \gamma \max_{a'} Q(s', a'; \theta_i^-) - Q(s, a; \theta_i)\right)^2 \right]
$$

It is clear that bootstrapping is comparable to the critic loss, except that the $max$ operation is lacking from the critic.

As i see it, (correct me if I'm wrong):

$Q(s_t,a_t) = V(s_{t+1}) + r_t$ where $a_t$ is the actual action that had been taken.

The critic, as I understand, estimates $V(s)$

My question:

What exactly is the critic calculating?

What In actor critic outputs $Q(s_{t+1},a_{t+1})$?

It seems to me like the critic calculates the average next state $s_{t+1}$ value, over all possible actions, with their corresponding probabilities, yielding

$Q(s_t, a_t) = r_t + \sum_{a_{t+1} \in A}P(a_{t+1}|s_t)V(s_{t+1})$

Which would mean that in order to get $Q(s_{t+1}, a_{t+1})$ for the above formula, I would need to calculate

$Q(s_{t+1}, a_{t+1}) = r_{t+1} + \sum_{a_{t+2} \in A}P(a_{t+2}|s_{t+1})V(s_{t+2})$

Where $V(s_{t+2})$ is the critic output on $s_{t+2}$, a state we get to by taking action $a_{t+1}$ from state $s_{t+1}$ but I am not sure that is indeed the meaning of the critic output and still it is unclear to me how to get $Q(s_{t+1}, a_{t+1})$ from actor critic.

If indeed that is what's being calculated, then why is it mathematically true that an improvement is being made? Or why does it make sense (even if not mathematically always true)?

Practical use:

I want to use actor critic with experience replay in an environment with a large action space (could be continuous). Therefore, I cannot use the $max$ term. I need to understand the correct equation for the critic loss, and why it works.

When using the loss function for the critic described in your question, the Actor-Critic is an on-policy approach (as are most Actor-Critic methods). Your intuition as to what it is learning seems to be quite close, but the notation/terminology is not quite on point.

First it's important to realize that the $Q(s, a)$ critic is an estimator, we're training it to estimate state-action values. You could say that we are training it such that it can hopefully provide accurate estimates of:

$$Q_w^{\pi} (s_t, a_t) \approx \mathbb{E}_{\pi} \left[ r_t + \gamma V^{\pi}(s_{t+1}) \right].$$

You'll notice I've added quite a number of symbols there in comparison to your $Q(s_t, a_t) = r_t + V(s_{t+1})$:

I have added the $\pi$ superscript to $Q$ and $V$; this denotes the behaviour policy, which is the policy that we're using to generate experience. In on-policy methods, this is equal to the target policy (the policy for which we're learning to predict values). Adding this superscript makes explicit the fact that we're learning expected returns for states and state-action pairs that are only accurate under the assumption that we continue following the $\pi$ policy from state $s_{t+1}$ onwards.
I added the discount factor $\gamma$, which is probably just a tiny detail you forgot.
I added $\mathbb{E}_{\pi}$ to indicate that we're trying to estimate an expectation under $\pi$ (and the environment's dynamics).

So, the critic is trained to estimate $Q^{\pi}(s, a)$, which can intuitively be interpreted as the long-term discounted rewards that we expect to collect when executing $a$ in $s$, and selecting actions according to the distribution $\pi$ subsequently. It definitely still is trying to estimate $Q(s, a)$ values for state-action pairs, not just $V(s)$ values for states alone.

  What In actor critic outputs $Q(s_{t+1},a_{t+1})$?

In practice, when using the loss function described in your question, $a_{t+1}$ really simply is a single action selected in an actual trajectory of experience by the policy $\pi$. The trained network simply takes $s_{t+1}$ as input, and the output corresponding to a single action $a_{t+1}$ as selected by the policy is used as the value for $Q(s_{t+1},a_{t+1})$ in the update rule.

The update rule does not involve any sum over all actions, multiplied with their probabilities. The "trick" is that we do not just run the update rule a single time, but we expect to generate lots (sometimes millions) of trajectories as experience, and we repeatedly run the update rule. In different trajectories, we'll experience the different actions $a_t$ as samples with approximately the correct frequencies, and in expectation we'll have proper update targets (except for potential bias resulting from function approximation).

  I want to use actor critic with experience replay in an environment with a large action space

The Actor-Critic method in your question is, as I mentioned above, on-policy. This means that the experience used in update rules has to be generated according to exactly the same policy for which you are also learning value estimates. This is incompatible with the idea of experience replay, because old trajectories stored in a replay buffer were generated by older versions of your policy.

There are off-policy Actor-Critic methods which can correct for the mismatch in distributions and use experience replay, but these are going to be quite a bit more complicated. Examples are ACER and IMPALA.


What exactly is an interpretable machine learning model? From this page in Interpretable-ml book and this article on Analytics Vidhya, it means to know what has happened inside an ML model to arrive at the result/prediction/conclusion.
In linear regression, new data will be multiplied with weights and bias will be added to make a prediction.
And in boosted tree models, it is possible to plot all the decisions as trees that results in a prediction.
And in feed-forward neural networks, we will have weights and biases just like linear regression and we just multiply weights and add bias at each layer, limiting values to some extent using some kind of activation function at every layer, arriving finally at prediction.
In CNNs, it is possible to see what happens to the input after having passed through a CNN block and what features are extracted after pooling (ref: what does a CNN see?).
Like I stated above, one can easily know what happens inside an ML model to make a prediction or conclusion. And I am unclear as to what makes them un-interpretable!. So, what exactly makes an algorithm or it's results un-interpretable or why are these called black box models? Or am I missing something?

In a simple linear model of the form $y = \beta_0 + \beta_1 x $ we can see that increasing $x$ by a unit will increase the prediction on $y$ by $\beta_1$. Here we can completely determine what the effect on the models prediction will be by increasing $x$. With more complex models such as neural networks it is much more difficult to tell due to all the calculations that a single data point is involved in. For instance, in a CNN as you mentioned, if I changed the value of a pixel in an image we were passing through the CNN you wouldn't really be able to tell me exactly the effect this would have on the prediction like you can with the linear model.


Is a super-intelligence limited by the intelligence of its creator? An argument against the possibility of super-intelligence is that the intelligence of a creation will be limited by the intelligence of its creator. How reasonable is this argument?

Of course the intelligence of a product is limited by the intelligence of its creator. Just not to the intelligence of its creator.

That would be about as reasonable as the idea that the speed of a car is limited to the speed of its creator.

Or the playing strength of a chess program to the Elo of its creator.

Or the ability of a neural network to differentiate between dozens of dog breeds to the dog expertise of its creator.

So, not very reasonable.


What are the meanings of these (P(x;y), P(x;y,z),P(x,y;z))? I was reading a machine learning book that uses probabilities like these:

$P(x;y), P(x;y,z), P(x,y;z)$

I couldn't find what they are and how can I read and understand them?

Apart from the context, I saw one of these probabilities on it here:

This means "Parameterized by".

First, we all agree on the idea of conditional probabilities:

$$P(X | Y) = P(X,Y) / P(Y)$$

That is, the probability that X happens given that we've seen Y happen, is the fraction of worlds in which Y happened that also contain X. This is uncontroversial.

If you're a Bayesian, you might view parameters themselves as variables in a statistical model. So you might want to speak about the probability of a parameter taking on a certain value, or the probability of the data given that certain parameters have taken on certain values. In that case, you might write something like $P(D | \theta)$ to denote the probability of the data given a parameter $\theta$.

If you're a frequentist, you might find notation like this unsettling, because parameters don't have probabilities under the frequentist view, but instead have fixed values. You could talk about the probability of observing the data under a particular parameterization by defining a family of parameterized probability density functions, and then writing something like $P(D;\theta)$. You might also do this as a Bayesian if you wanted to clearly differentiate between model parameters and other things you might observe.

$P(x;y)$ then is read as "The probability of x under probability density function P, parameterized by y."

$P(x;y,z)$ is "The probability of x under probability density function P, parameterized by y and z."

$P(x,y;z)$ is "The probability of x and y jointly, under probability density function P, parameterized by z."

You could also write something like $P(X|Y; z)$, which would be "The conditional probability of X given that we observe Y, under probability density function P, parameterized by z."

An example of the latter would be in something like logistic regression. We might wish to know $P(L | D ; \theta)$, where L is the label of the data, D are the features we observe, and $\theta$ are the coefficients of the regression.


What is the expectation of an empirical model in model based RL? In the paper - "Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems", on page 1083, on the 6th line from the bottom, the authors define expectation of the empirical model as
$$\hat{\mathbb{E}}_{s,s',a}[V(s')] = \sum_{s' \in S} \hat{P}^{a}_{s, s'}V(s').$$
I didn't understand the significance of this quantity since it puts $V(s')$ inside an expectation while assuming the knowledge of $V(s')$ in the definition on the right.
A clarification in this regard would be appreciated.
EDIT:
The paper defines $\hat{P}^{a}_{s, s'}$ as,
$$\hat{P}^{a}_{s, s'} = \frac{|(s, a, s', t)|}{|(s, a, t)|}.$$
Where $|(s, a, t)|$ is the number of times state $s$ was visited and action $a$ was taken and $|(s, a, s', t)|$ as the number of times among the $|(s, a, t)|$ times $(s, a)$ was visited when the next state landed in was $s'$ during model learning.
No explicit definition for $V$ is provided however, $V^{\pi}$ is defined as the usual expected discounted return, using the same definition as Sutton and Barto or other sources.

If I understand your question correctly, the significance of this is due to the fact that $s'$ is random. In the RHS of the equation it is assumed that $V(\cdot)$ is known for each state, but the quantity is measuring the expected value of the next state given the current state and action.


How Does AlphaGo Zero Implement Reinforcement Learning? AlphaGo Zero (https://deepmind.com/blog/alphago-zero-learning-scratch/) has several key components that contribute to it's success:

A Monte Carlo Tree Search Algorithm that allows it to better search and learn from the state space of Go
A Deep Neural Network architecture that learns the value and policies of given states, to better inform the MCTS.

My question is, how is this Reinforcement Learning?  Or rather, what aspects of this algorithm specifically make it a Reinforcement Learning problem?  Couldn't this just be considered a Supervised Learning problem?

If you learn a policy or a value function from experience (that is, interaction with an environment), that's RL. In the case of AlphaGo, the MCTS is used to acquire the experience.

RL could in fact be considered supervised learning (SL) or, more specifically, self-supervised learning, where the experience corresponds to the labels in SL, especially nowadays with techniques like experience replay.


How to make a fair comparison of a convolutional neural network (cNN) vs a mutlilayer perceptron (MLP)? I'm working with deep learning on some EEG data for classification, and I was wondering if there's any systematic/mathematical way to define the architecture of the networks, in order to compare their performance fairly.

Should the comparison be at the level of neurons (e.g. number of neurons in each layer), or at the level of weights (e.g. number of parameters for training in each type of network), or maybe something else?

One idea that emerged was to construct one layer for the MLP for each corresponding convolutional layer, based on the number of neurons after the pooling and dropout layers.

Any ideas? If there's any relative work or paper regarding this problem I would be very grateful to know.

Thank you for your time

Konstantinos

The best best way to monitor an architectures performance would be comparing the resource utilization, model accuracy, value loss and confusion matrix.

e.g VGG16 consumes less system resources in comparison to inception V3

There is an article that goes in depth.


Does the paper "On the difficulty of training Recurrent Neural Networks" (2013) assume, falsely, that spectral radii are $\ge$ square matrix norms? (link to paper in arxiv)

In section 2.1 the authors define $\gamma$ as the maximum possible value of the derivative of the activation function (e.g. 1 for tanh.) Then they have this to say:

  We first prove that it is sufficient for $\lambda_1 < \frac{1}{\gamma}$, where $\lambda_1$ is the absolute value of the largest eigenvalue of the recurrent weight matrix $W_{rec}$, for the vanishing gradient problem to occur.

Then they use the submultiplicity ($\|AB\| \le \|A\|\|B\|$) of the 2-norm of the Jacobians to obtain the following inequality:

$$ \forall x, \| \frac{\partial x_{k+1}}{\partial x_k} \| \le \| W_{rec}^\top \| \| diag(\sigma'(x_k))\| < \frac{1}{\gamma} \gamma < 1 $$

Here

$x_k$ is the pre-activated state vector of the RNN
$W_{rec}$ is the weight matrix between states (i.e. $x_k = W_{rec} \times \sigma(x_{k-1}) + b$ )
$\sigma()$ is the activation function for the state vector
$diag(v)$ is the diagonal matrix version of a vector $v$

They appear to be either substituting the norm of the weight matrix $\|W_{rec}^\top\|$ for its largest eigenvalue $|\lambda_1|$ (eigenvalues are the same for transposes) or just assuming that this norm is less than or equal to the eigenvalue. This bothers me because the norm of a matrix is bounded below, not above, by this eigenvalue/spectral radius (see lemma 10 here and this math SE question)

They seem to assume that

$$\| W_{rec}^\top \| \le \lambda_1 < \frac{1}{\gamma}  $$

But really

$$ \| W_{rec}^\top \| \ge \lambda_1 $$

It is an error. But it is also not the in final version of the paper (arxiv.) The final version of the paper can be found here where they replace "absolute value of the largest eigenvalue" with "largest singular value".

  We first prove that it is sufficient for $\lambda_1 < \frac{1}{\gamma}$, where $\lambda_1$ is the largest singular value of $W_{rec}$, for the vanishing gradient problem to occur.


Is the smartest robot more clever than the stupidest human? Most humans are not good at chess. They can't write symphonies. They don't read novels. They aren't good athletes. They aren't good at logical reasoning. Most of us just get up. Go to work in a factory or farm or something. Follow simple instructions. Have a beer and go to sleep.

What are some things that a clever robot can't do that a stupid human can?

First Question

To treat this question in a scientific way, because I think it is a reasonable enough question that draws on the realities of postmodern culture in post industrialized societies to be treated scientifically, we should define some things.

The most difficult is intelligence, which is the realm in which smartness, cleverness, and stupidity reside.

Let's go through the list.

Most humans are not good at chess, but humans invented the game.
Most humans can't write symphonies, but humans invented them.
Most humans don't read novels, but digital computers can't write them yet and can't learn ethical balance through the reading of them like humans can.
Most humans are not Olympic level athletes, yet humans developed Olympics and robots are not yet Olympiads.
Most humans (whether or hot they are good at logical reasoning) don't employ it much other than to, "Get by."

The question is fine until it devolves into the dismissal of the intelligence that people apply to their method of earning income, which for many if not most people requires more than just getting up, commuting in, following some simple instructions, and going to sleep mildly inebriated.  Let's replace this last part with this.

Most humans do not work with the intention of optimizing quality of the product or service by measuring their own quality and seeking educational resources to improve the velocity, reliability, or accuracy of their work output (unless programs are instituted to incentivize these things in the workplace.)

If we define intelligence as the union of these things, for simplicity's sake, we have this (which is subject to change as AI develops).

Playing chess: AI wins
Designing games: Humans win
Writing symphonies: Humans win
Writing novels: Humans win
Absorbing ethics from stories: Humans win
Olympic gold: Humans win
Logical consistency: AI wins

We must, to interpret the above list correctly concede two things:

Machines may have the ability to do something according to specific quality standards but not be configured, trained, or connected appropriately to prevail.
Humans may have the ability to do something according to specific quality standards but not be educated, trained, or be properly motivated to prevail.

Second Question

What are some things that a clever robot can't do that a stupid human can?  These are a few, but they are of particular importance from certain perspectives.

Love their family and friends
Have compassion without reason
Decide what to learn
Hunt
See a future danger approaching
Entertain others
Pray

I would not dismiss these human propensities as irrelevant, even from a scientific perspective.  I would also not dismiss the possibility that these things are beyond the capabilities of silicon based entities.


Why can't neural networks learn functions outside of the specified domains? I understand that neural nets are fundamentally interpolative tools. Meaning, given a training dataset, a well trained neural net can approximate values within the domain of the training dataset. However, we are unsure about their behavior once we test against values outside that domain.

Speaking in the context of Imagenet, a NN trained on one of the classes in Imagenet will probably be able to predict an image of the same class outside Imagenet because Imagnet itself covers a huge domain for each class that whatever image we come across in the wild, its features will be accounted for by Imagnet.

Now, this intuition breaks down for me when I talk about simple functions with simple inputs. For example, consider $sin(x)$. Our goal is to train a neural net to predict the function given $x$ with a training domain $[-1, 1]$. Theoretically, the neural net should not be able to predict the values well outside that domain, right? This seems counterintuitive to me because the function behaves in a very simple and periodic way that I find it hard to believe that a neural net cannot figure out the proper transformation of that function even outside the training domain.

In short, are neural nets inherently unable to find a generalizable transformation outside the training domain no matter how simple is the function we are trying to approximate? Is this a property of the Deep Learning framework?

Are there problems where researchers were able to learn a robust generalizable transformation using neural nets outside the training domain? What are the possible conditions so that such results can happen?

The problem you discuss extends past the machine but to the man behind the machine (or woman). ML can be broken down into 3 components, the model, the data, and the learning procedure. This by the way extends to us as well. The model is our brain, the data is our experience and sensory input, and the learning procedure is there but unknown (for now $<$insert evil laugh$>$).

Your model's inability to understand that it is a sin function is normally by construction. First, though lets start with the fact that if someone showed me $sin(x)$ on $[-1,1]$, a sinusoid is the last thing id think of:

 It looks almost linear. So for sake of argument i'm going to continue on the assumption you meant an entire period ($[-\pi, \pi])$.

Now, given this most people who ever got past pre-calculus would assume sinusoid, as youd expect, but let me show you how unstable that idea is. Lets use a full period, shifted by $\frac{\pi}{2}$

Now i don't know about you but my spidey-sense would make me think of a Gaussian before anything. This is the same sinusoid along an entire period but just the slight difference changes an entire perspective.

Now lets talk about some of the mathematics. Given the uncountable $(x,sin(x))$ pairs between $[-1, 1]$, we can extract the function $sin(x)$ with certainty (assuming infintely differentiable) by simply taking the taylor series along any of the points in that domain. More so, we only need an infinitiscemally small continuous subset of the function to achieve that result, but as a person, would you be able to tell from $[-.0001, .0001]$? (i don't think so!), because by default we quantize, and what were gonna give to the computer is a quantization as well. We are giving it a finite countable set of points and expect it to to generalize to an uncountable set. This is silly, since there exists an infinite set of functions that can fit those exact points (concept of overfitting steps in here).

So if its so unfeasible for a computer to correctly solve the function, why can us humans extrapolate it so well from the $[-\pi, \pi]$ domain? The answer is our preset biases. Growing up we have dealt so much with periodicity, specifically sins and cosines, that graphed period triggered something in our brains and we just know what it is. But this isnt always a benefit, i could draw you a low-ordered polynomial on that domain that would make you think that exact same thought making you wrong.

So going back to my initial point, the problems lies with the models creator and not the model. Like i explained just now, its unfeasible without preset biases to learn the function we want, so what should we do? give up? NO, its our goal as the model architect to figure out what we want and how we can best achieve it! so if we want it to accomplish this task, lets try our best in modeling these biases. A quick example of how you could do that, is to force the function to fit the fourier coefficients, thatll probably solve this quite quickly (ofcourse this limits this models ability to solve for other common functions)

Takeaway: It's not the model thats struggling, its us using preset biases that the model does not have access to. The solution to this is be clever and figure out how to have it either start with these preset biases or learn them.


Why does k-means have more bias than spectral clustering and GMM? I ran into a 2019-Entrance Exam question as follows:

The answer mentioned is (4), but some search on google showed me maybe (1) and (2) is equal to (4). Why would k-means be the algorithm with the highest bias? (Can you please also provide references to valid material to study more?)

I'm not an expert on clustering, but here's my take below. Note that this is only based on theoretical arguments, I haven't had enough clustering experience to say if this is generally true in practice.
K-means vs GMM
K-means has a higher bias than GMM because it is a special case of GMM. K-means specifically assumes the clustering is spherical (meaning each dimension is weighted equally important) and that the clustering problem is a hard clustering problem (each data point can only belong to one label). So, theoretically, K-means should perform equal to GMM (under very specific conditions) or worse. More info
K-means vs GMM (identity covariance matrix)
K-means has a higher bias than GMM (identity covariance matrix) because it is also a special case. K-mean specifically assumes the hard clustering problem, but GMM does not. Because of this, GMM has stronger estimates for the mean of the centroids. More specifically,

[GMM] estimates the cluster means as weighted means, not assigning observations in a crisp manner to one of the clusters. In this way it avoids the problem explained above and it will be consistent as ML estimator (in general this is problematic because of issues of degeneration of the covariance matrix, however not if you assume them spherical and equal).

In practice, if you generate observations from a number of Gaussians with same spherical covariance matrix and different means, K-means will therefore overestimate the distances between the means, whereas the ML-estimator for the mixture model will not.

So, theoretically, K-means should perform equal to GMM (identity covariance matrix) or worse. More info
K-means vs Spectral clustering
K-means has a higher bias then spectral clustering because spectral clustering effectively uses K-means after processing more information from the matrices.

Spectral clustering usually is spectral embedding, followed by k-means in the spectral domain.

So yes, it also uses k-means. But not on the original coordinates, but on an embedding that roughly captures connectivity. Instead of minimizing squared errors in the input domain, it minimizes squared errors on the ability to reconstruct neighbors. That is often better.

More info


What is emperical distribution in MLE? I was reading the book Deep Learning by Ian Goodfellow. I had a doubt in the Maximum likelihood estimation section (Pg 131). I understand till the Eq 5.58 which describes what is being maximized in the problem.
$$
 \theta_{\text{ML}} = \text{argmax}_{\theta} \sum_1^m \log(p_{\text{model}}(x^{(i)};\theta))
$$
However the next equation 5.59 restates this equation as:
$$
\theta_{\text{ML}} = \text{argmax}_{\theta}  E_{x \sim \hat{p}_{\text{data}}}(\log(p_{\text{model}}(x;\theta))
$$
where $$\hat{p}_{\text{data}}$$ is described as the empirical distribution defined by the training data.
Could someone explain what is meant by this empirical distribution? It seems to be different from the distribution parametrized by theta as that is described by $$ p_{\text{model}} $$

The idea behind this kind of reasoning is that there is a "true" distribution (unknown to us, mere mortals) and that the data is generated following this distribution. But what we don't really know the shape of the distribution, all we know is the distribution of the data that we have. This is called the empirical distribution.   Let's see a simple example to illustrate the point.
Let's consider a die. Each number is equally likely to show up if we throw the die, so the true underlying distribution is uniform over the set $\{1, 2,\dots6\}$. Now let's say you ask your friend to throw the die 60 times, what you will see is likely something close to uniform over the set $\{1, 2,\dots6\}$ (not really uniform though, as that would be highly unlikely). This distribution is the empirical one, and as you collect more and more samples it will converge to the actual underlying distribution.
In your case what happened is the following:
$x_1,\dots, x_m$ is your sample (in the example above, the $60$ numbers that you see as your friend throws the die). This sample defines a distribution, $\hat{p}_{data}$. In the example above, $\hat{p}_{data}$ would likely be close to the uniform distribution over $\{1, \dots, 6\}$. Now you can think about the sum over $x_1\dots, x_m$ of $\log(p_{model}(x^{(i)};\theta))$ as the average of $\log(p_{model}(x;\theta))$, where $x$ is drawn according to $\hat{p}_{data}$.
Let me make another example with some actual numbers. Let's say you toss a fair coin $7$ times and you see $$\{H, H, H, T, H, T, H\}.$$
The empirical distribution is $\mathbb{P}(H)=5/7$, $\mathbb{P}(T)=2/7$. So if you compute the expectation $$\mathbb{E}_{x \sim \hat{p}_{data}}[\log(p_{model}(x;\theta)]$$ you get
$$\log(p_{model}(H;\theta)\cdot\mathbb{P}(H) +\log(p_{model}(T;\theta)\cdot\mathbb{P}(T) = \\ \frac{5}{7}\log(p_{model}(H;\theta) +\frac{2}{7}\log(p_{model}(T;\theta)),$$
which is what you will get if you compute the first sum you wrote.


Is it common in RL research with Atari/ALE to automatically press FIRE to start games? In some Atari games in the Arcade Learning Environment (ALE), it is necessary to press FIRE once to start a game. Because it may be difficult for a Reinforcement Learning (RL) agent to learn this, they may often waste a lot of time executing actions that do nothing. Therefore, I get the impression that some people hardcode their agent to press that FIRE button once when necessary.

For example, in OpenAI's baselines repository, this is implemented using the FireResetEnv wrapper. Further down, in their wrap_deepmind (which applies that wrapper among others), it is implied that DeepMind tends to use this functionality in all of their publications. I have not been able to find a reference for this claim though.

My question is: is it common in published research (by DeepMind or others) to use the functionality described above? I'd say that, if this is the case, it should be explicitly mentioned in these papers (because it's important to know if hardcoded domain knowledge was added to a learning agent), but I have been unable to explicitly find this after looking through a wide variety of papers. So, based on this, I'd be inclined to believe the answer is "no". The main thing that confuses me then is the implication (without reference) in the OpenAI baselines repository that the answer would be "yes".

Based on this comment in the Issue I created about this question on github, it looks like there is confirmation that at least DeepMind does not use this kind of functionality in their Atari experiments, contrary to what is implied by the comments in the OpenAI baselines code.


Is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$? I'm learning the basics of RL and I'm struggling to understand the notion of terminal state in MDPs.
To ask my question straightforwardly: is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$? If I need to be more restrictive, assume a game setting, for example, chess.
My first hypothesis would be to define the terminal state as the state $s_T$ such that $p(s',r|s_T,a) = p(s',r|s_T)$, a state from which the transition is independent of the agent's actions. But that does not seem quite right. First, there is no particular reason why this state should be unique. Second, from this definition, it could also just be an intermittent state of "lag".

I don't know if there is a general definition of the terminal state based on the MDP transition probabilities.
But remember that we define our MDP problem in a $\mathbb{S}$ set of all possible states and a $\mathbb{A}(s)$ representing the set of all possible actions for each state. Based on that, probably there aren't any possible actions for the terminal state, so the transition probability $p(s', r |s_T, a)$ can be undefined. Based on that, in episodic tasks we need to define a the set of all possible terminal states $\mathbb{S}^+$ and distinguish this with $\mathbb{S}$.


What is the purpose of an activation function in neural networks? It is said that activation functions in neural networks help introduce non-linearity.

What does this mean?
What does non-linearity mean in this context?
How does the introduction of this non-linearity help?
Are there any other purposes of activation functions?

Almost all of the functionalities provided by the non-linear activation functions are given by other answers. Let me sum them up:

First, what does non-linearity mean? It means something (a function in this case) which is not linear with respect to a given variable/variables i.e. $f(c1.x1 + c2.x2...cn.xn + b) != c1.f(x1) + c2.f(x2) ... cn.f(xn) + f(b).$
NOTE: There is some ambiguity about how one might define linearity. In polynomial equations we define linearity in somewhat a different way as compared to in vectors or some systems which take an input $x$ and give an output $f(x)$. See the second answer.
What does non-linearity mean in this context? It means that the Neural Network can successfully approximate functions (up-to a certain error $e$ decided by the user) which does not follow linearity or it can successfully predict the class of a function that is divided by a decision boundary that is not linear.
Why does it help? I hardly think you can find any physical world phenomenon which follows linearity straightforwardly. So you need a non-linear function that can approximate the non-linear phenomenon. Also, a good intuition would be any decision boundary or a function is a linear combination of polynomial combinations of the input features (so ultimately non-linear).
Purposes of activation function? In addition to introducing non-linearity, every activation function has its own features.

Sigmoid $\frac{1} {(1 + e ^ {-(w1*x1...wn*xn + b)})}$
This is one of the most common activation function and is monotonically increasing everywhere. This is generally used at the final output node as it squashes values between 0 and 1 (if the output is required to be 0 or 1). Thus above 0.5 is considered 1 while below 0.5 as 0, although a different threshold (not 0.5) maybe set. Its main advantage is that its differentiation is easy and uses already calculated values and supposedly horseshoe crab neurons have this activation function in their neurons.
Tanh $\frac{e ^ {(w1*x1...wn*xn + b)} - e ^ {-(w1*x1...wn*xn + b)})}{(e ^ { (w1*x1...wn*xn + b)} + e ^ {-(w1*x1...wn*xn + b)}}$
This has an advantage over the sigmoid activation function as it tends to centre the output to 0 which has an effect of better learning on the subsequent layers (acts as a feature normaliser). A nice explanation here. Negative and positive output values maybe considered as 0 and 1 respectively. Used mostly in RNN's.
Re-Lu activation function - This is another very common simple non-linear (linear in positive range and negative range exclusive of each other) activation function that has the advantage of removing the problem of vanishing gradient faced by the above two i.e. gradient tends to 0 as x tends to +infinity or -infinity. Here is an answer about Re-Lu's approximation power in-spite of its apparent linearity. ReLu's have a disadvantage of having dead neurons which result in larger NN's.
Also, you can design your own activation functions depending on your specialized problem. You may have a quadratic activation function which will approximate quadratic functions much better. But then, you have to design a cost function that should be somewhat convex in nature, so that you can optimise it using first-order differentials and the NN actually converges to a decent result. This is the main reason why standard activation functions are used. But I believe with proper mathematical tools, there is a huge potential for new and eccentric activation functions.
For example, say you are trying to approximate a single-variable quadratic function say $a.x^2 + c$. This will be best approximated by a quadratic activation $w1.x^2 + b$ where$w1$ and $b$ will be the trainable parameters. But designing a loss function that follows the conventional first-order derivative method (gradient descent) can be quite tough for non-monotonically increasing function.
For Mathematicians: In the sigmoid activation function $(1 / (1 + e ^ {-(w1*x1...wn*xn + b)})$ we see that $e ^ {-(w1*x1...wn*xn + b)}$ is always < 1. By binomial expansion, or by reverse calculation of the infinite GP series we get $sigmoid(y)$ = $1 + y + y^2.....$. Now in a NN $y = e ^ {-(w1*x1...wn*xn + b)}$. Thus we get all the powers of $y$ which is equal to $e ^ {-(w1*x1...wn*xn + b)}$ thus each power of $y$ can be thought of as a multiplication of several decaying exponentials based on a feature $x$, for eaxmple $y^2 = e^ {-2(w1x1)} * e^ {-2(w2x2)} * e^ {-2(w3x3)} *...... e^ {-2(b)}$. Thus each feature has a say in the scaling of the graph of $y^2$.
Another way of thinking would be to expand the exponentials according to Taylor Series:
$$e^{x}=1+\frac{x}{1 !}+\frac{x^{2}}{2 !}+\frac{x^{3}}{3 !}+\cdots$$
So we get a very complex combination, with all the possible polynomial combinations of input variables present. I believe if a Neural Network is structured correctly the NN can fine-tune these polynomial combinations by just modifying the connection weights and selecting polynomial terms maximum useful, and rejecting terms by subtracting the output of 2 nodes weighted properly.
The $tanh$ activation can work in the same way since output of $|tanh| < 1$. I am not sure how Re-Lu's work though, but due to its rigid structure and problem of dead neurons we require larger networks with ReLu's for a good approximation.
But for a formal mathematical proof, one has to look at the Universal Approximation Theorem.

A visual proof that neural nets can compute any function
The Universal Approximation Theorem For Neural Networks- An Elegant Proof

For non-mathematicians some better insights visit these links:
Activation Functions by Andrew Ng - for  more formal and scientific answer
How does neural network classifier classify from just drawing a decision plane?
Differentiable activation function
A visual proof that neural nets can compute any function


Using unsupervised learning for classification problems Let's say there are two types of cancer(Type 1 and Type 2). Say we want to see if one of pour friends has cancer Type 1 or 2. We can treat this as a classification problem. But what if we use unsupervised learning (clustering) to separate the data into to 2 different groups and see each whether each item in group 1 belongs to a person with cancer Type 1 or 2. We will then  see whether our friend belongs to group 1 or 2. I know it is stupid to do this and we have to do extra work but can we even do this?

Let's say that the features are only the age and the height (I know it's really dumb but just bear with me). The data associated with people with cancer Type 1 is [10, 150], [12, 153], [9, 143], [13, 160] and for people with Type 2 cancer : [20, 175], [23, 180], [19, 174]. Let's say we plot the data on a graph (without labelling the data) and the unsupervised program (Clustering) just separates the two groups (Say group 1 for Type 1). We then can see that to whom each data in group 1 belongs. We see those people have cancer Type 1. So given new data, we see what group our friend  belongs to. If she/he belonged to group 1, he's got cancer Type 1 and if not, she/he has cancer Type 2.

You cannot use clustering as an "unsupervised classifier", with the goal of targeting specific known classes that are already defined. For that you need supervised learning and a labelled dataset.

A clustering algorithm would only separate into clusters as you hope if the feature variables you use are already:

All strongly associated with the trait you want the clusters to find
Carefully scaled to match the expected relationship

Now this might happen by chance with an arbitrary dataset. However in practice in a complex dataset with many features (as you might use in cancer classification) you would need to select and scale just relevant features. Thus you would need to already know the relationship between features and class from some other model.

The main task of clustering is to discover naturally occurring groups within your data, that might give you insight into the distributions within a dataset. This might let you compress or summarise information about examples. The summary feature might even be predictive, but it would very rarely overlap with pre-defined classes, even when the full feature vector could be used in an accurate supervised learning predictive model.

With your given example of [age, height] you might discover that the population splits naturally into people who are still growing, fully-grown youths and mature people. If this happens to overlap with predictions of type 1 and type 2 cancer that might be interesting - perhaps a person's cluster is even a good predictive variable for having type 1 or type 2 cancer. However, whatever the features are, it is highly unlikely that it would be so strong a predictive variable that you would use it on its own to make a classification. The only variables that would be that predictive would be the clinical measurements of the cancer growth that clinicians already use to decide the cancer type.

Potentially you could build a Bayesian model around the clusters, where you can ask questions such as:

What is the probability that a random person is in cluster A $p(x \in A)$?
What is the probability that a random person has cancer type 1 $p(x \in C_1)$?
What is the conditional probability that a person with type 1 cancer is found in cluster A $p(x \in A|x \in C_1)$?

Bayes' Theorem would then allow you to calculate the probability of having type 1 cancer based on their clustering:

$$p(x \in C_1 | x \in A) = \frac{p(x \in A|x \in C_1)}{(x \in A)}p(x \in C_1)$$

To do this fully with Bayesian statistics would involve setting some prior beliefs about the probabilities and using the data from your analysis to adjust them, which is beyond the scope of this answer. However, you could as a crude shortcut just take the measured frequencies, and that might be fine as a guide and give you some intuitions about what your clustering is doing towards helping with predictions.


Do I need LSTM units everywhere in the network? I have recently begun researching LSTM networks, as I have finished my GA and am looking to progress to something more difficult. I believe I am using the classic LSTM (if that makes any sense) and have a few questions.

Do I need LSTM units everywhere in the network? For example, can I only use LSTM units for the first and last layer and use feedforward units everywhere else?

How do I go about implementing bias values into an LSTM?

Assuming I create a network that predicts the next few words of a sentence, does that mean my outputs should be every possible word that the network could conceivably use?

For question 1) I dont understand what you are getting at... LSTM cells will work on a contiguous block of inputs, where it sequentially uses the states from a previous time step and the new input to generate the next ones.

question 2) Please look into the LSTM atchitecture
As you see, biases are already there, is there somewhere specific you want it, that it isnt?

question 3) Generally yes, but the normalization step can be expensive (such as softmax), so if you want to get clever, you can use negative sampling or hierarchial softmax-- but generally, you you predict a probability over all possible words given the previous text


Improving DQN with fluctuations Hello :) I'm pretty new to this community, so let me know if I posted anything incorrectly and I'll try to change it.
I'm working on the project which aim is to create self-driving agent in CARLA. I built a neural network Xception (decaying  ε-greedy). The other parameters are:

EPISODES: 100
GAMMA: 0.3
EPSILON_DECAY: 0.9
MIN_EPSILON: 0.001
BATCH: 16

Due to the limited computer resources I chose 100 or 300 epochs to train the model, but it generates much fluctuations:

EPISODES: 100
GAMMA: 0.7
EPSILON_DECAY: 0.9
MIN_EPSILON: 0.001
BATCH: 16

Can anyone suggest how can I improve my results? Or it is only the issue of small number of epochs?

It is not clear form your question, how you use your replay buffer. Basically, you have to store all states/reward tuples and train your agent on the entire buffer.
Moreover, you should give the agent time to explore (all) states of the world. But if you want to speed up training, you can try to implement importance sampling


What exactly does "channel" refer to in tensorflow/lucid? Tensorflow/Lucid is able to visualize what a "channel" of a layer of a neural network (image recognition, Inception-v1) responds to. Even after studying the tutorial, the source code, the three research papers on lucid and comments by the authors on Hacker News, I'm still not clear on how "channels" are supposed to be defined and individuated. Can somebody shed some light on this? Thank you.

https://github.com/tensorflow/lucid
https://news.ycombinator.com/item?id=15649456

The channel they are talking about is the depth of the layer L.

In this image, the channel is 5. There are 5 $3*2$ filters

They optimize an image from noise, to better respond to the filter F, in the layer L. Then you obtain this kind of image, and you can try to interpret the prupose of that filter (it learns to detect eyes, faces, wheels, and so on)

EDIT: To be more precise, you visualise the channel only if you optimise images for every filters, otherwise, you obtain a filter visualisation (if you do the optimisation one time, for one filter)

They call it "channel", because in a colored image, you have 3 dimensions : witdh, height, and channel (for color), as layers which have also 3 dimensions.


Why do we use the softmax instead of no activation function? Why do we use the softmax activation function on the last layer?
Suppose $i$ is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take $i$th value, it would be the highest value because $e$ is an increasing function, so that's why I am asking this question. Taking argmax(vec) and argmax(softmax(vec)) would give us the same value.

Short answer: Generally, you don't need to do softmax if you don't need probabilities. And using raw logits leads to more numerically stable code.
Long answer: First of all, the inputs of the softmax layer are called logits.
During evaluation, if you are only interested in the highest-probability class, then you can do argmax(vec) on the logits. If you want probability distribution over classes, then you'll need to exponentiate and normalize to 1 - that's what softmax does.
During training, you'd need to have a loss function to optimize. Your training data contains true classes, so you have your target probability distribution $p_i$, which is 1 at your true class and 0 at all other classes. You train the network to produce a probability distribution $q_i$ as an output. It should be as close to the target distribution $p_i$ as possible. The "distance" measure between two probability distribution is called cross-entropy:
$$ H = - \sum p_i \log q_i $$
As you can see, you only need logs of the output probabilities - so the logits will suffice to compute the loss. For example, the keras standard CategoricalCrossentropy loss can be configured to compute it from_logits and it mentions that:

Using from_logits=True is more numerically stable.


Why did Distributional Q Learning go out of popularity? I read some papers (for example, this) and blogs that spoke about the advantages of distributional Q learning. However, it no longer seems to come up in literature. Did it have any shortcomings that led to its failure? If yes, can someone can talk about it here?

Actually, distributional RL is a well-studied field in deep RL. Generally speaking, distributional RL needs more computing sources (1.1X), because of the quantile head. We can also find new distributional RL literature in NeurIPS2020 :).


Is there an AI that can extract proper nouns from free text? I have some free text (think: blog articles, interview transcripts, chat comments), and would like to explore the text data by analysing the proper nouns it contains.
I know of many ways to simply look up the text against a 'list' of proper nouns. The problem with the approach is many false positives and false negatives, as well as inaccuracies where one proper noun (e.g. "John Allen") is identified as two proper nouns ("John" and "Allen"), as well as other problems, mostly to do with long or unusual proper nouns (e.g. "the Gulf of Carpentaria" - a single proper noun containing the word "of", and long names like "Joost van der Westhuizen"). These kinds of longer, non-conformist proper nouns tend to really trip up grep-style proper noun identification models.
Does anyone know if any AI available to the public can more accurately identify proper nouns in free text?

This is a hard problem, unless you have a list of proper nouns you want to recognise. If John Allen is in this list, then you can easily use a longest match to prefer it over John or Allen. The same applies to the other examples you give.
Capitalisation on its own is not very reliable, as words at the beginnings of sentences are also capitalised, and sometimes technical terms or emphasis are expressed By Capitalising Them In Mid-Sentence. It's not what I would do, but you really have to expect anything in free text.
You could go some way by looking for sequences of proper nouns with of, de or van etc between them. There is probably a reasobaly small list of those connectors.
You can set up a grammar to capture complex names, but by far the most reliable way is to have a list. Unfortunately there is no easy solution for this. I would approach it iteratively, ie process a segment of text, tidy up the results, generate a list from them, and repeat with more text. Everything that is in the list you accept, other candidates you vet before adding them to the list. Eventually you should get fewer candidates that are not in the list.
You can probably kick-start your list with a gazetteer, ie a list of names and places. There are several of those on the web.
In general this procedure is referred to as Named Entity Recognition (NER). There are many ways to solve it, and my suggestion here is a pragmatic approach, which works reasonably well with not too much effort.


How to build a DQN agent which can be trained through interactive learning? I am trying to create a chatbot whose dialogue policy model will be trained through reinforcement learning. Dialogue Policy is responsible for selecting the action to take based on the given state of the conversation.

All implementations I see for RL are trained from an environment taken from Gym or created manually. These environments provide the next state, rewards etc to the model based on which it is trained.

Since I am creating a dialogue policy model which will be trained through real user conversations, I cannot provide a "pre-defined" environment which can provide the states and rewards. I am planning to train it myself by talking to it and providing rewards and next state (which I think is called interactive learning).

But I was not able to find any implementations, tutorials or articles on interactive learning. I am not able to figure out how to create such a model, how to take care of the episodes, sessions etc. This will be a continuous learning that will go on for months maybe. I have to save the model each day and continue training the next day by loading the model from that same state.

Can anyone guide me in the correct direction on how to approach this? Any githubs links, articles, tutorials of such implementations will be highly appreciated. I am aware this question seems too broad, but some hints will be very helpful for a newbie like me.

I suggest you to start reading this one on BERT and this one on GPT-2.

  I am aware this question seems too broad, but some hints will be very helpful for a newbie like me.

I'm not sure you want to create your chatbot using RL architecture at all. But if you want to implement such an idea the right way to go with that is by using iterative approach and starting from a really simple base:

Define a list of actions for your agent. Might be 10-words
vocabulary with simple one-word answers.
Define you simple environment. It might have 20 different states (e.g. greetings/conversation starters hey, hello)
Map good responses (hello -> hi) to positive rewards and -1 for the rest of them.
If you want to move forward with DQN try a simple Q-learning algorithm on your simple environment and train agent to properly response to those.
Once you have done the above you can make your vocabulary bigger
Now replace your table of input words which gives you a state for your network with NN/word2vec/any other NLP approach which will convert your input sentence to a state.
Manually provide a reward as a response for what you agent did according to learned policy.
Do that infinite amount of time and your agent probably will figure out how to respond properly.

NOTE: DQN uses discrete action space. So in such a case you will have only limited or REALLY HUGE actions space. It's all possible combinations between all possible words in you vocabulary (assuming action maps to some sentence)