What is the optimal exploration-exploitation trade-off in Q*bert? I am training an RL agent with Deep Q-learning + Experience Replay on the Q*bert Atari environment. After 400,000 frames, my agent appears to have learned strategic information about the game, but none about the environment. It has learned that a good immediate strategy is to simply jump down both diagonals and fall of the board, thus completing a large portion of the first level. However, it remains to understand neither the boundaries of the board to prevent jumping off, nor anything about avoiding enemies. I’m asking this here, instead of Stack Overflow because it is a more general question with less of a need in terms of programming understanding. Simply, I am asking whether or not this is a matter of a pore exploration policy (which I presume). If you agree, what should be a better exploration policy for Q*bert that would facilitate my agent’s learning experience?
As per the request of a comment:

Could you add what your current exploration approach is, and what options you are using for your Deep Q Learning implementation (e.g. replay size, batch size, NN architecture, steps per target network copy, or if you are using a different update mechanism for the target network). Also if you are using any other approach different to the classic DQN paper such as in state representation.

Here are my parameters:

Exploration policy: epsilon = min(1.0, 1000 / (frames + 1))
Replay Memory = 20,000 frames
Batch size = 32 transitions
NN architecture: Conv2D(64, 3, 2), Dropout(0.2), Dense(32, relu), Dense(32, relu), Dense(num_actions, linear)
Steps per target network copy: 100

I can spot three, maybe four, things in your implementation that could be contributing to incomplete learning that you are observing.
More exploration in long term
I think you have correctly identified that exploration could be an issue. In off-policy learning (which Q-learning is an instance of), it is usual to set a minimum exploration rate. It is a hyperparameter that you need to manage. Set too high, the agent will never experience the best rewards as it will make too many mistakes. Set too low, the agent will not explore enough to find the correct alternative actions when the opportunity to learn them occurs.
I would suggest for you something like:
epsilon = max(min(1.0, 1000 / (frames + 1)), 0.01)

You can choose numbers other than 0.01, but I think that is a reasonable start for many Atari games. You could try higher, up to 0.1 in games which are more forgiving of mistakes.
Remove dropout
I am not sure why, but I always have problems with dropout in RL neural networks. Try removing the dropout layer.
More convolutional layers
Convolutional layers are very efficient generalisers for vision and grid-based problems. You won't really benefit much from having a single layer though. I would add another two, increase the number of output channels.
Maybe state representation?
It is not clear from your description whether you are using a single colour frame for the state representation, or stacked greyscale frames for the last 3 inputs. It should be the latter, and if you want to more closely replicate the orginal DQN Atari paper, you should take the previous 4 frames as input.
In addition, you should be normalising the input into range $[0,1]$ or $[-1,1]$. The native image range $[0,255]$ is tricky for neural networks to process, and quite common for value functions to get stuck if you don't normalise.


How to construct Transformers to predict multidimensional time series? There is plenty of information describing Transformers in a lot of detail how to use them for NLP tasks. Transformers can be applied for time series forecasting.  See for example "Adversarial Sparse Transformer for Time Series Forecasting" by Wu et al.
For understanding it is best to replicate everything according to already existing examples. There is a very nice example for LSTM with flights dataset https://stackabuse.com/time-series-prediction-using-lstm-with-pytorch-in-python/.
I guess I would like to know how to implement transformers for at first univariate (flight dataset) and later for multivariate time series data. What should be removed from the Transformer architecture to form a model that would predict time series?

There is an implementation of the paper ("Adversarial Sparse Transformer for Time Series Forecasting"), in Python using Pytorch, here. Although it has the training and evaluation functionality implemented, it appears to be lacking a function for running a prediction.  Maybe you can fork it and extend it.
UPDATE
There is also a paper, "Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting", by Zhou et al., which does forecasts on univariate and multivariate data.  Their code is here.


What is the optimal value function of the shifted version of the reward function? Similarly to this question that I asked some time ago, what is the optimal value function of the shifted (by some constant $c$) version of some reward function? More precisely, let's assume that $r(s, a)$ is our original reward function and $q_*(s, a)$ the corresponding optimal state-action value function. What would be the optimal state-action value function of the new reward function $r'(s, a) \triangleq r(s, a) + c$, where $c \in \mathbb{R}$?

Theorem
The optimal state-action value function of $r'(s, a) \triangleq r(s, a) + c$, for $c \in \mathbb{R}$, would be
\begin{align}
q_*(s, a) + c + c\gamma + c \gamma^2 + c \gamma^3 + \dots
&=q_*(s, a) + c \left( 1 + \gamma + \gamma^2 + \gamma^3 + \dots \right) \\
&= q_*(s, a) + c \left( \sum_{k=0}^{\infty} \gamma^{k} \right) \\
&=q_*(s, a) + c\left(\frac{1}{1 - \gamma}\right) \\
&=q_*(s, a) + \frac{c}{1 - \gamma},
\end{align}
where $\gamma < 1$ is the discount factor (and $\sum_{k=0}^{\infty} \gamma^{k}$ is a geometric series).
Therefore, the optimal policy does not change if we shift the reward function by some constant.
Proof
The proof follows the same reasoning of the proof in this answer, i.e. we show that the Bellman optimality equation also holds in this case.
\begin{align}
q_*(s,a) + \frac{c}{1 - \gamma}
&= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left((r + c) + \gamma \max_{a' \in\mathcal{A}(s')} \left( q_*(s',a') + \frac{c}{1 - \gamma}  \right) \right) \tag{1}\label{1}
\end{align}
Given that $\frac{c}{1 - \gamma}$ is a constant, it does not affect the max, given that we add this constant to all pairs $(s', a')$, for all $a' \in\mathcal{A}(s')$. This holds even if $c$ is negative. So, we can take that addition out of the max, but note that we sum (rather than multiply, like the proof in the other answer).
\begin{align}
q_*(s,a) + \frac{c}{1 - \gamma}
&= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left((r + c) + \gamma \left (\frac{c}{1 - \gamma} +  \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \right) \\
&= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left((r + c) +  \frac{c \gamma}{1 - \gamma} +  \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \\
&= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left(r + \frac{c(1 - \gamma) + c \gamma}{1 - \gamma} +  \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \\
&= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left(r + \frac{c - c\gamma + c \gamma}{1 - \gamma} +  \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \\
&= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}} \left ( p(s',r \mid s,a)\frac{c}{1 - \gamma} \right) + \\ & \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}  \left( p(s',r \mid s,a) \left(r + \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \right) \tag{2}\label{2} \\
\end{align}
Given that $p(s',r \mid s,a)$ is a probability distribution, then the affine combination of $\frac{c}{1 - \gamma}$, where the affine weights are $p(s',r \mid s,a)$, is $\frac{c}{1 - \gamma}$. To see that this is true, suppose that we have the probability vector $[0.2, 0.5, 0.3]$ and let $c$ be a constant, then $0.2c + 0.5c + 0.3c = c$.
So, equation \ref{2} becomes
\begin{align}
q_*(s,a) + \frac{c}{1 - \gamma}
&= \frac{c}{1 - \gamma}  + \sum_{s' \in \mathcal{S}, r \in \mathcal{R}} p(s',r \mid s,a) \left(r + \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right)\\
q_*(s,a)
&=\sum_{s' \in \mathcal{S}, r \in \mathcal{R}} p(s',r \mid s,a) \left(r + \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right)
\end{align}
which is the Bellman optimality equation (see equation 3.20 of Sutton & Barto book, p. 64).


What are some common heuristics that might be innate? Here's a question I might ask an AI to solve:

"Colour the states of the USA using just 4 colours".

Now, a common heuristic a human might use is to start at one state and "work their way out". Or start at an edge state. Now this seems to work best rather than just colouring states in a random order like a computer might do. And it means a human is often better than a computer because a computer might just start colouring random states and get into trouble very quickly.

(Also I wonder if this is a learned heuristic or would a child develop this on his/her own?)

Now the question is, whether this heuristic is an innate optimisation strategy, or just laziness on the part of the human. i.e. colouring things close together takes less effort. Either way it leads to a good strategy.

But I wonder if there are any other examples of heuristics that humans inately use without realising it, that lead to good strategies?

One heuristic that computers often don't know is

"If you're trying to play a game don't keep turn around and go the other way for no reason."

Again, a human would not do this, but again this could be laziness on the part of a human. It takes more effort to turn around than keep going in one direction to explore it.

Perhaps Occam's razor counts.

Occam's razor is the meta-heuristic that "the simplest explanation is the most likely to be correct". I consider it a meta-heuristic because itself doesn't provide explanations, only means of comparing them among another. It is a heuristic because like all heuristics, it can lead to false conclusions, but is usually right.


Can a large discrete action space be represented using Gaussian distributions? I have a large 1D action space, e.g. dim(A)=2000-10000. Can I use continuous action space where I could learn the mean and std of the Gaussian distributions that I would use to sample action from and round the value to the nearest integer? If yes, can I extend this idea to multi-dimensional large action space?

The answer is "it depends". Once you have arranged the actions into order, a key trait is whether the action value function has a simple enough shape that sampling from a Gaussian policy function would give consistent expected returns, enough that learning can occur. If the underlying "true" value function has a lot of high frequency noise then learning would be slow. In the worst case, if the action value $Q(s,a_{n})$ and $Q(s,a_{n+1})$ is not correlated for any $n$, then it will not be possible to learn with the approximation at all.
You may have some sense of how similar actions $a_{n}$ and $a_{n+1}$ are. If the actions represent different ordinal choices, such as selecting an integer number of items to perform some task with such as buy/sell or transport, then in many environments there will often be a strong correlation between outcome of choosing e.g. $a_{900}$ and $a_{901}$. If this holds in general, then that is a good indicator that you can treat $n$ as being continuous, and use learned parameters of a simple distribution function to find optimal policies (and in addition this could be far more efficient than using a discrete representation).
It might not matter if for a small fraction of cases the difference in outcomes between $a_{n}$ and $a_{n+1}$ is large, provided that successive approximations to the optimal policy can improve towards optimal through adjusting mean $\mu(s)$ and standard deviation $\sigma(s)$.
There may still be difficult cases that cannot be learned by the approximation - for instance if a specific action $a_n$ is optimal for a given state $s$, but $a_{n-1}$ and $a_{n+1}$ are a lot worse, then the training process of a typical policy gradient approach may never settle upon $\mu(s) = n$ and $\sigma(s) \approx 0$, because any intermediate values between the starting policy and the optimal one will perform badly.
For expanding into more dimensions, the same ideas apply to each dimension separately. You may want to use different distributions, or even have one dimension that uses a continuous model with a few parameters whilst another remains discrete with a free parameter for each choice.


Are there architectures to generate pictures from four labels? (VAEs, GANs) I want to try something with image creation via NNs. I have come across Variational Autoencoders and Generative Adversarial Networks as possible solutions but have only found image creatinon with CIFAR-100 and GANs (two class-labels, non continuous)
Im looking for an idea for generating a picture with four continuous labels (Age and Rotation around X-, Y-, Z-axis). Is there anything usable for this?
Im especially looking for an VAE-Model on this task.

This is quite a difficult task and is still an open area of research. The idea behind the GANs is to map latent features (e.g. rotation, age) to an output image, but the problem is that the source data comes from an unknown probability distribution. GANs aim to approximate it by sampling this latent feature vector from a simple known distribution (e.g. normal).
However, such a distribution usually won't fit the target distribution and hence the latent features become entangled as they will reproduce statistics of the real data (e.g. gender and beard)
There are several attempts to solve this problem. One of them is Conditional GAN. For example, with a conditional GAN it can be possible to translate a segmentation mask into an image (Pix2Pix). Another approach has been proposed in StyleGAN. The authors introduce an intermediate feature vector that the model learns to disentangle in an unsupervised manner. Each value in this intermediate represents a meaningful attribute of the resulting image, but it is still not possible to control the latent features in a supervised manner. That is, after training you should manually test each of them to identify its relations.


What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative? I created an OpenAI Gym environment, and I would like to check the performance of the agent from OpenAI Baselines DQN approach  on it.
In my environment, the best possible outcome for the agent is 0 - the robot needs zero non-necessary resources to complete a task. The goal is to minimize the need for resources: for each needed resource, there is a penalty of -1. In many states, only certain actions make physical sense. How do I deal with this?
There was already a question about the handling of invalid moves on AI StackExchange, recommending to ignore the invalid moves. However, ignoring them would imply returning the same state and a 0 reward, the best possible outcome, which is clearly not the case. Setting drastic negative rewards also does not seem to work, since even promising handling paths are compromised by invalid actions and the corresponding drastic negative reward.
What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best) or negative?
My ideas/ questions on this for the OpenAI Baselines DQN approach implementation

Is there any way to set the initial Q-values for the actions? I could set -infinity for the invalid actions.

Is there any way to limit the set of valid actions per state? When after the env.step(action) function the new state is returned, can I somehow define which actions are valid for it?

1) Is there any way to set the initial Q-values for the actions?
You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network weights directly, at least. That would defeat the purpose of using backpropagation to optimize the weights and find the optimal parameters and Q-values.
2) Is there any way to limit the set of valid actions per state?
Again, not directly through the network. That would imply different network architectures for different states, which would imply optimizing several different networks. Unless you used some convoluted approach like Net2Net to keep the networks synced.
However, you can bypass the network itself.
You can, however, on top of the network, filter its outputs. Let's say you have a set of states, and in state X, only 2 out of all 5 actions are valid. When getting the Q-values for all the actions in DQN, you can detect if you're in state X, and instead of choosing the greedy action out of all actions, choose the greedy action out of all valid actions. Better yet, have your environment just send a list of all valid actions for whatever state you're in. Makes this more general, as it removes your need for discrete states, having lists of valid actions for each state, etc.
You can also really just ignore invalid actions.
You claimed

However, ignoring them would imply returning the same state and a 0 reward, the best possible outcome, which is clearly not the case.

If your environment has no time limit, then the best action really is to stand still and avoid wasting resources. However, if there is a time-limit, eventually the agent should receive some sort of penalty for not completing the task. Otherwise, if there is no time limit, you can also give him an award for completing the task. Imagining you have 10 resources, instead of having your reward interval be from [-10; 0], shift it to [0, 10] by awarding 10 points at the end of the game. This gives your agent the motivation to complete the task instead of doing useless actions, and discards your need for action-filtering at all.


How should I encode the structure of a neural network into a genome? For a deterministic problem space, I need to find a neural network with the optimal node and link structure. I want to use a genetic algorithm to simulate many neural networks to find the best network structure for the problem domain.

I've never used genetic algorithms for a task like this before. What are the practical considerations? Specifically, how should I encode the structure of a neural network into a genome?

Section 4.2 of "Essentials of Metaheuristics" has a wealth of information on alternative ways of encoding graph structures via Genetic Algorithms.

With particular regard to evolving ANNs, I would personally not be inclined to implement this sort of thing 'from scratch':

The field of neuroevolution has been around for some time, and the implementation some of the methods, such as Neuroevolution of Augmenting Topologies (NEAT) now incorporate the results of much practical experience.

According to the above link:

  We also developed an extension to NEAT called HyperNEAT that can evolve neural networks with millions of connections and exploit geometric regularities in the task domain. The HyperNEAT Page includes links to publications and a general explanation of the approach.


At which point we have to stop post pruning in decision tree? Post pruning is start from downward discarding subtree and include leaf node performance. so what is the best point or condition of the tree where we have to stop further pruning.

There are a variety of conditions we can use when deciding whether to prune a sub-tree or not after generating a decision tree model. There are three common approaches.

We can prune branches with less support than a specific threshold. These are branches which were constructed using very few points from the training data.
We can prune branches where the information gain from a split (or any other splitting measure we are using, like GINI), is smaller than a threshold.
You can do what is done in Quinlan's 4.5 & C5.0 learners (which are the standard approaches; J48 is another implementation of the same algorithm). Quinlan performs a Chi-squared-like test for the relationship between the attribute we split upon and the target attribute. If the relationship is statistically significant, then the split is preserved. If not, it is not. The "confidence factor" parameter found in most implementations of these algorithms corresponds to the $\alpha$ value used in determining whether the relationship is considered significant. This approach captures the idea that we should prefer to keep branches that have few datapoints, but a very strong signal, or that have a weak signal, but very many datapoints supporting the pattern, since both cases are less likely to be overfitting that cases where we have weak signals and small numbers of points.


What is an end-to-end AI project? I often read about the so-called end-to-end AI (or analytics) projects, but I couldn't find a definition of it. What is an end-to-end AI project?  Can someone explain what is meant/expected when someone asks you "Have you already implemented an end-to-end AI project"?

It's a pretty grey area.

I think they are usually talking about the full spectrum of the effort in taking an "AI project" all the way from the drawing board to production. This could easily be anything from a Stock Market Price Prediction model, to deploying a customer service chatbot/Virtual Assistant. I find that people who ask such questions only care about the fact that "AI" is being used to solve a problem and actually is working in the real world.

The effort in the Model would be the standard DataScience/Machine learning process of obtaining data, feature engineering, training and refining your model and eventually deploying it somewhere where it works. It could be in a web browser, it could be in an app, an API endpoint, etc.

For the customer service chatbot it could easily be largely a pure programming project utilising a cloud provider's resource for to conversational capabilities, but as long as it falls under the bracket of "AI" and has been completed and deployed to production/an environment where it works, its end to end.

Disclaimer: Depending on the technical knowledge of who is asking, they might care that you have done some real machine learning work in the development process, and so the example of the bot wouldn't fly.

Hope this helps somehow.


Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning? Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning?
I could not find a clear answer on this. I would say yes it is Multi Agent Learning, as there are two Agents playing against each other.

Depends on perspective.
On one hand, you have an agent playing in an environment with another agent also evolving. This falls under the definition of Multi-Agent Learning, as can be seen with works such as

Michael Bowling and Manuela Veloso. Multiagent learning using a variable learning
rate. Artificial Intelligence, 136(2):215 – 250, 2002.

Michael Bowling. Convergence and no-regret in multiagent learning. In Proceedings of
the 17th International Conference on Neural Information Processing Systems, NIPS’04,
pages 209–216, Cambridge, MA, USA, 2004. MIT Press.

M. D. Awheda and H. M. Schwartz. Exponential moving average q-learning algorithm. In
2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning
(ADPRL), pages 31–38, April 2013.

Sherief Abdallah and Victor Lesser. A multiagent reinforcement learning algorithm with
non-linear dynamics. Journal of Artificial Intelligence Research, 33:521–549, 2008.

However, you can also claim that you simply have a single agent learning on a non-stationary environment (the environment contains both the game rules and the opponent), and you simply learn on that basis. From this perspective, there is no multi-agent learning at all.


Why are the initial weights of neural networks randomly initialised? This might sound silly to someone who has plenty of experience with neural networks but it bothers me...
Random initial weights might give you better results that would be somewhat closer to what a trained neural network should look like, but it might as well be the exact opposite of what it should be, while 0.5, or some other average for the range of reasonable weights' values, would sound like a good default setting.
Why are the initial weights of neural networks randomly initialized rather than being all set to, for example, 0.5?

You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue.

http://www.deeplearningbook.org/contents/optimization.html
Perhaps the only property known with complete certainty is that the initial
parameters need to “break symmetry” between different units. If two hidden
units with the same activation function are connected to the same inputs, then
these units must have different initial parameters. If they have the same initial
parameters, then a deterministic learning algorithm applied to a deterministic cost
and model will constantly update both of these units in the same way. Even if the
model or training algorithm is capable of using stochasticity to compute different
updates for different units (for example, if one trains with dropout), it is usually
best to initialize each unit to compute a different function from all of the other
units. This may help to make sure that no input patterns are lost in the null
space of forward propagation and no gradient patterns are lost in the null space
of back-propagation.


Did Alphago zero actually beat Alphago 100 games to 0? tl;dr
Did AlphaGo and AlphaGo play 100 repetitions of the same sequence of boards, or were there 100 different games?
Background:
Alphago was the first superhuman go player, but it had human tuning and training.
AlphaGo zero learned to be more superhuman than superhuman.  Its supremacy was shown by how it beat AlphaGo perfectly in 100 games.
My understanding of AlphaGo and AlphaGo are that they are deterministic, not stochastic.
If they are deterministic, then given a board position they will always make the same move.
The way that mathematicians count the possible games in chess is to account for different board positions.  As I understand it, and I could be wrong, if they have the exact same sequence of board positions then it does not count as a different game.
If they make the same sequence of moves 100 times, then they did not play 100 different games, but played one game for 100 repetitions.
Question:
So, using the mathematical definition, did AlphaGo and AlphaGo Zero play only one game for 100 iterations or did they play 100 different games?
References:

https://www.scientificamerican.com/article/ai-versus-ai-self-taught-alphago-zero-vanquishes-its-predecessor/
https://deepmind.com/blog/article/alphago-zero-starting-scratch
https://mathworld.wolfram.com/Stochastic.html

Did AlphaGo and AlphaGo [Zero] play 100 repetitions of the same sequence of boards, or were there 100 different games?

There were 100 different games. You can view some example games between AlphaGo [Lee] and AlphaGo Zero here. They are clearly all different.
This statement in the question shows a misunderstanding:

My understanding of AlphaGo and AlphaGo [Zero] are that they are deterministic, not stochastic.

The Monte Carlo Tree Search (MCTS) algorithm used for look-ahead planning in AlphaGo and Alpha Zero is inherently stochastic. It samples from the huge tree of possibilities in a game like Go by making weighted random choices at all branch points. That means play can progress stochastically with two such agents opposing each other, as many board states will resolve into selecting semi-randomly between "best" moves that would be very closely ranked by each agent in the limit of very long search times.
Whilst this solves the main point of your question, it is worth noting that there can be a related effect in self-play algorithms, even if they are partially stochastic. That is, it is possible to have one agent that develops a specific weakness by chance, that another agent consistently takes advantage of, such that agent A consistently beats agent B, and wins in a very similar fashion each time (maybe deterministically, maybe across a range of different games all with a similar mistake). However it may be the case that also:

Neither agent is strong in general.

Another agent C can beat B consistently, but will lose to A consistently. There would then be no clear way to rank agents A, B, and C without further measurements.

Agents trained through self play therefore do need to be trained and tested against a wide range of opponents to verify this is not happening and that the  skill level assessment is valid more generally. I believe this was done with all the AlphaGo variants built by DeepMind.
The MCTS algorithm does help a little with this scenario as it can correct for weaknesses in how a trained neural network rates early board positions. The look-ahead planning of MCTS makes initial ratings less relevant to eventual action selection - effectively it refines those learned ratings using the samples from current position.


How to derive matrix form of the Bellman operators? Reading the Retrace paper (Safe and efficient off-policy reinforcement learning) I saw they often use a matrix form of the Bellman operators, for example as in the picture below. How do we derive those forms? Could you point me to some reference in which the matter is explained?
I am familiar with the tabular RL framework, but I'm having trouble understanding the steps from operators to this matrix form. For example, why does $Q^{\pi} = (I -\gamma P^{\pi})^{-1}r$? I know that for the value $V$ we can write
\begin{align}
V =  R + \gamma P^{\pi} V \\
V - \gamma P^{\pi} V =  R \\
V (I -\gamma P^{\pi}) =  R \\
V = R(I - \gamma P^{\pi})^{-1}
\end{align}
but this seems slightly different.
Picture from Safe and efficient off-policy reinforcement learning:

There's not much to derive here it's simply a definition of Bellman operator, it comes from Bellman equation. If you're wondering why
\begin{equation}
Q^{\pi} = (I - \gamma P^{\pi})^{-1}r \tag{1}
\end{equation}
they state that $Q^{\pi}$ is a fixed point which means if you apply Bellman operator to it you get the same value
\begin{equation}
T^{\pi}(Q^{\pi}) = Q^{\pi}
\end{equation}
You can easily check that since from $(1)$
\begin{equation}
r = (I-\gamma P^{\pi})Q^{\pi}
\end{equation}
if you plug it in definition of Bellman operator you get
\begin{align}
T^{\pi}(Q^{\pi}) &= r + \gamma P^{\pi} Q^{\pi}\\
&= (I - \gamma P^{\pi})Q^{\pi}+  \gamma P^{\pi} Q^{\pi}\\
&= Q^{\pi}
\end{align}


CAPTCHA based on text comprehension and random tokens I developed a novel type of CAPTCHA based on text comprehension and random tokens. Given a task Pick the first pair of adjacent letters and a random token 8NBA596V, the user has to provide the solution NB. It offers basic protection and an attacker can solve individual tasks with specific effort. I am curious, whether contemporary AI can solve it generically?
You can access more example tasks here:
https://www.topincs.com/manual/captcha
There is a task database and at every attempt a new task is presented with a new random token. They always have a solution of varying length and pure guessing thus has limited chances of success. It is easy to attack an individual task by writing a small piece of code, thus a large task database is essential. What intrigues me is the question whether natural language processing or machine learning at its current state can attack the CAPTCHA generically by building a model of the meaning of the task
– essentially a predicate in a tiny universe of discourse – and then applying it to the random token.

(Assuming English.) In your specific example, there would be $26^2$ combinations of capitalized letters. Also assuming a fixed-length token of eight, that gives you $26 * 26 * 7 = 4732$ possible combinations. My intuition is that the key space is too small.
It may actually be simpler for a machine to solve the CAPTCHA than that. Let's say that both your system and a generic one obscure text in an image with equal ability. The generic one is a string of length six composed of capital letters and numbers. So the attacker needs to make an informed guess from $(26 + 10)^6 = 2,176,782,336$ possibilities. That keyspace is $460,013$ times larger than your system with the longer string. It gets worse though.  An attacker could use a probabilistic approach, determining a "letter probability" for each of the eight positions, taking the two adjacent positions that maximize the probability, and then choosing the letter with the highest probability for each of those solutions. It would not be a guaranteed win for the attacker. But it would be guessing at a higher probability than $\frac{1}{4732}$.
Also, it may be easier for an ML system to distinguish certain letters from others. B/P may be confused more easily compared to M/T (I'm hypothesizing here). So the attacker's ML could be more confident in some letter pairs than others. This is analogous to the idea of weak keys in cryptography.
I like where your head is at. But this may not be the result you're looking for. In theory, you could have multiple rules beyond just "Pick the first pair of adjacent letters." But each new rule would add a negligible benefit on top of what is a fundamental problem in the design.


Why isn't my implementation of DQN using TensorFlow on the FrozenWorld environment working? I am trying to test DQN on FrozenWorld environment in gym using TensorFlow 2.x. The update rule is (off policy)
$$Q(s,a) \leftarrow Q(s,a)+\alpha (r+\gamma~ max_{a'}Q(s',a')-Q(s,a))$$

I am using an epsilon greedy policy.
In this environment, we get a reward only if we succeed. So I explored with 100% until I have 50 successes. Then I saved the data of failures and success in different bins. Then I sampled (with replacement) from these bins and used them to train the Q network. However, no matter how long I train the agent doesn't seem to learn.

The code is available in Colab. I am doing this for a couple of days.

PS: I modified the code for SARSA and Expected SARSA; nothing works.

I see at least 3 issues with your DQN code that need to be fixed:

You should not have separate replay memories for successes/failures. Put all of your experiences in one replay memory and sample from it uniformly.
Your replay memory is extremely small with only 2,000 samples. You need to make it significantly larger; try at least 100,000 up to 1,000,000 samples.
Your batch_target is incorrect. You need to train on returns and not just rewards. In your train function, compute the 1-step return $r + \gamma \cdot max_{a'} Q(s',a')$, remembering to set $max_{a'} Q(s',a') = 0$ if $s'$ is terminal, and then pass it to model.fit() as your prediction target.


Drone training, how to train without training data? I setupped a small drone simulator using PhysX, the time step is at 200 hz, while motors update like regular ESCs (at 50 Hz). I computed the inertia matrix, tweaked a bit mass of components to be real, air drag, gravity etc. After a first partial success in tuning a PID algorithm I got bored to find and hunt perfect values, and started thinking to tune it with a NN, but then I thinked, why using PID at all if NN can find better solutions?.

I started playing with NN, but then I realized I have no traning data.

Sure I could do

NN.Train(input,expectedOutput);

but what is actually the expected output? To be useful it has to be the thrust force of propellers tuned to keep input (position and orientation) stable in the desired place.

But actually I don't (Want to) know in advance the thrust that every single propeller has.

Since it is a simulation it is ok spending some computing time between each simulation step (eventually I'll try to optimize it later to fit it in a microcontroller)..

So, is it possible given a regular NN implementation where I can select number of:

Input neurons
Hidden neurons
Output neurons

find a way to train my model live? I need a way to tell the NN

  hei this time you performed X keep going or go the opposite way.

It's debatable whether neural networks can find a better solution than PID, if your goal is to simply keep the output around a certain reference point PID should do a perfect job pretty much. If you really want to use "intelligent" control with NN you can look into reinforcement learning.
Few interesting papers that directly adress your problem:

Autonomous helicopter flight via reinforcement learning
An Application of Reinforcement Learning to Aerobatic Helicopter Flight
Autonomous Helicopter Control Using Reinforcement Learning Policy Search Methods

You can also find more in the references of these papers.


AI to recognize keyword in a phrase within a context I am building a search engine and I am looking for an open source AI algorithm to recognize the keyword in a search phrase within a particular context. So if a user passes something in the line of how far is Russia? and the context is location then the AI should return Russia

Here is a glimpse for you.

I once contributed to an open source project via Github, and the open source artificial intelligence library/API which helped a lot and I think will as well help you out is wit, it provides location data.

And the good thing about "wit", is it provides wit_location entity which provides information on the captured location for instance "time-zone, locality, region / location"

For example: in sentence "How far is Russia?", it captures "far" as the distance to the target location entity before this returns

{"value":"location_name",Russia"}

In the entity value. (don't forget)

To further improve the quality and search engine racking to return values in future, that's if your project goes well, you can simply provide an optimal "coords" map in context when calling the "wit-API"

This will also help you to factor into your search engine racking of your returned values, which will as well take the location's proximity to the "coords" into  account.

Hope this can give you some insights.


What is a temporal feature? What is a temporal feature, what features make something temporal in nature? Is this problem agnostic? How does it change from different fields of study?

In general, the expression "temporal feature" might refer to any feature that is associated with or changes over time.
However, in the context of signal processing, a temporal feature might refer to any feature of the data before being transformed to the Fourier, frequency or spectral domain, using the Fourier transform. In this context, the domain of the untransformed data is often called "time domain" (as opposed to the "frequency" or "spectral" domain, which is the domain of the transformed data), even though it might not be strictly associated with or defined as a function of time. For example, in image processing, an image can be interpreted as a 2D signal. The domain of an image can be referred to as the "time domain", even though it is usually and more correctly referred to as the "spatial domain" (given that an image can be thought of as a function from a pixel, which is defined by two numbers $x$ and $y$, to a value, e.g. a grayscale value). You can transform this image, using the Fourier transform, to the spectral domain. In that case, the domain of the result of the transformation can be referred to as the "spectral domain".
In the paper Learning Temporal Features Using a Deep Neural Network and its Application to Music Genre Classification, the authors define spectral and temporal features

Extracting features from audio that are relevant to the task at hand is a very important step in many music information retrieval (MIR) applications, and the choice of features has a huge impact on the performance. For the past decades, numerous features have been introduced and successfully applied to many different kinds of MIR systems. These audio features can be broadly categorized into two groups: 1) spectral and 2) temporal features.
Spectral features (SFs) represent the spectral characteristics of music in a relatively short period of time. In a musical sense, it can be said to reveal the timbre or tonal characteristics of music. Some of popular SFs include: spectral centroid, spectral spread, spectral flux, spectral flatness measure, mel-frequency cepstral coefficients (MFCCs) and chroma.
On the other hand, temporal features (TFs) describe the relatively long-term dynamics of a music signal over time such as temporal transition or rhythmic characteristics. These include zero-crossing rate (ZCR), temporal envelope, tempo histogram, and so on.
The two groups are not mutually exclusive, however, and many MIR applications use a combination of many different features.

These definitions are given in the context of signal processing (as mentioned above).
To conclude, the meaning of "temporal feature" might change depending on the context. Hence, you should interpret it given the context, but it is almost always associated with time (in some way).


How and where can I start learning AI if I have considerable knowledge of Java & C++? I have deep interest in AI and want to start learning how to implement current method. I know about java and C++; are these languages sufficient?

I'd appreciate suggestions regarding free online courses/websites that utilize Java/C++. If I lag some knowledge which is required before starting AI then please let me know.

If you are coming from Java, it would make a lot of sense to play around with deeplearning4J at first. From there I would start learning python as this is the primary language used today in ML.

Lectures are a great way to get your feet wet in understanding and applying ML concepts.

My favorites are:

Stanford ML(Andrew NG): https://www.youtube.com/watch?v=UzxYlbK2c7E (covers classic algorithms in depth)

MIT AI(Patrick Winston): https://www.youtube.com/watch?v=TjZBTDzGeGg&t=2s (great starter course that covers basic concepts beautifully)

Deepmind RL(David Silver): https://www.youtube.com/watch?v=2pWv7GOvuf0&t=7s (Best RL course available)

Beyond that, start implementing these concepts. There are hundreds of tutorials on computer vision, natural language processing, and reinforcement learning.


How to scale all positive continuous reward? My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between 5 and 6, therefore achieving the optimum episodic reward will be harder.
What scaling methods are recommended? (like min-max scaling or reward ** 3)
How can I emphasize the episodic reward?

I’ll try to find where I found it, but normalizing the rewards has always worked for me. Assuming you have a list of the discounted returns for each action, you subtract the whole list by its average value then divide it by its standard deviation. In Python with NumPy, that would look like:
returns -= np.mean(returns)
returns /= np.std(returns)

This puts the returns in a small and consistent range that keeps learning similar with different rewards.


Can the decoder in a transformer model be parallelized like the encoder? Can the decoder in a transformer model be parallelized like the encoder?

As far as I understand, the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder, this is not possible (in both training and testing), as self-attention is calculated based on previous timestep outputs. Even if we consider some techniques, like teacher forcing, where we are concatenating expected output with obtained, this still has a sequential input from the previous timestep.

In this case, apart from the improvement in capturing long-term dependencies, is using a transformer-decoder better than say an LSTM, when comparing purely on the basis of parallelization?

Can the decoder in a transformer model be parallelized like the encoder?
The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual translation (or, in a wider sense, generating output sequences for new input sequences during a testing phase).
What exactly is parallelized?
Also, it's worth mentioning that "parallelization" in this case means to compute encoder or decoder states in paralllel for all positions of the input sequence. Parallelization over several layers is not possible: the first layer of a multi-layer encoder or decoder still needs to finish computing all positions in parallel before the second layer can start computing.
Why can the decoder be parallelized position-wise during training?
For each position in the input sequence, a Transformer decoder produces a decoder state as an output. (The decoder state is then used to eventually predict a token in the target sequence.)
In order to compute one decoder state for a particular position in the sequence of states, the network consumes as inputs: 1) the entire input sequence and 2) the target words that were generated previously.
During training, the target words generated previously are known, since they are taken from the target side of our parallel training data. This is the reason why computation can be factored over positions.
During inference (also called "testing", or "translation"), the target words previously generated are predicted by the model, and computing decoder states must be performed sequentially for this reason.
Comparison to RNN models
While Transformers can parallelize over input positions during training, an encoder-decoder model based on RNNs cannot parallelize positions. This means that Transformers are generally faster to train, while RNNs are faster for inference.
This observation leads to the nowadays common practice of training Transformer models and then using sequence-level distillation to learn an RNN model that mimicks the trained Transformer, for faster inference.


Edge AI device to run inference I'm trying to figure out what's the best way to get a trained model that does facial detection to run on a camera that's connected to a computer and perform inference.

Since I don't need the device to train the model, is there a low cost solution that can still do at least 10fps on a 1280x720 image?

Is there any device that can run the model and have camera and computer all built in a small device?

If not, any suggestion on what's the best way to put one together and deploy it in production environment?

Thanks!

Since you said detection, you could do this on OpenCV: HAAR Cascade Detection for faces. It's swift, and should fit your use case. I've seen such scripts running on small Tello drones and gets job done relatively well.

Definitely there is hardware built around for this purpose. Look for Nvidia Jetson Nano and Jetson TK1/TX1. It's got sufficient juice to take even 30fps IMO.


Is there an AI model for generating sarcastic sentences? I am currently working on an Android AI app.

I am aware of AI models to generate random sentences. However, is there an AI model for generating sarcastic sentences?

A simple form of sarcasm involves a direct reversal of the literal meaning of the statement, eg "Great weather we're having" (during a thunderstorm), "just what I needed" (when something goes wrong).

The problem with doing this in random sentences is that you may have no context to establish the reversal of the literal meaning.

You could possibly construct them by using a template along the lines of "Just what I needed - (random bad thing happened) today"

Or, when an outcome of a process is calculated, if it is not the desired outcome, instead of returning "mission unsuccessful" or "mission not yet complete", the AI could say "you're having a great day, aren't you? - mission unsuccessful" or "great work, genius - mission not yet complete".

Most random sentences will not be suitable for sarcasm, so it could only be applied in specific circumstances.

It is not clear from your question what the context is for these random sentences, and therefore it is not clear whether that context would be suitable for sarcasm at all.


How to properly resume training of deep Q-learning network? I'm currently training a deep q-learning network. Due to resource limitations, I am not able to train the model to the desired performance in one go. So what I'm doing now is training the model for a certain number of episodes and save the resultant model. Later, I will load up the previously saved model and resume training.
However, what I'm noticing is that when training resumes, the average rewards goes back to very low again, compared to what it achieved at the end of the previous training session. What I'm currently doing is to load up the previously saved model into the prediction and target models, and I keep all hyperparameters unchanged.

Is this behaviour expected?
If not, how do I properly resume training of a deep q-learning
network?
Do I start off with the epsilon value at the end of the
previous session, currently I reinitialize that as well?

Do I start off with the epsilon value at the end of the previous session, currently I reinitialize that as well?

You should probably re-start with $\epsilon$ at the value you left off at. Using high values of epsilon may cause the neural network to forget some of what it learned from close-to-optimal policies in favour of learning possibly useless values of states and actions that are not important to a more highly-trained agent.
Also, you should either save, or wait to refill experience replay memory before restarting training updates to the neural network. Working from a small memory may also cause the neural network to overfit to specific samples and generalise less well - at least temporarily until the memory fills up again.
From your description, I am assuming that you are monitoring the average reward per training episode. This is metric that is easy to collect, but that has a problem in off-policy RL. The problem is that you are seeing the results from your behaviour policy, not your learned policy. When using $\epsilon$-greedy in Q learning, with a high value of epsilon, then the behaviour policy is likely to perform badly.

Is this behavior expected?

I would expect it if you re-set $\epsilon$ to a high value.

If not, how do I properly resume training of a deep q-learning network?

I recommend that you look at the following things:

Re-start with $\epsilon$ at or close to where you left off. Perhaps allow starting epsilon to be passed in to the script as an argument.

Measure performance of the greedy policy at checkpoints - e.g. run 100 test episodes at the end of every 500 training episodes - and use plots of that to decide how well your agent is performing. You can still plot training performance, but it should not be your guide to how well the agent is learning.

Save experience replay so far at checkpoints too, and reload it on re-start. Alternatively, allow for experience replay to fill significantly before allowing update steps in the training loop.

You could consider these in priority order. Note that just the first one may appear to "fix" your problem, but that is because you are not yet properly measuring your agent's performance.


Is the QuickDraw with Google neural net a convolutional neural network? Does anyone know, or can we deduce or infer with high probability from its characteristics, whether the neural network used on this site

https://quickdraw.withgoogle.com/

is a type of convolutional neural network (CNN)?

I believe they don't use CNNs. The most important reason why it's because they have more information than a regular image: time. The input they receive is a sequence of (x,y,t) as you draw on the screen, which they refer as "ink". This gives them the construction of the image for free, which a CNN would have to deduce by itself.

They tried two approaches. Their currently most successful approach does the following:

Detect parts of the ink that are candidates of being a character
Use a FeedForward Neural Network to do character recognition on those candidates
Use beam search and a language model to find most the most likely combination of results that results into a word

Their second approach is using an LSTM (a type of Recurrent Neural Network) end-to-end. In their paper they say this was better in a couple languages.

Source: I was an intern in Google's handwriting team in summer 2015 (on which I believe quickdraw is based), but the techniques I explained can be found in this paper.


How to use computer vision to find corners of a soccer field based on location coordinates? I  want to use computer vision to allow my robot to detect the corners of a soccer field based on its current position. Matlab has a detectHarrisFeatures feature, but I believe it is only for 2D mapping.
The approach that I want to try is to collect the information of the lines (using line detection), store them in a histogram, and then see where the lines intersect based on their angles.
My questions are:

How do I know where the lines intersect?
How do I find the angles of the lines using computer vision?
How do I update this information based on my coordinates?

I am in the beginning stages of this task, so any guidance is much appreciated!

Finding lines in an image often leads to the Hough line transform. Many libraries implement it, including OpenCV. Getting the lines should answer subsequent questions (and it doesn't, please consider having one question per post, and some other sites on StackExchange may be better suited than AI.SE).

Alternative approaches based on Machine Learning may also exist. It may be interesting at this point to look into libraries that do human pose/gait recognition. Some library may be repurposed to recognize the "field gait", assuming the field outer mark is its "gait"-equivalent.


Dense-Sparse-Dense CNN training I want to implement DSD: Dense-Sparse-Dense training for deep neural networks by Han et al. In short, the paper suggest the following training scheme to improve the network accuracy:

Train as usual till convergence,
prune the network and train with sparsity constraint,
remove the sparsity constraint and let the pruned connections to recover.

My question is about step 2: train with sparsity constraint. The paper mentions training with a binary mask specifying the pruned weights to keep "untouched" so the sparsity constraint is satisfied, however that means implementing a dedicated layer that takes the binary mask as an additional blob and handles it accordingly.

I wonder a simpler approach will give the same result: what if after the pruning step I keep the location of the pruned weights and then use dense training, but after each iteration to zero the originally pruned weights?

The forward path is the same, taking zero weights for the pruned weights anyway.  But would it negatively affect the backward path - since the constraint isn't there, or is it equivalent to the formal training scheme?

Thanks.

Your approach will give a similar result.
The difference with using the constraint is that the constraint allows the network to essentially confirm that it doesn't need the connections before it reallocates them (i.e. if the weight is lessened by the constraint and it results in something not working, then the weight will start increasing).


Should I need to use BERT embeddings while tokenizing using BERT tokenizer? I am new to BERT and NLP and I am a little confused with tokenization and word embedding.
My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use BertEmbedding for generating its corresponding word vectors of the tokens or I can train my own word2vec model to generate my word embedding while using BertTokenizer?
Pardon me if this question doesn't make any sense.

word2vec and BERT are completely different things. You don't need to use BERT Tokenizer for training word2vec model. But, if you want to go ahead. Here's a link which can help you.


NLP Identifying important key words in a corpus I am intrigued with the idea of Zettelkasten but unsatisfied with the current implementations. It seems to me that a machine learning and NLP approach could be productive by helpfully identifying “important” keywords on which to links could be created, with learning to help narrow the selection of keywords over time.
My problem is that it’s been 30 years since AI classes in grad school and things have moved on. I’m sure I could become an nlp expert with study but I don’t wanna. So I’m looking for guidance: what are the right terms to describe identifying keywords in context, ideally with some semantic content; how would I apply ML with my training to improve the keyword identification.
I’d love references, ideas, and packages references. Python is preferred, but not strongly; I write most common (and many uncommon, SNOBOL and COBOL anyone?) languages so language isn’t all that much of an issue.

I am sure there are complex methods to extract keywords, but the standard one which should serve as a strong baseline is the RAKE graph algorithm https://pypi.org/project/rake-nltk/. It should work reasonably well in most text domains.


Implementing Gradient Descent Algorithm in Python, bit confused regarding equations I'm following the guide as outlined at this link: http://neuralnetworksanddeeplearning.com/chap2.html
For the purposes of this question, I've written a basic network 2 hidden layers, one with 2 neurons and one with one neuron. For a very basic task, the network will learn how to compute an OR logic gate so the training data will be:
X = [[0, 0], [0, 1], [1, 0], [1, 1]]
Y = [0, 1, 1, 1]

And the diagram:

For this example, the weights and biases are:
w = [[0.3, 0.4], [0.1]]
b = [[1, 1], [1]]

The feedforward part was pretty easy to implement so I don't think I need to post that here. The tutorial I've been following summarises calculating the errors and the gradient descent algorithm with the following equations:
For each training example $x$, compute the output error $\delta^{x, L}$ where $L =$ Final layer (Layer 1 in this case). $\delta^{x, L} = \nabla_aC_x \circ \sigma'(z^{x, L})$ where $\nabla_aC_x$ is the differential of the cost function (basic MSE) with respect to the Layer 1 activation output, and $\sigma'(z^{x, L})$ is the derivative of the sigmoid function of the Layer 1 output i.e. $\sigma(z^{x, L})(1-\sigma(z^{x, L}))$.
That's all good so far and I can calculate that quite straightforwardly. Now for $l = L-1, L-2, ...$, the error for each previous layer can be calculated as
$\delta^{x, l} = ((w^{l+1})^T \delta^{x, l+1}) \circ \sigma(z^{x, l})$
Which again, is pretty straight forward to implement.
Finally, to update the weights (and bias), the equations are for $l = L, L-1, ...$:
$w^l \rightarrow w^l - \frac{\eta}{m}\sum_x\delta^{x,l}(a^{x, l-1})^T$
$b^l \rightarrow b^l - \frac{\eta}{m}\sum_x\delta^{x,l}$
What I don't understand is how this works with vectors of different numbers of elements (I think the lack of vector notation here confuses me).
For example, Layer 1 has one neuron, so $\delta^{x, 1}$ will be a scalar value since it only outputs one value. However, $a^{x, 0}$ is a vector with two elements since layer 0 has two neurons. Which means that $\delta^{x, l}(a^{x, l-1})^T$ will be a vector even if I sum over all training samples $x$. What am I supposed to do here? Am I just supposed to sum the components of the vector as well?
Hopefully my question makes sense; I feel I'm very close to implementing this entirely and I'm just stuck here.
Thank you
[edit] Okay, so I realised that I've been misrepresenting the weights of the neurons and have corrected for that.
weights = [np.random.randn(y, x) for x, y in zip(sizes[:-1], sizes[1:])]

Which has the output
[array([[0.27660583, 1.00106314],
   [0.34017727, 0.74990392]])
array([[ 1.095244  , -0.22719165]])

Which means that layer0 has a weight matrix with shape 2x2 representing the 2 weights on neuron01 and the 2 weights on neuron02.
My understanding then is that $\delta^{x,l}$ has the same shape as the weights array because each weight gets updated indepedently. That's also fine.
But the bias term (according to the link I sourced) has 1 term for each neuron, which means layer 0 will has two bias terms (b00 and b01) and layer 1 has one bias term (b10).
However, to calculate the update for the bias terms, you sum the deltas over x i.e $\sum_x \delta^{x, l}$; if delta has the size of the weight matrix, then there are too many terms to update the bias terms. What have I missed here?
Many thanks

There seems to be a mismatch between the weights you provide and your network diagram. Since w[0] (the yellow connections) is meant to transform $ x \in \mathbb{R}^2 $ into the layer 0 activations which are $ \mathbb{R}^2 $, w[0] should be a matrix $ \in \mathbb{R}^{2 \times 2} $, not a vector in $\mathbb{R}^2 $ as you have. Likewise, your w[1] (the red connections) should be a vector $ \in \mathbb{R^2} $ and not a scalar. Finally, if you are indeed scaling the output of layer 1 (the blue connection), then you'll need an additional scalar value. However, the blue connection confuses me a bit as usually the activated output is used directly in the loss function, not a scaled version of it. Unless the blue connection stands for the loss function.
In short, I believe if you change the shapes of your weight matrices to actually represent your network diagram, your update equations will work. I'll go through the network below to make sure I illustrate my point.
$ x \in \mathbb{R}^{2} $, an input example
$ W^0 \in \mathbb{R}^{2 \times 2} $, the yellow connections
$ W^1 \in \mathbb{R}^2 $, the red connections
$ z^0 = xW^0 \in \mathbb{R}^{2} $, the weighted inputs to the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.
$ a^0 = \sigma(z^0) \in \mathbb{R}^{2} $, the output of the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.
$ z^1 = a^0 W^1 \in \mathbb{R} $, the weighted inputs to the layer 1 nodes. The dimensions of this should match the number of nodes at layer 1.
$ a^1 = \sigma(z^1) \in \mathbb{R} $, the output of the layer 1 nodes and thus the output of the network. The dimensions of this should match the number of nodes at layer 1.
Weight Updates
As you say before your edit, $\delta^1$, as the product of two scalars $\nabla_a C$ and $\sigma'(z^1)$, is also a scalar. Since $a^0$ is a vector in $\mathbb{R}^2$, then
$\delta^1(a^0)^T$ is also a vector in $\mathbb{R}^2$. This matches what we expect, as it should match the dimensions of $W^1$ to allow the element-wise subtraction in the weight update equation.
NB. It is not the case, as you say in your edit, that the shape of $\delta^l$ should match the shape of $W^l$. It should instead match the number of nodes, and it is the shape of $\delta^l(a^{l-1})^T$ that should match the shape of $W^l$. You had this right in your original post.
Bias Updates
This brings us to the bias updates.
There should be one bias term per node in a given layer, so the shapes of your biases are correct (i.e. $\mathbb{R}^2$ for layer 0 and $\mathbb{R}$ for layer 1). Now, we saw above that the shape of $\delta^l$ also matches the number of nodes in layer $l$, so again the element-wise subtraction in your original bias update equation works.
I also tried using this book to learn backprop, but I had a hard time connecting the variables with the different parts of the network and the corresponding code. I finally understood the algorithm in depth only after deriving all the update equations by hand for a very small network (2 inputs, one output, no hidden layers) and working my way up to larger networks, making sure to keep track of the shapes of the inputs and outputs along the way. If you're having trouble with the update equations I highly recommend this.
A final piece of advice that helped me: drop the $x$ and the summations over input examples from your formulations and just treat everything as matrices (e.g. a scalar becomes a matrix in $\mathbb{R}^{1 \times 1}$, $X$ is a matrix in $\mathbb{R}^{N \times D}$). First, this allows you to better interpret matrix orientations and debug issues such as a missing transpose operation. Second, this is (in my limited understanding) how backprop should actually be implemented in order to take advantage of optimized linalg libraries and GPUs, so it's perhaps a bit more relevant.


What does "immediate vector-valued feedback" mean? In the book Artificial Intelligence Engines: A Tutorial Introduction to the Mathematics of Deep Learning, James Stone says

  With supervised learning, the response to each input vector is an output vector that receives immediate vector-valued feedback specifying the correct output, and this feedback refers uniquely to the input vector just received; in contrast, each reinforcement learning output vector (action) receives scalar-valued feedback often sometime after the action, and this feedback signal depends on actions taken before and after the current action.

I fail to understand the part formatted in bold. Once we have a set of labeled examples (feature vector and label pairs), where is the "feedback" coming from? Testing and validation results of our calibrated model (say a neural network based one)?

By "immediate vector-valued feedback", they probably mean exactly the label in the "labeled examples" you mentioned.


How can I use a 2-dimensional feature matrix as the input to a neural network? How can I use a 2-dimensional feature matrix, rather than a feature vector, as the input to a neural network?
For a WWII naval wargame, I have sorted out the features of interest to approximate the game state $S$ at a given time $t$.

they are a maximum of 50 naval task forces and airbases on the map

each one has a lot of features (hex, weather, number of ships, type, distance to other task forces, distance to objective, cargo, intelligence, combat value, speed, damage, etc.)

The output would be the probability of winning and the level of victory.

The most obvious way to do this would be to simply "unroll" your matrix into a vector. Your example input matrix would get turned into the following input vector:

$$\left( \begin{array}{} a_1 & a_2 & \dots & a_t & b_1 & b_2 & \dots & b_t & c_1 & c_2 & \dots & c_t \end{array} \right)$$

I don't think there are any other clear ways to use an "input matrix" really. The only benefit I could see in using an input matrix rather than an unrolled vector (if it were possible to do so in whatever way) would be if doing so would somehow enable the learning algorithm to exploit the "domain knowledge" that certain input features are related to each other in special ways (i.e. features in the same row belong to the same unit, and features in the same column are the same "type" of feature, or other way around). Intuitively, I suspect something like this could be accomplished by restricting the number of connections you make to the next layer. For example, you could make a part of the next layer only be connected to all the $a_i$ features, a different part connected only to all the $b_i$ features, etc. Similarly, you could have a part that is connected only to the $a_1, b_1, c_1, \dots$ features, a different part only connected to the $a_2, b_2, c_2, \dots$ features, etc. I don't know for sure how well this would work though... just think that it could.


What is the process working on Tensorflow model.fit()? I created a binary image classification model. The dataset contains about 500K images in each class, with ratio = Train : Validation : Test = 7 : 2 : 1. Total images = 1M
I split my dataset into 5 parts (compute constraints)—5 training subsets, 5 validation subsets, and 1 test subset.
I trained and evaluated my model stage by stage. In first stage (evaluation), my model's accuracy was 65%. I re-fitted it with 2nd dataset and the accuracy was 43%. I did same process with the rest, and my accuracies were: 65%, 43%, 57%, 21%, 30%.
How can I train my model in staged training?
I want to train models with different datasets without reinitialize the weight every training process.

You can save weights during training by passing checkpoint callback to model.fit() method.
# Instantiate your model here
model = create_model()

# Set model configurations here
model.compile(loss=..., optimizer=..., metrics=...)

# Set checkpoint path
checkpoint_path = "model_weights.ckpt"

# Create a callback that saves the model's weights
    filepath=checkpoint_path,
    save_weights_only=True,
    monitor='val_loss',
    mode='min',
    save_best_only=True)

# Train the model with the new callback
model.fit(train_images_1,
          train_labels_1,
          epochs=50,
          batch_size=batch_size,
          callbacks=[cp_callback],
          validation_data=(test_images_1, test_labels_1),
          verbose=0)

After finishing training 1st dataset, model weights will be saved in file called model_weights.ckpt. Before starting training next dataset, load the model weights as below
# Create a new model instance
model = create_model()

# Set model configurations here
model.compile(loss=..., optimizer=..., metrics=...)

# Set checkpoint path
checkpoint_path = "model_weights.ckpt"

# Load the previously saved weights
model.load_weights(checkpoint_path)

# Create a callback that saves the model's weights
cp_callback = tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_path,
    save_weights_only=True,
    monitor='val_loss',
    mode='min',
    save_best_only=True)

# Train the model with the new callback
model.fit(train_images_2,
          train_labels_2,
          epochs=50,
          batch_size=batch_size,
          callbacks=[cp_callback],
          validation_data=(test_images_2, test_labels_2),
          verbose=0)

Repeat this for all datasets.


Current research on Gödel machines Is there any current research on Gödel machines? It seems that the last article by Jürgen Schmidhuber on this topic was published in 2012: http://people.idsia.ch/~juergen/goedelmachine.html

This paper Can Machines Design? An Artificial General Intelligence Approach (2018, presented at AGI-18 and published in the related proceedings here), which proposes the design Gödel machine, may be useful to you.
After a quick search, I have not found other relevant papers, so I suppose that the research on GMs is currently not very active.


Is there a general file type associated with AI projects? This is a general question.

Is there a general file type associated with AI projects?

Photoshop = .psd
Excel = csv
Artificial Intelligence = ?

No, there is no file type associated with AI projects in general.

Your examples of Photoshop and Excel are specific corporate branded products. These store bespoke data that only works with those products (plus maybe a few converters that can read the files for competitor products).

Even more general examples such as .jpg for images or .txt for text documents are not a good match to AI in general. AI is such a broad field, it is next to impossible to define a standard set of components of an "AI project" in order to build a single file format that could handle the contents of all AI projects.

That said, practical work in AI is likely to include use of specific file formats and extensions, depending on what you do. For instance if you work with Python and TensorFlow you are likely to use .py files for your source code, .ckpt for neural network training checkpoints, .pb for saved models. The last two extensions - ckpt and pb - are only semi-formal naming conventions though.

This lack of a single file extension, and use of maybe a dozen ones that you would be familiar with on a project, is true of most coding and software development work. If you work with an integrated devlopment environment (IDE), then you might have a single "master" file that allows you to load up all the resources in the project to work on them. That is entirely optional though, and may have any of a number of extensions depending on which IDE you use.

As a quick example, the Transformers project on github is a popular resource for natural language processing, a topic often considered as part of AI. A quick look at that project shows:

Various development tool configuration files either starting with . or with various extensions - .cfg, .yaml
.py files for Python source code
Script files with .sh extension or with extension removed for convenience of using on command line
Dcoumentation written with .txt, .md and .rst extensions.
Some .ipynb files used for Jupyter notebooks - a format that bundles documentation, Python scripts and their output for sharing work.


How does Hindsight Experience Replay learn from unsuccessful trajectories? I am confused by how HER learns from unsuccessful trajectories. I understand that from failed trajectories it creates 'fake' goals that it can learn from.
Ignoring HER for now, if in the case where the robotic arm reaches the goal correctly, then the value functions ($V$) and action-value functions ($Q$) that correspond to the trajectories that get to the goal quicker will increase. These high $Q$ and $V$ values are ultimately important for getting the optimal policy.
However, if you create 'fake' goals from unsuccessful trajectories - that would increase the $Q$ and $V$s of the environment that lead to getting the 'fake' goal. Those new $Q$ and $V$s would be unhelpful and possibly detrimental for the robotic arm to reach the real goal.
What am I misunderstanding?

Ignoring HER for now the $Q$ and $V$ functions operate on states and actions which are part of a Markov decision process which we call $M_0$.
Back to HER, the $Q$ and $V$ functions now take a goal as an additional parameter called the goal. We will denote individual goals $g_n$, the true goal $g_0$, and the set of all goals $G$. The set of goals is chosen such that every state matches at least one goal. We create a new MDP $M_1 = M_0 \times G$ (i.e. a larger MDP composed of multiple copies of $M_0$, all the states in each being tagged with a goal). The reward is either +1 or 0 depending on whether or not the goal and the component from $M_0$ match in some predefined sense. In HER trajectories are collected from the subset of $M_1$ where $g = g_0$ and added to the replay buffer. When training the $Q$ and $V$ functions we don't only use the original trajectories: we create new ones by substituting new values of $g$, which we do so strategically so as to include some trajectories with a positive reward.
Things to note:

HER doesn't assign rewards to states or trajectories in $M_0$: the reward function used is only defined for $M_1$
The performance of HER depends on $Q$ and $V$ both being models with some ability to extrapolate to unseen data points; such as neural networks, support vector machines, etc. It would not provide any benefit if applied to value tables.


Use AI to auto-correlate the words of human-translated texts? There are many, many literary works in the public domain, along with human translations, many of which have entered the public domain as well. (Public domain = easily available)

In order for me to advance my knowledge of e.g. Japanese, I'd like to read texts in Japanese using a yet to be written tool, and when I encounter an unknown word/phrase/sentence, I'd like to just click on a position in the text and be transferred to the corresponding position in the e.g. English translation (or original) of the text in question.

Let's also assume we have access to a dictionary that translates a fair amount of words between those two languages (and there are of course free dictionaries for many language pairs).

What ways are there to use AI toolkits plus some wiring and perhaps scripting/programming, to auto-correlate the positions of two versions of the same text, in two languages? The results do not - and in fact in many cases cannot - be perfect, but they should be roughly correct.

I'm aware that this is still not a straightforwards task, as there are complicating factors like inflection of verbs and other grammatical properties that make the use of dictionary tools much harder. Also, translators will often translate to words that don't have that mapping in any dictionary. Then there is the fact that words aren't delimited by spaces in languages like my example language Japanese - (but if it is easier to work with only space-separated languages like, say, Spanish, or Russian, I'd like to hear answers to this simpler problem as well). Also the order of words and even hole sub-clauses differs from language to language.

A simple, non-AI approximation would be to

figure out at what relative position in the source language text the user clicked (e.g. at the character on position 50.234%)
then go to that same relative position 50.234% in the target language text

This approximation could perhaps be used as the starting point for the AI, which would then use words and dictionaries to make the results more accurate.

You are basically describing the way Google Translate works.

There has been a lot of research in text alignment in the area of multi-lingual corpus linguistics. An early paper (with sourcode) is Gale and Church's A Program for Aligning Sentences in Bilingual Corpora (PDF).

In linguistics these are called parallel texts. On the wikipedia page you will find links to a number of alignment programs which attempt to solve that issue.


Are absence of labels for classes of interest in a vision dataset a big problem? I wish to be able to detect: pedestrians, cars, traffic lights

I have two large datasets:
 - One contains instances and labels of all three classes.
 - The other contains instances of all three but only labels for pedestrians and cars. ie. there are many unlabelled traffic lights.

I want to combine the two datasets and train Yolov3 on it. Will the unlabelled presence of objects of interest significantly affect detection performance of that category?

So depends on sizes— if the second dataset has a lot of traffic lights without labels and you in a basic setup train on it, it will cause large performance decreases on that class. If it’s small, then it may just be “noise” to the model and it’ll filter it out, and still perform well. My recommendation is to adjust the network slightly and add a flag input saying which dataset the input was from: if from first, train normally. if from the second then ignore all losses pertain to boxes/classes that the network thinks a traffic light was in: that way your labels don’t try to tell the mode l “there’s no traffic light there”


What are some good papers or resources for aspect extraction and opinion modelling from video or audio? I am quite new to deep learning. I just finished the deep learning specialization by Professor Andrew NG and Deep Learning AI. Now, my professor (instructor) has advised me to look into some classic papers for aspect extraction and opinion mining from video. Could anyone suggest me some resources where I can get started? Can anyone suggest some papers I should read? Maybe a course or a book or some links to descriptive sessions. Your help would be appreciated.

The paper A Multi-modal Approach to Fine-grained Opinion Mining on Video Reviews could be useful for your purposes, although I have read only the abstract

Despite the recent advances in opinion mining for written reviews, few works have tackled the problem on other sources of reviews. In light of this issue, we propose a multimodal approach for mining fine-grained opinions from video reviews that is able to determine the aspects of the item under review that are being discussed and the sentiment orientation towards them. Our approach works at the sentence level without the need for time annotations and uses features derived from the audio, video and language transcriptions of its contents. We evaluate our approach on two datasets and show that leveraging the video and audio modalities consistently provides increased performance over text-only baselines, providing evidence these extra modalities are key in better understanding video reviews.

and looked at some of the diagrams and figures in the paper.


How should I handle invalid actions (when using REINFORCE)? I want to create an AI which can play five-in-a-row/gomoku. I want to use reinforcement learning for this.
I use policy gradient method, namely REINFORCE, with baseline. For the value and policy function approximation, I use a neural network. It has convolutional and fully connected layers. All of the layers, except for the output, are shared. The policy's output layer has $8 \times 8=64$ (the size of the board) output unit and softmax on them. So it is stochastic.
But what if the network produces a very high probability for an invalid action? An invalid action is when the agent wants to check a square that has one X or O in it. I think it can be stuck in that game state.
How should I handle this situation?
My guess is to use the actor-critic method. For an invalid action, we should give a negative reward and pass the turn to the opponent.

Just ignore the invalid moves.

For exploration it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If you only punish illegal moves they will still retain some probability (however small) and therefore will be executed from time to time (however seldom). So you will always retain an agent which occasionally makes illegal moves.

To me it makes more sense to just set the probabilities of all illegal moves to zero and renormalise the output vector before you choose your move.


Can programming cause passive or aggressive behavior in AIs? Generally, people can be classified as aggressive (Type A) or passive. Could the programming of AI systems cause aggressive or passive behavior in those AIs?

The Wikipedia entry on this personality theory says of Type A people:

  The theory describes Type A individuals as ambitious, rigidly organized, highly status-conscious, sensitive, impatient, anxious, proactive, and concerned with time management. People with Type A personalities are often high-achieving "workaholics."

All of those attributes could conceivably be explicitly programmed in. Alternatively, most of them could arise from a basic goal of performing a certain task as efficiently as possible. After all, if you really want to carry out a task, you're going to get organized, you'll only do other things if they're asked of you by someone important, you won't want to get bogged down in irrelevant things, you'll actively pursue the necessary resources, and you'll want to use time as effectively as possible.

Note that this applies only to strong AIs, since weak AIs like image recognizers don't generally have personalities that we can interact with.

Now, just for fun, let's consider an overly aggressive personality, to the point of a disorder.

This Counselling Resource page seems helpful in describing what an aggressive person does. The page includes a bulleted list of common characteristics, which I distill into the following:

They attempt to gain dominance and control
They oppose to anything that places limits on them
They take advantage of others to further their own goals
They hide information from whose who would oppose them
They rarely decide to stop pursuing their desires (even impulses)

This all seems like a characterization of an AI designed to be the best at its task: the best out of any other agent, and the best it by itself could possibly be. Ruthless pursuit of the highest performance would involve taking control of all relevant resources (including other agents), demolishing barriers to the goal, thwarting those who would interfere with progress, and carrying out each possibly-useful idea/desire to completion.

In summary, yes, an AI's behavior and personality are programmable, either explicitly or through some kind of emergence.


Best approach for 2D Grid Image Segmentation I'm working on a project where I need to extract text from grocery discount flyers like the Costco announcement below (retrieved in a random google search, Costco is not the deal here):

If I just run OCR (like with Tesseract in python):

import cv2
import pytesseract
img = cv2.imread('costco.jpg')
text = pytesseract.image_to_string(img)
print(text)

I get:

  Cadbury Chocolate

  variety pack packet

  ere $12.99 i rom hagst 31026 2012
>
> Je laa
> + a
>
> Wrigley’s Excel Gum variety
>
> Backol 24
>
> $13.79 fom agus 26.202

  OFF

  Solon Extra virgin olive oil [...]

Which is a lot noisy.

My guess is that splitting the image to its base squares enchances the recognition.

However, I'm confused on how to do it. I can classify images using a CNN, but am not sure about object recognition.

Should I have a sliding window and train several "grid box" objects on a generic CNN and then provide this window data to be classified? How to adapt to distinct object window sizes?

This is a really cool problem. You already have a working model here are a few different ways of going forward with the project.

Grouping text based on locality. "no segmentation"  Text region extraction in a document image based on the Delaunay tessellation
Segmentation  Multiscale Edge-Based Text Extraction from Complex Images
Training a map of the image then extracting text form the map.  Multi Oriented Text Detection

Training the model to detect boxes followed by extracting the text from each box seems like a very smart direction to move in. This paper talks about the former.  automatic image segmentation and edge detection  Good luck!


How do I classify an image that contains only polygons? I have two closed polygons, drawn as connected straight black lines on a white background. I need to classify such images in to three forms

Two separate polygons
One polygon encloses the other
The two polygons overlap each other.

The polygons vary in sizes and location on the image, and the image contains only the polygons and the white background.

Which neural network architecture should I use to solve this problem?

To make it easy take small known CNN network like Alex Net, train like : input  is image, output is [1,0,0,] separate, [0,1,0], encloses, [0,0,1] overlap. Cause you task is easy i guess that will be fine. But it can be done without ML just analysing image with constant algorithm by where are the points of poligons.


Why do we have to train a model from scratch every time? I have started on Andrew Ng's machine learning course. It seems that machine learning is learning correlations with known data based on as many parameters as possible. For example, if we collect data on existing property prices with information on the land area, built-in area, type of building, age of the building, etc, it is possible to predict the price of another property if we input the value of the various parameters of this property.
Similarly, if we keep the images (the black and white pixels) of cats, we can tell whether a new picture is a cat if it bears some resemblance to the pixels of existing labeled cat images.

This approach sounds great, but is it practical? How much effort and zettabytes of data do we have to keep just to reach the brainpower of, say, a 3-year old, who can recognize dogs, cats, tigers, a Mustang, trucks, a hamburger restaurant, and so on?

Why does everyone have to repeat the effort of learning the same things?

If Google has already learned cats, or if someone already has a program to recognize handwritten digits, can this knowledge be shared and re-used? Or is it just a matter of paying for them?

Disclosure: I am a product manager on Google Cloud Platform.

  [...] why does everyone have to repeat the effort of learning the same things?

  If Google has already learned cats, or if someone already has a program to recognize handwritten digits, can this knowledge be shared and re-used? Or is it just a matter of paying for them?

You don't have to rebuild these machine learning models from scratch; you can reuse prebuilt machine learning algorithms, e.g., Google Cloud provides the following hosted APIs as a service:

Google Cloud Vision API to recognize images and provide text labels
Google Cloud Speech API to convert speech to text
Google Cloud Natural Language API to understand language, find entities, etc.
and more...

You can put these APIs together to build interesting applications, e.g.,

build a smart Raspberry Pi bot with Cloud Vision and Speech APIs (video presentation; blog post + video overview)


Can I detect unique people in a video? I am having a video feed with multiple faces in it. I need to detect each face and the gender as well and assign the gender against each person. I am not sure how to uniquely identify a face as Face1 and Face2 etc. I do not need to know their names, just need to track a person in the entire video. I thought of tracking methods but people are constantly moving and changing so a box location can be occupied by another person after some frames.

I am interested in a way where I can assign an id to a face but I am not sure how to do it. I can use Facial Recognition Based embedding on each face and track that. But that seems to be an overkill for the job. Is there any other method available or Facial Recognition/Embedding is the only method to uniquely identify people in a video?

Can I detect unique people in a video?

Yes. However, for some applications commercial systems are not good enough: German source
The concepts you are looking for are:

Optical Flow: for two frames, define for each point how it moved
Object tracking: given a single object in one frame, find it in the next. Optical flow helps with that.
Object detection: look at YOLO / SSD. Small introduction here.


Reinforcement Learning for an environment that is non-markovian I am a beginner in the field of Reinforcement Learning with only a couple of months of experience being in the field. Soon, I will start working on a project where we want to optimize the production of a chemical unit through reinforcement learning approach. From the SME's, we already obtained a simulator code that can take some input and render us the output. A part of our output is our objective function that we want to maximize by tuning the input variables. From a reinforcement learning angle, the inputs will be the agent actions, while the state and reward can be obtained from the output. We are currently in the process of building a RL environment, the major part of which is the simulator code described above.
We were talking to a RL expert and she mentioned that one of the thing that we have here conceptually wrong is that our environment will not have the Markov property in the sense that it is really a 'one-step process' with the process not continuing from the previous state and there is no sort of continuity in state transitions. She is correct there. This made me think, how can we get around this then. Can we perhaps append some part of the current state to the next state etc. More importantly, I have seen RL applied to optimal control in other examples as well which are non-markovian ex. scheduling, tsp problems, process optimization etc. What is the explanation in such cases? Does one simply assumes process to be markovian with unknown transition function?
Pardon me if my question appears too open ended but I am keen to seek some advice through this forum and would like to receive pointers helpful to think in right direction. Also, I think many others will find this question relevant too in the future.

RL is currently being applied to environments which are definitely not markovian, maybe they are weakly markovian with decreasing dependency.
You need to provide details of your problem, if it is 1 step then any optimization system can be used.


Will LMS always be convex function? If yes, then why do we change it for neural networks? In LMS(least mean square) since, we use a quadratic error function, and quadratic functions are generally parabola in (some convex like shape). I wonder whether that is the reason why we use least square error metric? If that is not the case(its not ALWAYS convex or reason WHY we use LMS), what is the reason then? why this metric changes for deep learning/neural networks but works for regression problems?

[EDIT]: Will this always be a convex function or is there any possibility that it will not be convex?

Square loss is fine for regression, since minimizing it is the same as maximizing the likelihood of the model parameters (under assumption that the error is Gaussian). However, if the model directly produces probabilities, then it is natural to use these probabilities directly within the loss. Hence, in all classification models we prefer to minimize negative log-likelihood of the correct class.

Note that choosing a natural loss leads to several practical advantages. In particular, applying a quadratic loss after a sigmoid activation leads to very poor gradients when the sigmoid is saturated in the wrong direction. The negative log-likelihood loss has no such problems.

This issue is not specific for neural networks. Logistic regression has used the negative log likelihood loss ever since 1958.


Are methods of exhaustive search considered to be AI? Some programs do exhaustive searches for a solution while others do heuristic searches for a similar answer. For example, in chess, the search for the best next move tends to be more exhaustive in nature whereas, in Go, the search for the best next move tends to be more heuristic in nature due to the much larger search space.

Is the technique of brute force exhaustive searching for a good answer considered to be AI or is it generally required that heuristic algorithms be used before being deemed AI? If so, is the chess-playing computer beating a human professional seen as a meaningful milestone?

If one thinks of intelligence as a continuous measure of optimization power (that is, how much better are outcomes for any unit of cognitive effort expended), then exhaustive search has non-zero intelligence (in that it does actually give better outcomes as more effort is expended) but very, very low intelligence (as the outcomes are better mostly by luck, and the amount of effort expended can be impossibly large).


Is there any difference between ConvNet and CNN? ConvNet stands for Convolutional Networks and CNN stands for Convolutional Neural Networks.
Is there any difference between both?
If yes, then what is it?
If no, is there any reason behind using ConvNet at some places and CNN at some other places in literature?

Both terms just mean convolutional neural network. I don't believe there is any particular reason to choose one over the other: ConvNet is slightly easier to say out loud and CNN is slightly shorter to write, but there is absolutely no difference in meaning.
For some contrasting examples in the literature, the EfficientNet paper chooses the term ConvNet and this paper on AlexNet chooses CNN. They are talking about the same sorts of neural networks though!


Is there a way to get landmark features automatically learned by a neural network? Is there a way to get landmark features automatically learned by a neural network without having to manually pre-label them in the images that are being fed into the network?

No, you can't. In CNN, if you want to detect landmark, you need to prepare data with region box, it's coordinates, width, height, than number of points that should be detected and points coordinates. Then your target vector should be,

This is your target vector. Optionally you can use YOLO algorithm.


Where is L2-regularization term applied I have a confusion on where exactly is the L2 regularization (weight decay) is added.
In various resources I have come across, I find two equations where L2 regularization is applied.
Adding R(W) to loss function makes sense because it tries decrease large weights. Also, I have seen equations where we add R(W) to the weight update term, 2nd equation in 2nd line as shown in this image:

In the above image, using the weight update rule that
W(final) = W(initial) + (alpha) * (Gradient of W),

I obtain a different equation as compared to the other equation which is commonly written in various resources.
Where exactly is the regularization term added, I previously thought it was only added in Loss function but that gives me a different weight update equation from what is commonly presented in resources.( Or is my interpretation of the equation wrong? )
I presume it is also added in weight update equation because while constructing models, we add regularization term.
model.add(Conv2D(256, (5,5), padding="same", kernel_regularizer=l2(reg)))

Would be grateful for any help.

The regularization terms are applied to the loss functions by default.
However, their gradients do appear in the update step as the gradient of loss appears in the update step.


How to make spacy lemmatization process fast? I am applying spacy lemmatization on my dataset, but already 20-30 mins passed and the code is still running.

Is there anyway to make it faster? Is there any option to do this process using GPU?

My dataset size is 20k number of rows & 3 columns

https://spacy.io/api/lemmatizer just uses lookup tables and the only upstream task it relies on is POS tagging, so it should be relatively fast. For large amounts of text, SpaCy recommends using nlp.pipe, which can work in batches and has built in support for multiprocessing (with the n_process keyword), rather than than simply nlp.
Also, make sure you disable any pipeline elements that you don't plan to use, as they'll just waste processing time. If you're only doing lemmatization, you'll pass disable=["parser", "ner"] to the nlp.pipe call.
Example code that takes all of the above into account is below.
import spacy
nlp = spacy.load("en_core_web_sm")

docs = ["We've been running all day.", "Let's be better."]

for doc in nlp.pipe(docs, batch_size=32, n_process=3, disable=["parser", "ner"]):
    print([tok.lemma_ for tok in doc])

# ['-PRON-', 'have', 'be', 'run', 'all', 'day', '.']
# ['let', '-PRON-', 'be', 'well', '.']
```