Why cannot an AI agent adjust the reward function directly? In standard Reinforcement Learning the reward function is specified by an AI designer and is external to the AI agent. The agent attempts to find a behaviour that collects higher cumulative discounted reward. In Evolutionary Reinforcement Learning the reward function is specified by
the agent’s genetic code and evolved in simulated Darwinian evolution over multiple generations.
Here too the AI agent cannot directly adjust the reward function and instead adjusts its behaviour
towards collecting higher rewards. Why do both approaches prevent the AI agent from changing its reward function at will? What happens if we do allow the AI agent to do so?

Why do both approaches prevent the AI agent from changing its reward function at will?

In RL for optimal control, the reward function is part of the problem formulation. That is, it describes the goals of the agent. Sometimes this is obviously something that should not be under the agent's control, if the reward is a real-world quantity that it should maximise - e.g. the amount of money it makes in profit - then it makes no sense that the agent could arbitrarily declare that the quantity is different to the thing that it observed.

Other times, there is some flexibility, an agent that needs to escape a maze in a short time could have -1 reward per time step inside the maze or -0.1 reward per time step, or +1 reward for escaping with a discount factor applied. However, the flexibility can only go so far before it describes a different problem. Changing the -1 per time step to +1 per time step means that the agent's goal switches from escaping to staying in the maze.

In general, multiplying all the rewards in a MDP by some positive constant does not change a reinforcement learning problem. Sometimes it may be worth doing this scaling to make it easier for a specific approach, such as neural networks, to work efficiently. However, this is not something to put directly under the agent's control, but a hyperparameter like the number of hidden layers in the neural network. As a hyperparameter, usually the reward scaling is very flexible and not something worth spending much effort tuning - unlike the architecture of a neural network.

  What happens if we do allow the AI agent to do so?

Unless significant constraints are placed on what is allowed to change, then the agent will get any amount of reward it "wants" by doing anything it "wants", within whatever constraints are placed on changes allowed to the reward function. Typically in RL this would result in an agent that acts more or less randomly whilst getting progressively higher and higher reward on each iteration. Or in other words, an agent that does not attempt to solve any kind of problem.

There are a few special cases where a reward function can be adjusted or learned. One common case is inverse reinforcement learning, where an agent's activities are observed, it is assumed to be solving an MDP-like problem, and you are interested in understanding how it solves it, including what reward function it is using. The reward function must be learned by fitting it to observations of the agent.


Are there any studies which attempt to use AI to guess the human emotion based on the brainwaves? Have there been any studies which attempted to use AI algorithms to detect human thoughts or emotions based on brain activity, such as using BCI/EEG devices?

By this, I mean simple guesses such as whether the person was happy or angry, or what object (e.g. banana, car) they were thinking about.

If so, did any of those studies show some degree of success?

As per this site

  Researchers recorded the complex patterns of electrical activity generated by someone’s brain, as the subject listened to someone talking.
  By feeding those brainwave patterns into a computer, they were able to translate them back into actual words — the same words that the volunteer had been hearing.

  The scientists behind the work believe they can now go further and read the unspoken thoughts of people using electrodes placed against the brain.

  In the experiment, each patient listened to a recording of spoken words for five to ten minutes, while the net of electrodes placed under their skull monitored activity in a part of the brain involved in understanding speech called Wernicke’s area.

  In one experiment, volunteers looked at black-and-white photographs while the scanner monitored activity in part of the brain that handles vision called the primary visual cortex.
  A computer predicted accurately the image that the person was looking at purely from the pattern of brain activity.

So AI might be able to read our emotions as well in near future.

I found that google glasses can detect people's emotion via facial expression, voice tone e.t.c, (just like us), obviously not what they are thinking in their brain.


Policy invariance under affine transformations of the reward function? In the context of a Markov decision process, this paper says "it is well-known that the optimal policy is invariant to positive affine transformation of the reward function".

On the other hand, exercise 3.7 of Sutton and Barto gives an example of a robot in a maze:

  Imagine that you are designing a robot to run a maze. You decide to give it a reward of +1 for escaping from the maze and a reward of zero at all other times. The task seems to break down naturally into episodes—the successive runs through the maze—so you decide to treat it as an episodic task, where the goal is to maximize expected total reward (3.7). After running the learning agent for a while, you find that it is showing no improvement in escaping from the maze. What is going wrong? Have you effectively communicated to the agent what you want it to achieve?

It seems like the robot is not being rewarded for escaping quickly (escaping in 10 seconds gives it just as much reward as escaping in 1000 seconds). One fix seems to be to subtract 1 from each reward, so that each timestep the robot stays in the maze, it accumulates $-1$ in reward, and upon escape it gets zero reward. This seems to change the set of optimal policies (now there are way fewer policies which achieve the best possible return). In other words, a positive affine transformation $r \mapsto 1 \cdot r - 1$ seems to have changed the optimal policy.

How can I reconcile "the optimal policy is invariant to positive affine transformation of the reward function" with the maze example?

This statement:

  (it is well-known that the optimal policy is invariant to positive affine transformation of the reward function).

is, as far as I know, and as you summarise, incorrect because simple translations to reward signal do affect the optimal policy, and the affine transform of a real number $x$ can be given by $f(x) = mx + c$

It is well known that optimal policy is unaffected by multiplying all rewards by a positive scaling, e.g. $f(x) = mx$ where $m$ is positive.

It is also worth noting that if an optimal policy is derived from Q values using $\pi(s) = \text{argmax}_a Q(s,a)$, then that policy function is invariant to positive affine transformations of action values given by $Q(s,a)$. Perhaps that was what the paper authors meant to write, given that they go on to apply normalisation to Q values.

The impact of the mistake is not relevant to the the rest of the paper as far as I can see (caveat: I have not read the whole paper).


Is it possible to vectorise a CNN? I am trying to write a CNN from scratch and am wondering if it is possible to vectorize the convolution step.
For example, if I had a dataset of 500 RGB images of size 32x32x3, and wanted the first convolutional layer to have 64 filters, how would I go about the vectorization of this layer?
Currently, I am running through all 500 images in a for loop, convoluting individually. I do this for all the images up to the flattening stage (where it essentially becomes a normal NN again), at which point I can implement the normal vectorisation approach to get to my output, etc.
A holistic overview of the process would be appreciated, as I am struggling to get my head around it and am struggling to find and information on the matter online.

Yes you can vectorize a CNN. See this github file for details: https://github.com/parasdahal/deepnet/blob/master/deepnet/layers.py

After looking through it it basically transposes the input to some dimension and apply matrix multiplication to the weight with some other kind of transfromation. Pls refer to the github repository for details.

Hope this can help you and have a nice day!


Is there ever a need to combine deep learning frameworks? (Eg. TensorFlow & Torch)? Imagine a simple scenario of having a large repository using one framework and integrated with data/robots, etc., then having a new feature requested and the framework missing some vital functionality that is available in another framework (say a new kind of layer).
For many mathematical libraries, it can be easy to reverse engineer the specific function that one would want form the other framework so as to not import the entire framework but in the case of deep learning, this isn't so trivial nor that easily testable.
In these cases, does one take on the problem of reverse engineering the functionality or do they attempt to combine the architectures? If it's the combination, what are the biggest issues with integration?

In the time since I asked this question, I have been able to combine Tensorflow and Chainer considerably well. That being said, one should try to avoid combining deep learning frameworks if one can for a few reasons:

It doubles the amount of documentation one needs to reference
It makes it difficult for new developers to become familiar with the
code base.
It creates multiple data transfer bottlenecks when one
has to transfer data between the deep learning different
configurations such as: CPU -> GPU (Tensorflow) -> CPU -> GPU
(Chainer) -> CPU ... etc.

That being said, I combined them because of time constraints on a research project which would not be finding its way into an industrial product. In this research setting, I still wouldn't recommend combining frameworks if one can avoid it but if it means doubling the time it takes to do the project for a proof of concept or experiment, then it seems reasonable and can drastically speed up development until the missing functionality is added to the desired framework.

There are 2 issues with integration:

Abstracting the networks so that the main
training/testing script does not depend on the framework.

This is important in the event that the missing functionality of your desired framework is added. It also helps new developers understand what the overall algorithm or experiment is without being bogged down by framework details.
Tensorflow and similar frameworks which require a Session object.

Chainer, PyTorch (which came from Chainer), and any other framework that doesn't require a something like Tensorflow's Session all play very nicely together and can be used (almost) interchangeably. For Session based frameworks, the main script/loop/experiment must all take place within a Session. With proper abstraction, my experiment only required a with tf.Session() as sess: to be used in the main experiment and was the only reference that Tensorflow was being used.

Although I still don't recommend combining frameworks for the reasons specified above, if one abstracts their framework calls enough, it is almost seamless to integrate deep learning frameworks.


Are there profitable hedge funds using AI? Is there any research in this area?

High-frequency trading is where you see it being used, essentially, decision-making algorithms analyzing and making transactions in microseconds.  It accounts for a significant percentage of market activity, and has been considered a source of greater market volatility (See: Flash Crashes).

You can bet that hedge funds are evaluating every form of AI for trading and predicting market trends in general, but, unlike academics (and a segment of the tech sector) I doubt they consider it in their interests to publish methods which competitors could then utilize.

It's difficult to source information on exactly what is being looked into and utilized in the financial sector because there is a lot of marketing noise (exaggerated claims,
 unreliable sources) but I did find these articles:

Will AI-Powered Hedge Funds Outsmart the Market? (Tech Review; 2/4/2016)
"AI Hedge Fund Is Said to Liquidate After Less Than Two Years (Bloomberg; 9/7/2018)"

My gut tells me that Machine Learning methods will surpass humans in these kinds of decisions before too long (educated guess), and only increase in utility as the dataset these algorithms draw from grows, because it's ultimately a statistical problem.

AI is related to Game Theory, the study of economic decision-making, in the context of utility.  Game Theory might be said to have had its greatest success in computing in general, via minimax, but has traditionally been much harder to apply in real world economics.  Minimax can be utilized in machine learning, and it's actually hard to think of any economic decision-making that wouldn't utilize it in some form.


How do I convert table-based to neural network-based Q-learning? I've used a table to represent the Q function, while an agent is being trained to catch the cheese without touching the walls.

The first and last row (and column) of the matrix are associated with the walls. I placed in last cell a cheese that agent must catch while being training.

So far, I've done it with dynamic states and, when necessary, I resized matrix with new states. I've used four actions (up, left, right and down).

I would like now to use an ANN to represent my Q function. How do I do that? What should be the input and output of such neural network?

A neural network (NN) is a "function approximator", that is, it is a model that can be used to approximate functions. In fact, a neural network with at least one hidden layer is a "universal" function approximator (that is, it can approximate any function).

In mathematics, a function $f$ is usually represented as a mapping of the form $f: D \rightarrow C$, where $D$ and $C$ are respectively the domain (inputs) and codomain (outputs) of $f$, and $\rightarrow$ means that $f$ "maps" $D$ to $C$. A NN (with at least one hidden layer) can thus approximate any function $f$ of this form.

In your context, the $Q$ table is a function: it is a mapping between states and actions (inputs) and Q values (outputs), which are the "expected future cumulative reward" (that you will obtain if you take a certain action from a certain state and then continue to follow the same policy). The Q function can thus more formally be denoted by $Q: (S, A) \rightarrow \mathcal{R}$, where $S$ is the "space of states" and $A$ is the "space of actions" in your problem. Initially, your Q table does not contain the correct (optimal) values. However, after training (or learning), hopefully, your Q table will be an approximation to the optimal Q function for your specific problem.

How do you then represent this table using a NN? What should be the inputs and outputs of the NN?

Let's suppose that your $Q$ table is implemented as matrix $M$. Then $M[s, a]$ is the $Q$ value for the state $s$ and action $a$. So, in this case, the combination of $s$ and $a$ is the input, whereas $M[s, a]$ is the output of your $Q$ function.

To represent this table as a NN, you can thus have as input of the NN the state and action, and as output the $Q$ value. You will then train the NN (using e.g. back-propagation) to learn the $Q(s, a)$, given a state $s$ and an action $a$ as input, for all $s \in \mathcal{S}$ and $a \in \mathcal{A}$. So, during your $Q$-learning algorithm, instead of using $M[s, a]$ to represent $Q(s, a)$, you will simply use the current output of your neural network.

Note that, in practice, I think, you will likely encounter problems while training the NN, if you update the weights of your NN at every time step (because e.g. NNs do not cope well with correlated data, and, in general, the "experience" data you will obtain from time step to time step will be highly correlated). Anyway, this is the basic idea of how to use a NN to represent a $Q$ function. There are other ways, but this is the simplest one, at least, conceptually.


Which methods or algorithms to develop a learning application? I am creating a game application that will generate a new level based on the performance of the user in the previous level.
The application is regarding language improvement, to be precise. Suppose the user performed well in grammar-related questions and weak on vocabulary in a particular level. Then the new level generated will be more focused on improving the vocabulary of the user.
All the questions will be present in a database with tags related to sections or category that they belong to. What AI concepts can I use to develop an application mentioned above?

even though that's not really AI, the easiest way to do that would probably be to put coefficients on each question

e.g. your question would have something like

grammar=0.55
vocabulary=0.45
if(won){
  success=-1
}else{
  success=1
}

the lower the level, the less questions of this type will appear

at the end of the level, you sum each question times how much did the player succeed

player.level is a real number array

for each question{
   player.level["grammar"]=player.level["grammar"]+question.grammar*question.success
   player.level["vocabulary"]=player.level["vocabulary"]+question.vocabulary*question.success
}

then when you initialize the level, generate the questions according to the player's level
choice is a random real number between 0 and sum of all player.level (offseted to be above 0)

for each player.level{
   if choice>player.level{
      choice=choice-player.level
   }else{
      return question_type=player.level.type
   }
}

though a "true" AI can make the same or better choices of questions, it will probably be slower, and require more data than a simple algorithm like this one.


Should I use U-net to label keys in a keyboard image? This is a 600*800 image.

Which algorithm/model should I use to get an image like the one below, in which each key is detected and labeled by a rectangle?
I guess this is some kind of a segmentation problem where U-net is the most popular algorithm, though I don't know how to apply it to this particular problem.

If you just need to draw a rectangle around each key, this is an object detection or template matching problem, so you can use any of the available models for object detection (e.g. YOLO) or any technique for multi-template template matching (e.g. you can use sequential RANSAC or t-linkage). In the first case, you will need a labeled dataset, while, in the second case, you will need the original image and the templates (in your case, a template would be an image of a key).
So, no, this is not a segmentation problem (which would be the task of classifying each pixel in the objects of interest, and not just locating the objects).


Equivalence between expected parameter increments in "Off-Policy Temporal-Difference Learning with Function Approximation" I am having a hard time understanding the proof of theorem 1 presented in the "Off-Policy Temporal-Difference Learning with Function Approximation" paper.
Let $\Delta \theta$ and $\Delta \bar{\theta}$ be the sum of the parameter increments over an episode under on-policy $T D(\lambda)$
and importance sampled $T D(\lambda)$ respectively, assuming
that the starting weight vector is $\theta$ in both cases. Then
$E_{b}\left\{\Delta \bar{\theta} | s_{0}, a_{0}\right\}=E_{\pi}\left\{\Delta \theta | s_{0}, a_{0}\right\}, \quad \forall s_{0} \in \mathcal{S}, a_{0} \in \mathcal{A}$
We know that:
$$
\begin{aligned}
&\Delta \theta_{t}=\alpha\left(R_{t}^{\lambda}-\theta^{T} \phi_{t}\right) \phi_{t}\\
&R_{t}^{\lambda}=(1-\lambda) \sum_{n=1}^{\infty} \lambda^{n-1} R_{t}^{(n)}\\
&R_{t}^{(n)}=r_{t+1}+\gamma r_{t+2}+\cdots+\gamma^{n-1} r_{t+n}+\gamma^{n} \theta^{T} \phi_{t+n}
\end{aligned}
$$
and
$$\Delta \bar{\theta_{t}}=\alpha\left(\bar{R}_{t}^{\lambda}-\theta^{T} \phi_{t}\right) \phi_{t} \rho_{1} \rho_{2} \cdots \rho_{t}$$
$$
\begin{aligned}
\bar{R}_{t}^{(n)}=& r_{t+1}+\gamma r_{t+2} \rho_{t+1}+\cdots \\
&+\gamma^{n-1} r_{t+n} \rho_{t+1} \cdots \rho_{t+n-1} \\
&+\gamma^{n} \rho_{t+1} \cdots \rho_{t+n} \theta^{T} \phi_{t+n}
\end{aligned}
$$
And it is proven that:
$$
E_{b}\left\{\bar{R}_{t}^{\lambda} | s_{t}, a_{t}\right\}=E_{\pi}\left\{R_{t}^{\lambda} | s_{t}, a_{t}\right\}
$$
Here is the proof, it begins with:
$E_{b}\{\Delta \bar{\theta}\}=E_{b}\left\{\sum_{t=0}^{\infty} \alpha\left(\bar{R}_{t}^{\lambda}-\theta^{T} \phi_{t}\right) \phi_{t} \rho_{1} \rho_{2} \cdots \rho_{t}\right\}$
$=E_{b}\left\{\sum_{t=0}^{\infty} \sum_{n=1}^{\infty} \alpha(1-\lambda) \lambda^{n-1}\left(\bar{R}_{t}^{(n)}-\theta^{T} \phi_{t}\right) \phi_{t} \rho_{1} \rho_{2} \cdots \rho_{t}\right\}$.
which I believe is incorrect since,
$E_{b}\{\Delta \bar{\theta}\}=E_{b}\left\{\sum_{t=0}^{\infty} \alpha\left(\bar{R}_{t}^{\lambda}-\theta^{T} \phi_{t}\right) \phi_{t} \rho_{1} \rho_{2} \cdots \rho_{t}\right\}$
$=E_{b}\left\{\sum_{t=0}^{\infty} \alpha \left(\sum_{n=1}^{\infty}(1-\lambda) \lambda^{n-1}\bar{R}_{t}^{(n)}-\theta^{T} \phi_{t}\right) \phi_{t} \rho_{1} \rho_{2} \cdots \rho_{t}\right\}$.
and taking out the second sigma will lead to a sum over constant terms.
Furthermore, it is claimed that in order to prove the equivalence above, it is enough to prove the equivalence below:
$$
\begin{array}{c}
E_{b}\left\{\sum_{t=0}^{\infty}\left(\bar{R}_{t}^{(n)}-\theta^{T} \phi_{t}\right) \phi_{t} \rho_{1} \rho_{2} \cdots \rho_{t}\right\} \\
=E_{\pi}\left\{\sum_{t=0}^{\infty}\left(R_{t}^{(n)}-\theta^{T} \phi_{t}\right) \phi_{t}\right\}
\end{array}
$$
Which I don't understand why. and even if it is the case there are more ambiguities in the proof:
$E_{b}\left\{\sum_{t=0}^{\infty}\left(\bar{R}_{t}^{(n)}-\theta^{T} \phi_{t}\right) \phi_{t} \rho_{1} \rho_{2} \cdots \rho_{t}\right\}$
$$=\sum_{t=0}^{\infty} \sum_{\omega \in \Omega_{t}} p_{b}(\omega) \phi_{t} \prod_{k=1}^{t} \rho_{k} E_{b}\left\{\bar{R}_{t}^{(n)}-\theta^{T} \phi_{t} | s_{t}, a_{t}\right\}$$
(given the Markov property, and I don't understand why Markovian property leads to conditional independence !)
$$=\sum_{t=0}^{\infty} \sum_{\omega \in \Omega_{t}} \prod_{j=1}^{t} p_{s_{j-1}, s_{j}}^{a_{j-1}} b\left(s_{j}, a_{j}\right) \phi_{t} \prod_{k=1}^{t} \frac{\pi\left(s_{k}, a_{k}\right)}{b\left(s_{k}, a_{k}\right)} \cdot \left(E_{b}\left\{\bar{R}_{t}^{(n)} | s_{t}, a_{t}\right\}-\theta^{T} \phi_{t}\right)$$
$$= \sum_{t=0}^{\infty} \sum_{\omega \in \Omega_{t}} \prod_{j=1}^{t} p_{s_{j-1}, s_{j}}^{a_{j-1}} \pi\left(s_{j}, a_{j}\right) \phi_{t} \cdot\left(E_{b}\left\{\bar{R}_{t}^{(n)} | s_{t}, a_{t}\right\}-\theta^{T} \phi_{t}\right)$$
$$=\sum_{t=0}^{\infty} \sum_{\omega \in \Omega_{t}} p_{\pi}(\omega) \phi_{t}\left(E_{\pi}\left\{R^{(n)} | s_{t}, a_{t}\right\}-\theta^{T} \phi_{t}\right)$$
(using our previous result)
$$=E_{\pi}\left\{\sum_{t=0}^{\infty}\left(R_{t}^{(n)}-\theta^{T} \phi_{t}\right) \phi_{t}\right\} . \diamond$$
I'd be grateful if anyone could shed a light on this.

First part is correct
\begin{align}
&\sum_{n=1}^{\infty} \alpha(1-\lambda)\lambda^{n-1} (\bar R_t^{(n)} - \theta^T \phi_t)\\
=& \alpha[\sum_{n=1}^{\infty} (1-\lambda)\lambda^{n-1} \bar R_t^{(n)} - \sum_{n=1}^{\infty} (1-\lambda)\lambda^{n-1} \theta^T \phi_t]
\end{align}
$\sum_{n=1}^{\infty} (1-\lambda)\lambda^{(n-1)}$ sums to $1$ so we have
\begin{equation}
\alpha[\sum_{n=1}^{\infty} (1-\lambda)\lambda^{n-1} \bar R_t^{(n)} - \theta^T \phi_t]
\end{equation}
For the second part it's enough to prove equivalence for any $n$ because result contains sum over $n$. If you have 2 sums $\sum x_n$, $\sum y_n$ then the sums will be equal if for any $n$, $x_n = y_n$.

For the third part, we are in state $s_t$ and we already took action $a_t$ so we have
\begin{align}
&E_b \{ \sum_{t=0}^{\infty} (\bar R_t^{(n)} - \theta^T\phi_t)\phi_t \rho_1\rho_2\cdots\rho_t\}\\
=& \sum_{t=0}^{\infty} E_b \{(\bar R_t^{(n)} - \theta^T\phi_t)\phi_t \rho_1\rho_2\cdots\rho_t\}\\
=& \sum_{t=0}^{\infty} E_b \{\phi_t \rho_1\rho_2\cdots \rho_t\} E_b \{(\bar R_t^{(n)} - \theta^T\phi_t)|s_t, a_t\}
\end{align}
that is because $\rho_i, i = 1, \ldots, t-1$ depends on $s_i, a_i$. Because of Markov property expectation over $\bar R_t^{(n)}$ doesn't depend on those state it only depends on $s_t, a_t$ so they are independent. We don't need to consider $\phi_t$ and $\rho_t$ in expectation over $\bar R_t^{(n)}$ either because, like I said, we are in state $s_t$ and we took $a_t$ so they are already decided they would be considered a constant. We can then split total expectation in part $E_b \{\phi_t \rho_1\rho_2\cdots \rho_t\}$ for getting to state $s_t$ and taking action $a_t$ and part $E_b \{(\bar R_t^{(n)} - \theta^T\phi_t)|s_t, a_t\}$ for expectation over $\bar R_t^{(n)}$ after we got to state $s_t$ and took action $a_t$.


How to make a bot to learn from previous games How to train a bot, given a series of games in which he did (initially random) actions, to improve its behavior based on previous experiences?

The bot has some actions: e.g. shoot, wait, move, etc. It's a turn based "game" in which, for know, I'm running the bots with some objectives (e.g. kill some other bot) and random actions. So every bot will have a score function that at the end of the game will say, from X to Y (0 to 100?) if they did well or not.

So how to make the bots to learn of their previous experiences? Because this is not a fixed input as the neural networks take, this is kind of a list of games, each one in which the bot took several actions (one by every "turn"). The IA functions that I know are used to predict future values.. I'm not sure is the same.

Maybe I should have a function that gets the "more similar previous games" that the bot played and checked what were the actions he took, if the results were bad he should take another action, if the results were good then he should take the same action. But this seems kind of hardcoded.

Another option would be to train a neural network (somehow fixing the problem of the fixed input) based on previous game actions and to predict the future action's results in score (something that I guess it's similar to how chess and Go games work) and choose the one that seems to have better outcome.

I hope this is not too abstract. I don't want to hardcode much stuff in the bots, I'd like them to learn by their own starting from a blank page.

Reinforcement learning

The problem that you describe, namely, choosing a good sequence of actions based on a reward/score received based on the whole sequence (and possibly significantly delayed), is pretty much the textbook definition of Reinforcement learning.

As with quite a few other topics, deep neural networks currently seem to be a promising way for solving this type of problems. This may be a beginner-friendly description of this approach.


What effect does batch norm have on the gradient? Batch norm is a technique where they essentially standardize the activations at each layer, before passing it on to the next layer. Naturally, this will affect the gradient through the network. I have seen the equations that derive the back-propagation equations for the batch norm layers. From the original paper: https://arxiv.org/pdf/1502.03167.pdf

However, I have trouble understanding if there is an intuitive understanding of what effect it actually has on the network. For instance, does it help with the exploding gradient problem, since the activations are rescaled, and the variance of them is constrained?

"Naturally, this will affect the gradient through the network." this statement is only partially true, let's see why by starting explaining the real aim of batch normalisation.

As the title of the paper suggest, the aim of batch normalisation is to decrease training time by reducing covariance shift. What is covariance shift? We can conceive it as the variation that can happen between between the values of two layers of a network. We are all familiar with the concept that if we have input features with different unite scale, like kilos and euros, most likely lot of values will have different magnitude orders, like thousands for the weight could appear often with hundred of thousands for the money. When applying the activation function to different order of magnitude this discrepancy will remain, causing the values in the first layer to assume a really broad range of values. This is not good, since high fluctuation means more time to converge to stable values. This is why values fed to neural nets are always standardised.

The authors applied the same logic to the hidden layers, arguing that a deep neural net can be conceived as a repetition of itself (every hidden layer is an input layer that send features to another hidden layer) ergo the features should be normalised at every layer. How can we do it? Normalising every batch is the most natural way to procede, but in this way there is the risk to end up transforming the internal representation of a layer, because normalising is not a linear transformation. This is way the authors propose a clever way to perform hidden layers normalisation, which consists in a classic normalisation followed by a linear scaling performed with two trainable parameters $\beta$ and $\gamma$ (which appear in the last step of the batch norm scratch code below).

A really important thing to notice is that the mean and variance used to perform the classic normalisation are mean and variance calculated on the mini batch. I will explain why this is important in a sec, first I want to stress out that the $\beta$ parameter can actually bring to increase overfitting when batch norm is randomly stucked on top of other layer. The reason why is that, as we see in the scale and shift step, $\beta$ is nothing but a biased term, added to perform the shift of the mean of the batch hidden values. So, to avoid having an extra bias that lead to overfit, the bays on the previous layer should be removed, leaving only the classic matrix of weights parameters.

Back to the gradient problem, we can see that in itself doesn't necessarily lead to increased performances, but it does provide an advantage in terms of hidden layer values convergence. The x axis on the two right sub plots of the figure below represent the variation of the hidden values of net trained with and without batch norm. When trained with batch norm the hidden values reach stable ranges after few iteration. This help the net to reach high accuracy in less iterations (first subplot on the left) but we can see that even without batch norm the net reach eventually high accuracy.

The only help provided by batch norm to the gradient is the fact that, as noticed before, the normalisation is firstly performed by calculating the mean and variance on individual batches. This is important because this partial estimation of mean and variance introduces noice. Similarly to drop out, which has a regularisation effect due to the noice generated by randomly deactivating part of the weights, batch norm can introduce regularisation by adding noice due to the larger or smaller mean and variance estimated on individual batches. But still, batch norm was not introduced as a regularisation technique, and the equation you put on the question simply prove that it is possible to calculate the derivatives of the equations applied to perform the batch norm.


Can we use the recursive least squares as a learning algorithm to an ADALINE? I'm new to neural network, I study electrical engineering, and I just started working with ADALINEs.

I use Matlab, and in their Documentation they cite :

  However, here the LMS (least mean squares) learning rule, which is
  much more powerful than the perceptron learning rule, is used. The
  LMS, or Widrow-Hoff, learning rule minimizes the mean square error and
  thus moves the decision boundaries as far as it can from the training
  patterns.

The LMS algorithm is the default learning rule to linear neural network in Matlab, but a few days later I came across another algorithm which is : Recursive Least Squares (RLS) in a 2017 Research Article by Sachin Devassy and  Bhim Singh in the journal: IET Renewable Power Generation, under the title : Performance analysis of proportional resonant and ADALINE-based solar photovoltaic-integrated unified active power filter where they state :

  ADALINE-based approach is an efficient method for extracting
  fundamental component of load active current as no additional
  transformation and inverse transformations are required. The various
  adaptation algorithms include least mean square, recursive least
  squares etc.

My questions are:

Is RLS just like LMS (i mean can it be used as a learning algorithm too)?
If yes, how can I customize my ADALINE to use RLS instead of LMS as a learning algorithm (preferably in Matlab, if not in Python) because I want to do a comparative study between the two Algorithm!

RLS is a second order optimizer, so, unlike LMS which takes into account an approximation of the derivative of the gradient, RLS also considers the second order derivative. You can study more about second order methods in sub-section "8.6 Approximate Second-Order Methods" of the following book available online:
https://www.deeplearningbook.org/contents/optimization.html
Deep Learning
An MIT Press book
Ian Goodfellow and Yoshua Bengio and Aaron Courville


What are some techniques/method that can be used to train and detect objects like cars and humans? I have used OpenCV to train Haar cascades to detect face and other patterns. However I later realized that Haar tends to give a lot of false positives and I learned of Hog would give a more accurate results. But OpenCV doesn't have a good documentation of how to train hogs, I have googled a bit and found results that includes SVM and others.

OpenCV also has versioning problem where they move certain classes or functions somewhere else.

Are there any other techniques/method that I can use to train and detect objects and patterns? Preferably with proper documentation and basic tutorial/examples. Language preference: C#, Java, C++, Python

The terms you are looking for are deeplearning and convolutional neural networks for object detection. Google responds well to these terms.
From academical point of view you can start from:
Single shot multibox detector: https://arxiv.org/pdf/1512.02325v5.pdf
Or Faster-RCNN:
https://arxiv.org/pdf/1506.01497.pdf
These are not simple architectures and there are many improvements to them but these give you an idea of the current state of the art methods.
There are many implementations of both of these networks in deeplearning libraries for python (eg in Tensorflow, PyTorch).


Are there neural networks with 3-dimensional topologies? The topologies (or architectures) of the neural networks that I have seen so far are only 2-dimensional. So, are there neural networks whose topology is 3-dimensional (i.e. they have a width, height, and depth)?

Yes. Convolutional neural networks are usually 3-dimensional. In fact, they usually deal with images (e.g. RGB images), which can already be 3-dimensional.


How should I select the features for predicting diseases (in particular when patients specify their health issues)? My aim is to train a model for predicting diseases. Now, according to this Wikipedia article, diseases are classified based on the following criteria in general:

Causes (of the disease)
Pathogenesis (the mechanism by which the disease progresses)
Age
Gender
Symptoms (of the disease)
Damage (caused by the disease)
Organ type (e.g. heart disease, liver disease, etc.)

Are these features used for predicting diseases universally (i.e. all types of diseases)? I don't think so. There can be other attributes as well. For example, traveling in the case of coronavirus.
So, are there better features for predicting diseases?
Or which ones among them are better than the others, when patients specify their health issues?

So for Medical Prognosis, there are some variables that commonly come up like Age, Sex, Ascites, Hepato, Spider, Status of the disease and many others but it depends on the disease. You'll commonly encounter these variables if you're doing regression or classification.
Also, if you're reading Radiology Reports for getting the input for the model then you also have to take care of jargons. The same symptoms can be written in various ways but all point towards the same prognosis i.e., there can be synonyms for labels. Try reading this to get more information on how we can do information extraction from Radiology Reports. This is the famous CheXpert paper


Is a subset of a problem solution, used to recreate complete solution considered a heuristic? This is a question about a nomenclature - we already have the algorithm/solution, but we're not sure whether it qualifies as utilizing heuristics or not.

feel free to skip the problem explanation:

  A friend is writing a path-finding algorithm - an autopilot for an
  (off-road) vehicle in a computer game. This is a pretty classic
  problem - he finds a viable, not necessarily optimal but "good enough"
  route using the A* algorithm, by taking the terrain layout and vehicle
  capabilities into account, and modifying a direct (straight) line path
  to account for these. The whole map is known a'priori and invariant,
  though the start and destination are arbitrary (user-chosen) and the
  path is not guaranteed to exist at all.

  This cookie-cutter approach comes with a twist: limited storage space.
  We can afford some more volatile memory on start, but we should free
  most of it once the route has been found. The travel may take days -
  of real time too, so the path must be saved to disk, and the space in
  the save file for custom data like this is severely limited. Too
  limited to save all the waypoints - even after culling trivial
  solution waypoints ('continue straight ahead'), and by a rather large
  margin, order of 20% the size of our data set.

  A solution we came up with is to calculate the route once on start,
  then 'forget' all the trivial and 90% of the non-trivial waypoints.
  This both serves as a proof that a solution exists, and provides a set
  of points reaching which, in sequence, guarantees the route will take
  us to the destination.

  Once the vehicle reaches a waypoint, the route to the next one is
  calculated again, from scratch. It's known to exist and be correct
  (because we did it once, and it was correct), it doesn't put too much
  strain on the CPU and the memory (it's only about 10% the total route
  length) and it doesn't need to go into permanent storage (restarting
  from any point along the path is just a subset of the solution
  connecting two saved waypoints).

Now for the actual question:

The pathfinding algorithm follows a sparse set of waypoints which by themselves are not nearly sufficient as a route, but allow for easy, efficient  calculation of the actual route, simultaneously guarantying its existence; they are a subset of the full solution.

Is this a heuristic approach?

(as I understand, normally, heuristics don't guarantee existence of a solution, and merely suggest more likely candidates. In this case, the 'hints' are taken straight out of an actual working solution, thus my doubts.)

Whether or not a label fits any particular instance depends on what you're using the label for. If something specific is riding on whether this approach is a 'heuristic' or not, that context is important.

But I wouldn't call this a heuristic, because I think of that as a shortcut for solving a problem, not either storing a solution or reformulating the problem (which is how I'd think of this).


How to choosing the random value for parameter w in deep learning network? I did watch the course DeepLearning of Andrew Ng and he told that we should create parameter w small like:

parameters['W' + str(l)] = np.random.randn(layer_dims[l], layer_dims[l - 1]) ** 0.001

But in the last application assignment. They choose another way:

layers_dims = [12288, 20, 7, 5, 1]
def initialize_parameters_deep(layer_dims):
    np.random.seed(3)
    parameters = {}
    L = len(layer_dims)

    for l in range(1, L):
        parameters['W' + str(l)] = np.random.randn(layer_dims[l], layer_dims[l - 1]) / np.sqrt(layer_dims[l - 1])
        parameters['b' + str(l)] = np.zeros((layer_dims[l], 1))
        assert (parameters['W' + str(l)].shape == (layer_dims[l], layer_dims[l - 1]))
        assert (parameters['b' + str(l)].shape == (layer_dims[l], 1))
    return parameters

And the result of this way is very good but if I choose w like the old above, It's just have 34% correct!

So do you can explain ?

Weights initialisation is strictly related to the vanishing/exploding gradient problem. For a complete explanation, please check this awesome page (also from deeplearning.ai). Here I'll summarise the main concepts:

initialising weight all to zero will cause all weights to have the same derivative value with respect to the loss function, hence the network would be incapable of learning anything.
initialising to zero the biases has no drawback since they are constants (no effect at all when computing the derivative anyway).
initialising weights with too high or too small values will lead to an exploding gradient (oscillating gradient values without convergence) or a vanishing gradient (small gradient values variation that converge before reaching the loss global minimum).

In order to avoid these problems, a method called Xavier initialisation has been proposed: the weights should be initialised in such a way that they will generate activations scores with a distribution that has:

Mean 0
Constant variance across layers (i.e. no vanishing/exploding)

The value "np.sqrt(layer_dims[l - 1])" pup up when imposing the second constrain.  For a formal prove check the page I linked. To grasp the concept just focus on the fact that the variance of the weights of a layer depends on the amount of nodes of the previous layer. This mean that for layers preceded by layers with a big amount of hidden nodes, the weights will be initialised with a small variance, and this is ok cause we don't want a small bunch nodes to have stronger influence on the subsequent activations. But in layers which instead are preceded by layers with a small amount of nodes, it's ok to allow the weights to vary more.


Why teaching only search algorithms in a short introductory AI course? I understood that the concept of search is important in AI. There's a question on this website regarding this topic, but one could also intuitively understand why. I've had an introductory course on AI, which lasted half of a semester, so of course there wasn't time enough to cover all topics of AI, but I was expecting to learn some AI theory (I've heard about "agents"), but what I actually learned was basically a few search algorithms, like:

BFS
Uniform-cost search
DFS
Iterative-deepening search
Bidirectional search

these search algorithms are usually categorized as "blind" (or "uninformed"), because they do not consider any information regarding the remaining path to the goal.

Or algorithms like:

Heuristic search
Best-first search
A
A*
IDA*

which usually fall under the category of "informed" search algorithms, because they use some information (i.e. "heuristics" or "estimates") about the remaining path to the goal.

Then we also learned "advanced" search algorithms (specifically applied to the TSP problem). These algorithms are either constructive (e.g., nearest neighbor), local search (e.g., 2-opt) algorithms or meta-heuristic ones (e.g., ant colony system or simulated annealing).

We also studied briefly a min-max algorithm applied to games and an "improved" version of the min-max, i.e. the alpha-beta pruning.

After this course, I had the feeling that AI is just about searching, either "stupidly" or "more intelligently".

My questions are:

Why would one professor only teach search algorithms in AI course? What are the advantages/disadvantages? The next question is very related to this.
What's more than "searching" in AI that could be taught in an introductory course? This question may lead to subjective answers, but I'm actually asking in the context of a person trying to understand what AI really is and what topics does it really cover. Apparently and unfortunately, after reading around, it seems that this would still be subjective.
Are there AI theories that could be taught in this kind of course?

There is lots of misconceptions about AI, specifically the idea that it is about making computers "think" like humans, simulating brain, the sci-fi robots taking over the world, all the philosophical discussions around brain as machine etc. The practice/reality of AI is about "using computing to solve problems" which basically means you take any problem, represent it as a computing problem and then design the algorithm to solve the computing problem which lead to solving the original problem. These search algorithms are general purpose algorithms for general purpose computing problems i.e any real world problem can be represented by these general purpose computing problem and then these algorithms can be used to solve them.

Remember, its about problem solving and its about general purpose computing problems that can represent any real world problem.


What are some well-known problems where neural networks don't do very well? Background: It's well-known that neural networks offer great performance across a large number of tasks, and this is largely a consequence of their universal approximation capabilities.  However, in this post I'm curious about the opposite:
Question: Namely, what are some well-known cases, problems or real-world applications where neural networks don't do very well?

Specification:
I'm looking for specific regression tasks (with accessible data-sets) where neural networks are not the state-of-the-art.  The regression task should be "naturally suitable", so no sequential or time-dependent data (in which case an RNN or reservoir computer would be more natural).

Here's a snippet from an article by Gary Marcus

  In particular, they showed that standard deep learning nets often fall
  apart when confronted with common stimuli rotated in three dimensional
  space into unusual positions, like the top right corner of this
  figure, in which a schoolbus is mistaken for a snowplow:

  .
  .
  .

  Mistaking an overturned schoolbus is
  not just a mistake, it’s a revealing mistake: it that shows not only
  that deep learning systems can get confused, but they are challenged
  in making a fundamental distinction known to all philosophers: the
  distinction between features that are merely contingent associations
  (snow is often present when there are snowplows, but not necessary)
  and features that are inherent properties of the category itself
  (snowplows ought other things being equal have plows, unless eg they
  have been dismantled). We’d already seen similar examples with
  contrived stimuli, like Anish Athalye’s carefully designed, 3-d
  printed foam covered dimensional baseball that was mistaken for an
  espresso

  Alcorn’s results — some from real photos from the natural
  world — should have pushed worry about this sort of anomaly to the top
  of the stack.

Please note that the opinions of the author are his alone and I do not necessarily share all of them with him.

Edit: Some more fun stuff

1) DeepMind's neural network that could play Breakout and Starcraft
saw a dramatic dip in performance when the paddle was moved up by a few pixels.

See: General Game Playing With Schema Networks

While in the latter, it performed well with one race of the character but not on a different map and with different characters.

Source

2)

  AlphaZero searches just 80,000 positions per second in chess and
  40,000 in shogi, compared to 70 million for Stockfish and 35 million
  for elmo.

What the team at Deepmind did was to build a very good search algorithm. A search algorithm that includes the capability to remember facets of previous searches to apply better results to new searches. This is very clever; it undoubtedly has immense value in many areas, but it cannot be considered general intelligence.

See: AlphaZero: How Intuition Demolished Logic (Medium)


Can experience replay be used for training after completing every single epoch? The DQN implements replay memory. Based on my research, I believe the replay memory starts to get used for training once there is enough experience in the memory buffer. This means the neural network gets trained while the game plays.

My question is, if I am to play the game 10000 epochs, store all the experiences and then train from the experiences would that have the same effect as training and while running through 10000 epochs? Is it frowned upon to do it this way? Are there any advantages?

My question is, if I am to play the game 10000 epochs, store all the experiences and then train from the experiences would that have the same effect as training and while running through 10000 epochs?

No, it will not. In general, for anything other than simple environments, this will give a worse result. The trouble is during those 10,000 epochs you will have been working with your initial behaviour policy to collect experience, which will not be close to optimal. Off-policy learning adjusts for that and attempts to learn the value function of current best-guess optimal policy. However, this is not perfect, and it will learn better the closer your behaviour policy is to the target policy. An initial random policy will in many cases be too different from the target policy for learning to be effective.

There are two main causes of imperfection in off-policy learning:

Variance. The more discrepancy between behaviour and target policies there is, the higher the variance will be in the sampled values, and more samples will be required to get the same accuracy.
Sampling bias. The distribution of observed states and actions will affect function approximators such as neural networks, which reduce loss functions against an assumed population on input/output pairs. Unless adjusted somehow (and basic Q-learning in DQN does not adjust this), the population you train against will be from the behaviour policy.

  Are there any advantages?

It may be faster, and result in cleaner design, to separate data gathering and learning components. If you have a distributed architecture you can dedicate some machines to generating the experience and others to analysing it. However, this still benefits from routine updates to behaviour policies used to collect experience, based on learning so far.


Why is the perplexity a good evaluation metric for chatbots? A few papers I have come across say that BLEU is not an appropriate evaluation metric for chatbots, so they use the perplexity.
First of all, what is perplexity? How to calculate it? And why is perplexity a good evaluation metric for chatbots?

With perplexity you are trying to evaluate the similarity between the token (in your case probably sentences) distribution generated by the model and the one in the test data.

For instance, assuming you have $M$ sentences $s_1, \dots, s_M$, each with probability $P(s_i)$, the perplexity is $$2^{-l},$$ where $l = \frac{1}{M} \sum P(s_i) \log P(s_i)$ for $i \in [1 \dots M]$.

Note that while perplexity might be useful to capturing certain aspects of the model, it is by no means perfect, and, even if you are able to reach great perplexity scores, it will not necessarily translate to a good or even working chat bot.


With Monte Carlo off-policy learning what do we correct by using importance sampling? I do not understand the link of importance sampling to Monte Carlo off-policy learning.
We estimate a value using sampling on whole episodes, and we take these values to construct the target policy.
So, it is possible that in the target policy, we could have state values (or state action values) coming from different trajectories.
If the above is true, and if the values depend on the subsequent actions (the behavior policy), there is something wrong there, or else, better, something I do not understand.
Linking this question with importance sampling, do we use this ro value to correct this inconsistency?
Any clarification is welcome.

We estimate a value using sampling on whole episodes, and we take this values to construct the target policy.

The crucial bit that you are missing is that there is no single value of $V(s)$ (or $Q(s,a)$) of a state (or a state action pair). These value functions are always defined with respect to some policy $\pi(a|s)$ and is given the notation of $V^{\pi}(s)$ (or $Q^{\pi}(s,a)$).
The off-policy learning problems are arising when you have two policies: the generation policy $\mu(a|s)$ and the target policy $\pi(a|s)$. Your MC sampling data came from an agent following $\mu$, while you want to improve your target policy $\pi$. It is pretty straightforward from here that you'd need to weight your calculations with factors like $\frac{\pi(a_i|s_i)}{\mu(a_i|s_i)}$ - that's what importance sampling is.


Encourage Deep Q to seek short-term reward I understand that gamma is an important factor in determining the rewards for a deep Q agent, however during testing of my network I am noticing that the agent is outputting more actions to "do nothing" as it learns more about it's given data set (financial stock data).

I have tried tweaking the gamma at different levels ranging from 0 - 1 and everywhere in between, however as the agent continues to learn, the times between actions is getting longer, and longer. This behaviour is undesirable, and it is preferable that the agent be making more often, short-term actions even if they result in reduced reward.

Does anyone have any tips on how to achieve this? Would a minus gamma have adverse effects on the network?

TLDR:
Time between actions becoming increasingly long over time, would prefer an agent that makes many actions over long-term ones.

Thanks for the responses - after taking into account the 'risk' the network experiences when working with unstable financial data I modified the activation layers from RelU to Sigmoid - this has lead to a considerable improvement to both results (profit!) and additionally the rate at which actions are carried out.

I will also edit the above question to provide further details.


Are there any artificial neuromorphic systems which can mimic the brain? I'd like to know whether there were attempts to simulate the whole brain, I'm not talking only about some ANN on microchips, but brain simulations.

Vernor Vinge said that if we can scan a human brain and then simulate it: We can run it at 1000 times the speed. The brain will be able to do 1000 years of thinking in 1 year ect.

At this stage in history we have the computer power.

The trouble lies in cutting a brain up and scanning the 100 billion neurons and 12 million kilometres of axons and 100000 billion synapses.
And piecing together the connectome from all the data.

Sebastian Seung at MIT is working on automating this scanning process with machine learning. By gathering training data from thousands of people playing his Eyewire game

Henry Markram in Europe tried to do something similar with his Blue Brain Project.
He attempted to simulate the neocortical column of a rat. The EU gave him a billion euros to do this. Unfortunately he has been heavily criticised by the Neuroscience community. They claim that we don't know the physiology  well enough to make a valid simulation.

Check out his Ted Talk.

In the 1970s Sydney Brenner achieved a full brain scan of a C Elegans worm. This worm has one of the simplest biological neural networks having only 302 neurons.
Here is a picture of its connectome:

An accurate computer simulation of this worm would be a major stepping stone to uploading a human brain.


How to train the images of various sizes? I am practicing with an image dataset which is having different dimensions. <img src="https://i.stack.imgur.com/FnnVo.png" alt="here is the sample images
" />
If I simply crop and pad them to 1024X1024(the original images having smallest width is around 300 and largest is around 2400 and widths and heights of the images are not the same) I am not getting good val_accuracy. It's just giving 49% accuracy.
How to do image processing to these images because the brightness of the images is also changing. My task is to classify them into 5 classes.

I would use the ImageDataGenerator.flow_from_directory. Documentation is [here.][1]Make a train directory. Within it create 5 subdirectories one for each class and give them the desired class name. Place the associated images into the 5 class sub directories. Then use something like the code below. Set the image size to something standard particularly is you are using transfer learning. For MobileNet standard size is 224 X 224. You can use the Image Data Generator to augment your data for example set horizontal flip =True. You can also vary the brightness and rotation of the images.
tf.keras.preprocessing.image.ImageDataGenerator(
    featurewise_center=False,
    samplewise_center=False,
    featurewise_std_normalization=False,
    samplewise_std_normalization=False,
    zca_whitening=False,
    zca_epsilon=1e-06,
    rotation_range=0,
    width_shift_range=0.0,
    height_shift_range=0.0,
    brightness_range=None,
    shear_range=0.0,
    zoom_range=0.0,
    channel_shift_range=0.0,
    fill_mode="nearest",
    cval=0.0,
    horizontal_flip=True,
    vertical_flip=False,
    rescale=None,
    preprocessing_function=None,
    data_format=None,
    validation_split=0.0,
    dtype=None)
#for flow from directory
.flow_from_directory(
    directory,
    target_size=(256, 256),
    color_mode="rgb",
    classes=None,
    class_mode="categorical",
    batch_size=32,
    shuffle=True,
    seed=None,
    save_to_dir=None,
    save_prefix="",
    save_format="png",
    follow_links=False,
    subset=None,
    interpolation="nearest",
).

  [1]: https://keras.io/api/preprocessing/image/


Image dataset for pomegranate plant disease I am implementing a project on pomegranate plant disease in Machine learning. I want a dataset of all kind images of a healthy and unhealthy part of the pomegranate plant. I got a dataset from Fruit360 but that is only for pomegranate fruits but need for leaves also. Is there anyone who knows any website, link, version control system repository and/or any resource from which I get a dataset for leaves.

You can find the dataset in the following links:

Pomegranate Disease Detection Using Image Processing
fruits 360 datasets


Detect named entities inside words using spaCy I am using rasa nlu for training an NLU system to detect intents and slots. Now, some languages have word endings with their nouns (like Finnish, e.g. "in Berlin" -> "Berliinissä"). I have tried to annotate the characters in the training data as entities, but then I run the model, it doesn't detect the characters inside the word. When those characters are a separate word, only then they're detected. I am unable to think of an implementation to effectively detect named entities within a word. Suggestions needed.

You could use a tool to decompose composite word. There are many open source softwares for various languages. For example : german-word-splitter


Measuring logs from a picture I am interested in learning more about the capabilities of AI, one of my ideas with practical functionality is using images of the rear of a log hauling truck to measure the individual logs using AI. The diameter and length of the log determine the board foot of that log. The diameter is the variable, with most lengths being the same 8ft or 16ft length. The AI would have to measure the diameter usually in 1-inch increments.
I assume you would have to train the AI using pictures of logs that were manually measured. The images would mostly be straight on rear shots, not much off angle. However not every tree grows perfectly round so it will need to be able to measure those to maximize board feet.

Using AI long term it would be interesting if it could suggest the best way to cut the log gaining the most lumber board feet from the log.

What would be used to develop something like this? Obviously some type of image recognition with learning.

A good machine learning object detector, would be helpful in finding the logs but classical image processing would be required to accurately find the log profiles.  Neural networks are generally not used for computing geometric dimensions.  In addition, you will need to have a calibration method so that you can convert pixels to some distance unit (e.g. inches, centimeters, etc.).

The Timbeter app seems to do what you need (see below).

This app (see below)  is interesting although it doesn't solve your problem.  It appears good at counting but finding the profiles to within an inch is a bigger problem.

This blog "HOW MANY LOGS CAN A WOODCHUCK COUNT?" goes over the image processing challenges of counting logs.  The author doesn't completely solve the problem.

There is a post on Stackoverflow, How to count white object on Binary Image?
 which makes an attempt.


Beginner - Object classification data in a neural network Imagine I wish to classify images of digits from 0-9.
Let's say I have trained the network to recognise '1'.
If I were to train the same network to recognise '2', wouldn't the backpropagation process mess up the weights and biases for '1'?

Or do programs like Tensorflow allocate a new layer of neural network for different object classification? Thanks.

The answer is yes, retraining a neural network on a new dataset will alter its internal state such that it would no longer give the same output as before (in your case 'messing it up').

There are techniques to allow you to reuse sections of trained networks for other problems (this is called 'Transfer Learning'). Transfer Learning involves freezing the weights/biases for parts of the already trained network while adding extra new layers to the end of it and training those (with Back-propagation) for your new problem. For your digit recognition problem you would normally just train the network on all the digits at the same time.

Tensorflow is lower level than the idea of neural networks and layers. It is a library that allows you to write mathematical pipelines that implement the tensor calculations required to build a neural network. To do the sort of thing that you mentioned you would use a higher level library such as Keras (https://www.tensorflow.org/guide/keras) which now comes conveniently bundled with Tensorflow.


What artificial intelligence strategies are useful for summarization? If I have a paragraph I want to summarize, for example:

  Ponzo and Fila went to the mall during the day. They walked for a long while, stopping at shops. They went to many shops. At first, they didn't buy anything. After going to a number of shops, they eventually bought a shirt, and a pair of pants.

Better summarized as:

  They shopped at the mall today and bought some clothes.

What is the best AI strategy to automate this process, if there is one? If there isn't, is it because it would be dependent on first having an external information resource that would inform any algorithm? Or is it because the problem is inherently contextual?

The following post has a bit of math, which I hope helps to explain the problem better. Unfortunately it seems, this SE site does not support LaTex:

Document summarization is very much an open problem in AI research. One way this task is currently handled is called "extractive summarization". The basic strategy is as follows: Split this document into sentences and we will present as a summary a subset of sentences which together cover all the important details in the post. Assign sentence $i$, $1 \leq i \leq n$, a variable $z_i \in \{ 0, 1 \}$, where $z_i = 1$ indicates the sentence was selected and $z_i = 0$ means the sentence was left out. Then, $z_i z_j = 1$ if and only if both sentences were chosen. We will also define the importance of each sentence $w_i$ for the sentence $i$ and interaction terms $w_{i,j}$ between sentences $i$ and $j$.

Let $x_i$ be the feature vectors for sentence $i$. $w_i = w(x_i)$ captures how important it is to include this sentence (or the topics covered by it) while $w_{i,j} = w(x_i,x_j)$ indicates the amount of overlap between sentences in our summary. Finally we put all this in a minimization problem:

\begin{aligned}
\underset{z_i}{\text{maximize }} & \sum_{i} w_i z_i - w_{i,j} z_i z_j \\
\text{s.t. } & z_i = 0 \text{ or } 1
\end{aligned}

This tries to maximize the total weight of the sentences covered and tries to minimize the amount of overlap. This is an integer programming problem similar to finding the lowest weight independent set in a graph and many techniques exist to solve such problems.

This design, in my opinion, captures the fundamental problems in text summarization and can be extended in many ways. We will discuss those in a bit, but first, we need to completely specify the features $w$. $w_i = w(x_i)$ could be a function only of the sentence $i$, but it could also depends on the place of the sentence in the document or its context (Is the sentence at the beginning of a paragraph? Does it share common words with the title? What is its length? Does it mention any proper nouns? etc)

$w_{i,j} = w(x_i,x_j)$ is a similarity measure. It measures how many repetitions there will be if we include both words in the sentence. It can be defined by looking at common words between sentences. We can also extract topics or concepts from each sentence and see how many are common among them, and use language features like pronouns to see if one sentence expands on another.

To improve the design, first, we could do keyphrase extraction, i.e. identify key phrases in the text and choose to define the above problem in terms of those instead of trying to pick sentences. That is a similar problem to what Google does to summarize news articles in their search results, but I am not aware of the details of their approach. We could also break the sentences up further into concepts and try to establish the semantic meaning of the sentences ( Ponzo and Fila are people P1 and P2, a mall is a place P, P1 and P2 went to the place P at time T (day). Mode of transport walking.... and so on). To do this, we would need to use a semantic ontology or other common-sense knowledge databases. However, all the parts of this last semantic classification problem are open and I have not seen anyone make satisfactory progress on it yet.

We could also tweak the loss function above so that instead of the setting the tradeoff between the sentence importance $w_i$ and the diversity score $w_{i,j}$ by hand, we could learn it from data. One way to do this is to use Conditional Random Fields to model the data, but many others surely exist.

I hope this answer explained the basic problems that need to be solved to make progress towards good summarization systems. This is an active field of research and you will find the most recent papers via Google Scholar, but first read the Wikipedia page to learn the relevant terms


What is the relationship between these two taxonomies for machine learning with neural networks? Could you please let me know which of the following classification of Neural Network's learning algorithm is correct?

The first one classifies it into:

supervised,
unsupervised and
reinforcement learning.

However, the second one provides a different taxonomy on page 34:

learning with a teacher (error correction learning including incremental and batch training),
learning without a teacher (reinforcement, competitive, and unsupervised learning)
memory-based learning, and
Boltzmann learning.

Could you please let me know which of the following classification of Neural Network's learning algorithm is correct?

  The first one classifies it into:

  supervised,
  unsupervised and
  reinforcement learning.

Those three forms of Machine Learning are not really different forms of a Neural Network learning algorithm (or, really, any learning algorithm); they are different forms of learning problems. They basically describe how much information we give our learning algorithm during a learning process / what kind of information we give it, regardless of what algorithm we're using to learn.

In supervised learning, we give very detailed information; we give example pairs of inputs + desired outputs.
In unsupervised learning, we still give many example inputs, but don't give any desired outputs. This can be, for example, because we simply don't know ourselves what the desired outputs would be. The most common example of unsupervised learning is learning to create clusters; we'd typically give some measure of similarity or distance between instances, and ask a learning algorithm to create clusters of instances for us such that instances within the same cluster are "close" to each other, and instances in different clusters are far away from each other. This is different from supervised learning because we do not directly tell the learning algorithm in which cluster each example instance would belong.
In reinforcement learning, we typically try to learn about policies (~= behaviours) in environments where an agent can take actions. We typically do not exactly know what the best complete policy would be, but can occasionally give "hints" (reinforcement) in the form of numerical rewards. This is not completely supervised learning because we don't tell exactly what the optimal action in a state would be. You can imagine this as giving a cookie to a dog if he's been a good boy.

Now, there also are indeed Supervised Learning / Unsupervised Learning / Reinforcement Learning algorithms, but it's generally a property of the problem first and foremost; once you know what type of problem you're trying to solve, you'll look for a matching algorithm that can handle that problem.

  However, the second one provides a different taxonomy on page 34:

  learning with a teacher (error correction learning including incremental and batch training),
  learning without a teacher (reinforcement, competitive, and unsupervised - memory-based learning, and
  Boltzmann learning.

Honestly, this seems a bit all-over-the-place to me. Learning with a teacher vs. learning without a teacher can be viewed as supervised vs unsupervised learning above. I suppose Reinforcement Learning would be kind of in-between.

What they describe as memory-based learning is also often referred to as instance-based learning. This is suddenly not anymore about properties of learning problems, this is a type of learning algorithm. I'm not aware of instance-based learning being common in Neural Networks at all, indeed the example given in your link (the most common example) is the k-nearest neighbours algorithm, which doesn't really have any relation with Neural Networks. This is normally used for Supervised Learning problems.

Boltzmann learning is a particular kind of learning algorithms for specific types of Neural Networks (with a specific architecture), and generally associated with unsupervised learning (or "generative" learning, learning probability distributions for given input data).


What sort of game problems can neural networks trained/evolved with evolutionary algorithms solve, and how are they typically implemented? I'm interested mostly in the application of AI in gaming; in case this adjusts the way you answer, but general answers are more than welcome as well.
I was reading up on Neural Networks and combining them with Genetic Algorithms; my high-level understanding is that the Neural Networks are used to produce a result from the inputs, and the Genetic Algorithm is employed to constantly adjust the weights in the Neural Network until a good answer is found.
The concept of a Genetic Algorithm randomly mutating the weights on the inputs to a Neural Network makes sense to me, but I don't understand where this would be applied with respect to gaming.
For example, if I had some simple enemy AI that I want to have adapted to the player's play-style, is this a good opportunity to implement the AI as a Genetic-Algorithm combined with a Neural Network?
With these different suitable applications, how does one go about deciding how to encode the problem in such a way that it can be mutated by the Genetic Algorithm and serve as suitable on/off inputs to a Neural Network (actually, are Neural Networks always designed as on-off signals?)?

Without going in too much detail on how exactly Neural Networks and Generic Algorithms work, I can tell you that both the algorithms are not good candidates for computer games.  They work well in scientific environments where the system is "trained" on a huge data set to adjust the "weights" (variables) for a given problem.  This "training" process requires a lot of processing power, time and a large data set.

Computer games, however either needs to run in real-time (no time for training) or turn-based (not enough data for training).

Another problem is that computer games need to free up as much as possible system resources for physics, graphics, sounds and the user interface to improve the player's experience so game developers usually use other lighter techniques (like a rule-based system) to create the illusion of an AI player.


Why do we use a last-in-first-out queue in depth-first search? Why do we use a last-in-first-out (LIFO) queue in the depth-first search algorithm?

In the breadth-first search algorithm, we use a first-in-first-out (FIFO) queue, so I am confused.

We use the LIFO  queue, i.e. stack, for implementation of the depth-first search algorithm because depth-first search always expands the deepest node in the current frontier of the search tree. The search proceeds immediately to the deepest level of the search tree, where the nodes have no successors. As those nodes are expanded, they are dropped from the frontier, so then the search "backs up" to the next deepest node that still has unexplored successors.

A LIFO queue means that the most recently generated node is chosen for expansion. This must be the deepest unexpanded node because it is one deeper than its parent, which, in turn, was the deepest unexpanded node when it was selected.


How can I convert the probability score between 0 to 1 to another format? I have trained a multi-class CNN model using fastai. The model splits out probabilites for each of the three classes, which, of course, sum up to 1. The class with highest probability becomes the predicted class.

Is there any way I can convert them into 0 to 1 scale, where near to 0 value would mean class 1, near to 0.5 would mean class 2 and near to 1 would mean class 3?

You could maybe do something like this, it's a bit hackish
\begin{equation}
y = C_1\cdot 1 + C_2 \cdot 0.5 + C_3 \cdot 0
\end{equation}
$y$ represents the output and its bounded $\in [0, 1]$. $C_i$ is probability for class $i$. This way when $C_1 \approx 1, C_2 \approx 0, C_3 \approx 0$ you have
\begin{equation}
 y \approx 1\cdot 1 + 0.5 \cdot 0 + 0 \cdot 0 \approx 1
\end{equation}
when $C_1 \approx 0, C_2 \approx 1, C_3 \approx 0 $ you have
\begin{equation}
 y \approx 1\cdot 0 + 0.5 \cdot 1 + 0 \cdot 0 \approx 0.5
\end{equation}
and when $C_1 \approx 0, C_2 \approx 0, C_3 \approx 1 $ you have
\begin{equation}
 y \approx 1\cdot 0 + 0.5 \cdot 0 + 0 \cdot 1 \approx 0
\end{equation}


Do AlphaZero/MuZero learn faster in terms of number of games played than humans? I don't know much about AI and am just curious.
From what I read, AlphaZero/MuZero outperform any human chess player after a few hours of training. I have no idea how many chess games a very talented human chess player on average has played before he/she reaches the grandmaster level, but I would imagine it is a number that can roughly be estimated. Of course, playing entire games is not the only training for human chess players.
Nonetheless, how does this compare to AI? How many games do AI engines play before reaching the grandmaster level? Do (gifted) humans or AI learn chess faster?

According to Table S3 of the AlphaZero paper (p. 15)

AlphaZero was trained for 9 hours and, during these 9 hours, it played 44 million games of chess.
According to this Wikipedia article, the longest human lifespan is that of Jeanne Calment, who lived to age 122 years and 164 days.
Let's assume that humans cannot live more than 123 years (which is a reasonable assumption, although this record could eventually be broken). Let's also assume that a chess game lasts at least 10 minutes, which means that you can play at most 6 games in 1 hour, which means that you can play at most $6*24 = 144$ games in one day (assuming that you never sleep, which is, of course, impractical, but I'm just trying to show you an upper bound). Let's say that a year has 365 days. So, here is roughly the maximum number of games that a human could play
$$
6*24*365*123 = 6464880 \tag{1}\label{1},
$$
which is smaller than 44 million games by a factor of more than 6, i.e. any human could at most play 1/6 of the games that AlphaZero played, and \ref{1} is a very loose upper bound that doesn't take into account that humans need to sleep, eat, and do many other things.
So, humans learn to play chess a lot slower than AlphaZero. This is not surprising at all, given that computers can perform calculations a lot faster than us (that's why they are called computers), and this has been the case for many years. We (humans) just made computers make the right calculations for them to approximately play chess better than us. That's it.


What are the challenges faced by using NLP to convert mathematical texts into formal logic? From what I've figured

(a) converting mathematical theorems and proofs from English to formal logic is a straightforward job for mathematicians with sufficient background, except that it takes time.

(b) once converted to formal logic, computer verification of the proof becomes straightforward.

If we can automate (a), a lot of time and intellectual labour (that could be dedicated elsewhere) is saved in doing (b) on published research papers.

Note that if solving (a) in its entirety is hard, we could expect the mathematicians to meet the computer system halfway and avoid writing lengthy English paras that are hard to convert. If it becomes doable enough, submitting a formal logical version of your paper could even become a standard procedure that is expected.

Additional benefit of solving (a) would be to do the process in reverse: mathematicians could delegate smaller tasks and lemmas (both trivial and non-trivial tasks) to an automated theorem prover (ATP). Assisted theorem proving will become more popular and boost productivity, maybe even surprise us once in a while by coming up with proofs that the paper writer couldn't. This is further of value if we predict a sharp upward trajectory of the capability of ATPs in the future. If anything, this could be self-fulfilling, as the demonstration of potential for good ATPs combined by a large corpus of proofs and problems in formal logical format could drive an increase in research on ATPs.

Forgive me if I sound like a salesman, but how doable is this? What will be the main challenges faced in developing NLP-based AI to convert papers, and how tractable are these challenges given today's state of the field?

P.s. I understand that proofs generated by ATPs are often hard to understand intuitively and can end up proving results without clearly exposing the underlying proof method used. But it is still a benefit to be able to use the final results

I can see several challenges, and the list below is not exhaustive:

i. The main problem is how to model a problem of translating a language test into a formal language. It will probably be something like the automatic translators, but with some guarantees that the proof semantics will be preserved. If you are more interested in this path, I recommend researching what PAC, Information Theory, Computational Proof theory, Complexity theory can contribute to this modeling.

ii. Another problem is how to get the data reliable. You commented that as people used it they would generate this data. But the problem is not just collecting the data. How much you will trust the data and how you will measure the model's performance in translation.

iii. Another problem is more humane, how do you get mathematicians to use such a system? And how to make the model self-explainable.

I believe that this is one of the most difficult problems in machine learning. I once saw this video a while ago and I don't know if it can help anything. I also recommend the stack exchange of theoretical computer science, there you will probably have a more complete answer.


What is the "semantic level"? I am reading the paper Hierarchical Attention-Based Recurrent Highway Networks for Time Series Prediction (2018) by Yunzhe Tao et al.

In this paper, they use several times the expression "semantic levels". Some examples:

HRHN can adaptively select the relevant exogenous features in different semantic levels
the temporal information is usually complicated and may occur at different semantic levels
The encoder RHN reads the convolved features $(w_1,w_2,···,w_{T−1})$ and models their temporal dependencies at different semantic levels
Then an RHN is used to model the temporal dependencies among convolved input features at different semantic levels

What is the semantic level?

In language theory, there are generally several admitted levels that can be studied in relation with one another or independently. The semantic level is the one dealing with the meaning of the text ("semantic" comes from the greek and means "to signify"). The semantic level is therefore generally independent from the syntax and even the language used to convey the message.
Here is an interesting picture I found on the internet to illustrate my point.

EDIT: I took some time reading the paper. I think "semantic levels" refers here to the different neural networks layers used for the exogenous features.

Here is a modified version of their figure I've drawned to make it clearer:

In particular, from what I have understood so far, the attention coefficients apply to the whole semantic level (which I find not really clearly indicated on their figure)

The LHS of their figure would then be better described by this new one:

Hope this helps!


Image Recognition and Orientation Detection Hypothetically, the symbol (Triangle) is sticked to an item and i need to find and recognize that symbol and try to calculate the orientation of the item it is sticked into. In degrees. How would you guys suggest i approach this problem? Do i still need NN for this? Thank you :) I just need to hear other people's thoughts.

This is a quite trivial problem and using a NN would be overkill for this. The symbol you used (triangle) is problematic, because it looks exactly the same in 3 different positions (rotated 0°, 120° and 240°).

The easiest implementation would be using 2 distinguishable points with a fixed distance. To distinguish them you could color code them or give them different sizes. Those 2 points should be easy to identify in your picture using a pattern matching algorithm. Once you have the 2 points you calculate the vector between them and can then calculate the orientation of the vector, which basically tells you the orientation of the whole object your symbol is put on.

Edit - Answering the first comment:

The basic idea is the same, no matter what symbol you use. In case of the "L", your 2 points would be the end of both lines. You can easily find the symbol, calculate the vector between both points and use it to calculate the orientation using this vector. This approach can be used for every symbol that doesn't have radial symmetry. You can always define 2 points in this symbol and apply the method described.

The reason why a NN is overkill is the simplicity of the problem. NNs are mostly used for problems that are easy to solve for humans but very hard to implement with a simple algorithm. Tasks like classifying images, identifying hand written or distorted letters and so on. But your problem is easily solvable with simple algorithms, therefore the complexity of a NN is not necessary.

This doesn't mean that your task cannot be implemented with an NN. Actually it might be quite an interesting project to learn more about NNs. I would say your best bet is implementing it as a CNN, since this type of NN is good at detecting features in images like the direction of your symbol. I can recommend this course from Stanford University, if you want to learn more about CNNs.

For a real world application, there is no need to take the complex route using NNs. My suggested algorithm can be implemented with a few dozen lines of code and should be very robust.


Why a single trajectory can be used to update the policy network $\theta$ in A3C? The Deep RL bootcamp on policy gradient techniques gives the update equation for the policy network in A3C as

$\theta_{i+1} = \theta_i + \alpha \times 1/m \sum_{k=1}^m\sum_{t=0}^{H-1}\nabla_{\theta}log\pi_{\theta_i}(u_t^{(k)} | s_t^{(k)})(Q(s_t^{(k)},u_t^{(k)}) - V_{\Phi_i}^\pi(s_t^{(k)}))    $

However in the actual A3C paper, the gradient update is based on a single trajectory and there is no averaging of the gradient over $m$ trajectories as defined in the video ? The simple action-value actor-critic algorithm also does not seem to require an averaging over m trajectory.

I guess the gradient of the expectation of the Utility function, $\nabla_{\theta}J(\theta)$ in policy gradient methods where $\nabla_{\theta}J(\theta) = E_{\tau \sim p(\tau ; \theta)}[r(\tau)\nabla_{\theta}log p(\tau;\theta)]$ can be approximated using a single sample trajectory as shown in a deep reinforcement learning lecture by stanford where $J(\theta) \approx \sum_{t > 0}r(\tau)\nabla_{\theta}log\pi_{\theta}(a_t |s_t)$ and an average over sampled trajectories is not needed to compute the gradient for $\theta$ update in the direction of gradient.


Can ConvNets be used for real-time object recognition from video feed? Convolutional neural network are leading type of feed-forward artificial neural network for image recognition. Can they be used for real-time image recognition for videos (frame by frame), or it takes too much processing (assuming they're written in C-like language)?

For example for classification of type of animals based on the training from huge dataset.

We are getting there, with as usual some trade-off between quality and speed.
For example Lecture 8: Spatial Localization and Detection lecture shows some benchmarks (mAP = Mean Average Precision, higher is better; FPS = frame per second):


Why does a one-layer hidden network get more robust to poor initialization with growing number of hidden neurons? In a nutshell: I want to understand why a one hidden layer neural network converges to a good minimum more reliably when a larger number of hidden neurons is used. Below a more detailed explanation of my experiment:

I am working on a simple 2D XOR-like classification example to understand the effects of neural network initialization better. Here's a visualisation of the data and the desired decision boundary:

Each blob consists of 5000 data points. The minimal complexity neural network to solve this problem is a one-hidden layer network with 2 hidden neurons. Since this architecture has the minimum number of parameters possible to solve this problem (with a NN) I would naively expect that this is also the easiest to optimise. However, this is not the case.

I found that with random initialization this architecture converges around half of the time, where convergence depends on the signs of the weights. Specifically, I observed the following behaviour:

w1 = [[1,-1],[-1,1]], w2 = [1,1] --> converges
w1 = [[1,1],[1,1]],   w2 = [1,-1] --> converges
w1 = [[1,1],[1,1]],   w2 = [1,1] --> finds only linear separation
w1 = [[1,-1],[-1,1]], w2 = [1,-1] --> finds only linear separation

This makes sense to me. In the latter two cases the optimisation gets stuck in suboptimal local minima. However, when increasing the number of hidden neurons to values greater than 2, the network develops a robustness to initialisation and starts to reliably converge for random values of w1 and w2. You can still find pathological examples, but with 4 hidden neurons the chance that one "path way" through the network will have non-pathological weights is larger. But happens to the rest of the network, is it just not used then?

Does anybody understand better where this robustness comes from or perhaps can offer some literature discussing this issue?

Some more information: this occurs in all training settings/architecture configurations I have investigated. For instance, activations=Relu, final_activation=sigmoid, Optimizer=Adam, learning_rate=0.1, cost_function=cross_entropy, biases were used in both layers.

You grasped a bit of the answer.

  In the latter two cases the optimisation gets stuck in suboptimal local minima.

When you have only 2 dimensions, a local minima exists. When you have more dimensions, this minima gets harder and harder to reach, as its likelihood decreases. Intuitively, you have a lot more dimensions through which you can improve than if you only had 2 dimensions.

The problem still exists, even with 1000 neurons you could find a specific set of weights which was a local minimum. However, it just becomes so much less likely.


Are tabular reinforcement learning methods obsolete (or getting obsolete)? While learning RL, I came across some problems where the Q-matrix that I need to make is very very large. I am not sure if it is ever practical. Then I research and came to this conclusion that using the tabular method is not the only way, in fact, it is a very less powerful tool as compared to other methods such as deep RL methods.
Am I correct in this understanding that with the increasing complexity of problems, tabular RL methods are getting obsolete?

Am I correct in this understanding that with the increasing complexity of problems, tabular RL methods are getting obsolete?

Individual problems don't get any more complex, but the scope of solvable environments increases due to research and discovery of better or more apt methods.
Using deep RL methods with large neural nets can be a lot less efficient for solving simple problems. So tabular methods still have their place there.
Practically, if your state/action space (number of states times number of actions) is small enough to fit a Q table in memory, and it is possible to visit all relevant state/action pairs multiple times in a relatively short time, then tabular methods offer guarantees of convergence that approximate methods cannot. So tabular approaches are often preferred if they are appropriate.
Many interesting, cutting edge problems that are relevant to AI, such as autonomous robots acting in the real world, do not fit the tabular approach. In that sense, the approach is "obsolete" in that it no longer provides challenging research topics for practical AI (there are still unanswered theoretical questions, such as proof of convergence for Monte Carlo control).
It is still worth understanding tabular value-based methods in detail, because they form the foundations of the more complex deep learning methods. In some sense they represent ideal solutions that deep RL tries to approximate, and the design of tabular solutions can be the inspiration for changes and adjustments to neural-network methods.


Are filters fixed or learned? No matter what I google or what paper I read, I can't find an answer to my question. In a deep convolutional neural network, let's say AlexNet (Krizhevsky, 2012), filters' weights are learned by means of back-prop.
But how are kernels themselves selected? I know kernels had been used in image processing long before CNNs, hence I'd imagine there would be a set of filters based on kernels (see, for example, this article) that are proven to be effective for edge detection and the likes.
Reading around the web, I also found something about "randomly generated kernels". Does anyone know if and when this practice is adopted?

What are filters in image processing?
In the context of image processing (and, in general, signal processing), the kernels (also known as filters) are used to perform some specific operation on the image. For example, you can use a Gaussian filter to smooth the image (including its edges).
What are filters in CNNs?
In the context of convolutional neural networks (CNNs), the filters (or kernels) are the learnable parameters of the model.
Before training, the kernels are usually randomly initialised (so they are not usually hardcoded). During training, depending on the loss or error of the network (according to the loss function), the kernels (or filters) are updated, so that to minimise the loss (or error). After training, they are typically fixed. Incredibly, the filters learned by CNNs can be similar to the Gabor filter (which is thought to be related to our visual system [1]). See figure 9.19 of chapter 9 (p. 365) of the Deep Learning book by Goodfellow et al.
The number of kernels that are applied to a given input (and more than one kernel is often applied) in a CNN is a hyper-parameter.
What are the differences and similarities?
In both contexts, the words "kernel" and "filter" are roughly synonymous, so they are often used interchangeably. Furthermore, in both cases, the kernels are related to the convolution (or cross-correlation) operation. More specifically, the application of a filter, which is a function $h$, to an input, which is another function $f$, is equivalent to the convolution of $f$ and $h$. In mathematics, this is often denoted by $f \circledast h = g$, where $\circledast$ is the convolution operator and $g$ is the result of the convolution operation and is often called the convolution (of $f$ and $h$). In the case of image processing, $g$ is the filtered image. In the case of CNNs, $g$ is often called an activation map.
Further reading
Take a look at this and this answers for more details about CNNs and the convolution operation, respectively.
You may also want to have a look at Visualizing what ConvNets learn
 for some info about the visualization of the kernels learned during the training of a CNN.


TensorFlow training evaluation I'm having trouble with accuracy evaluation at the end of Session.

Training process looks like this:

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    epochs = 10
    batch_size = 128

    for e in range(epochs):
        shuffle_indices = np.random.permutation(np.arange(len(inputx)))
        X_train = inputx[shuffle_indices]
        y_train = inputy[shuffle_indices]

        epoch_loss = 0
        for i in range(int(len(inputx) // batch_size)):
            start = i * batch_size
            batch_x = X_train[start:start + batch_size]
            batch_y = y_train[start:start + batch_size]

            _, c = sess.run([optimizer, cost], feed_dict={x: batch_x, y: batch_y})
            epoch_loss += c

        print(e + 1, '/', epochs, 'loss:', epoch_loss)

At the end of this session I want to evaluate reached accuracy by calculating mean value of percentual differences between prediction and testy for each of 3 output layer neurons.

percentual_differences_for_neuron1 = # insert your advice here
accuracy_for_neuron1 = tf.reduce_mean(percentual_differences_for_neuron1)
print('Accuracy of neuron 1 is', '''another advice''')

What's the best way of doing this in TensorFlow?

Suppose you are dealing with a classification problem you can simply define

def error_rate_classification(p, t):
    return np.mean(p != t)

where p and t stand for predictions and targets resp.

If you deal with a regression problem the you should modify a bit this function. For example :

def error_rate_regression(p, t):
    return np.mean(np.abs(p-t) < 1e-2) # the criteria is not general

Note that for regression problem assessing the accuracy of the prediction is not as simple as it is in classification problem

Edit

In regression case, you can use the R2 regression standard metrics. Suppose yi is the ground truth value of the i-th element of your test (or validation) set and f(xi) is your i-th input prediction. The R2 is then

R2 = 1 - sum( (yi - f(xi))**2 )  / sum( (yi - y_mean)**2 )

where y_mean is the mean of the ground truth values of the test ( or validation) test.  You can find other examples in the scikit-learn regression metrics here


What do prediction/mean and label/mean represent in this TensorFlow code? I am pretty much a beginner in Tensorflow and simply follow a tutorial. There is no problem with my code, but I have a question regarding the output

accuracy: 0.95614034
accuracy_baseline: 0.6666666
auc: 0.97714674
auc_precision_recall: 0.97176754
average_loss: 0.23083039
global_step: 760
label/mean: 0.33333334
loss: 6.578666
prediction/mean: 0.3428335

I would like to know what does "prediction/mean" and "label/mean" represent?

All of these could be problem specific (except maybe accuracy). Most of it is documented here:

accuracy: Percentage of correct number of classifications
accuracy_baseline: Accuracy baseline based on labels mean. This is the best the model could do by always predicting one class. (source)
AUC or Area Under the (ROC) Curve is quite complicated, but tells you something about the true/false positive rate. In short: the AUC is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one.
auc_precision_recall: Is the percentage of relevant intstances, among the retrieved instances, that have been retrieved over the total amount of relevant instances.
average_loss: You're usually minimizing some function, and this is likely the average value of that function given the current batches.
loss: The current value of the loss (as above). Either the sum of the losses, or the loss of the last batch.
global_step: Number of iterations.
label/mean and prediction/mean: Not really sure, but I suspect that if you have two classes then the label/mean is the mean of the value labels, whilst prediction/mean could be the value of the corresponding predictions. (two classes could give you a value between 0 and 1)