How is the gradient of the loss function in DQN derived? In the original DQN paper, page 1, the loss function of the DQN is

$$
L_{i}(\theta_{i}) = \mathbb{E}_{(s,a,r,s') \sim  U(D)} [(r+\gamma \max_{a'} Q(s',a',\theta_{i}^{-}) - Q(s,a;\theta_{i}))^2]
$$

whose gradient is presented (on page 7)

$$\nabla_{\theta_i} L_i(\theta_i) = \mathbb{E}_{s,a,r,s'} [(r+\gamma \max_{a'}Q(s',a';\theta_i^-) - Q(s,a;\theta_i))\nabla_{\theta_i}Q(s,a;\theta_i)] $$

But why is there no minus (-) sign if $-Q(s,a;\theta_i)$ is parameterized by $\theta_i$ and why is the 2 from power gone?

In general, if you have a composite function $h(x) = g(f(x))$, then $\frac{dh}{dx} = \frac{d g}{df} \frac{d f}{dx}$. In your case, the function to differentiate is
$$L_{i}(\theta_{i}) = \mathbb{E}_{(s,a,r,s') \sim  U(D)} \left[ \left(r+\gamma \max_{a '} Q(s',a',\theta_{i}^{-}) - Q(s,a;\theta_{i}) \right)^2 \right]$$
So, we want to calculate $\nabla_{\theta_i} L_{i}(\theta_{i})$, which is equal to
$$\nabla_{\theta_i}  \mathbb{E}_{(s,a,r,s') \sim  U(D)} \left[ \left(r+\gamma \max_{a '} Q(s',a',\theta_{i}^{-}) - Q(s,a;\theta_{i}) \right)^2 \right] \label{2} \tag{2} $$
For clarity, let's ignore the iteration number $i$ in \ref{2}, which can thus more simply be written as
$$\nabla_{\theta}  \mathbb{E}_{(s,a,r,s') \sim  U(D)} \left[ \left(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta ) \right)^2 \right] \label{3} \tag{3} $$
The subscript of the expected value operator $\mathbf{e}=(s,a,r,s') \sim  U(D)$ means that the expected value is being taken with respect to the multivariate random variable, $\mathbf{E}$ (for experience), whose values (or realizations) are $\mathbf{e}=(s,a,r,s')$, and that follows the distribution $U(D)$ (a uniform distribution), that is, $\mathbf{e}=(s,a,r,s')$ are uniformly drawn from the experience replay buffer, $D$ (or $\mathbf{E} \sim U(D)$). However, let's ignore or omit this subscript for now (because there are no other random variables in \ref{3}, given that $\gamma$ and $a'$ should not be random variables, thus it should not be ambiguous with respect to which random variable the expectation is being calculated), so \ref{3} can be written as
$$\nabla_{\theta}  \mathbb{E} \left[ \left(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta ) \right)^2 \right] \label{4} \tag{4} $$
Now, recall that, in the case of a discrete random variable, the expected value is a weighted sum. In the case of a continuous random variable, it is an integral. So, if $\mathbf{E}$ is a continuous random variable, then the expectation \ref{4} can be expanded to
$$
\int_{\mathbb{D}} {\left(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta )\right)}^2 f(\mathbf{e}) d\mathbf{e}
$$
where $f$ is the density function associated with $\mathbf{E}$ and $\mathbb{D}$ is the domain of the random variable $\mathbf{E}$.
The derivative of an integral can be calculated with the Leibniz integral rule. In the case the bounds of the integration are constants ($a$ and $b$), then the Leibniz integral rule reduces to
$${\displaystyle {\frac {d}{dx}}\left(\int _{a}^{b}f(x,t)\,dt\right)=\int _{a}^{b}{\frac {\partial }{\partial x}}f(x,t)\,dt.}$$
Observe that the derivative is taken with respect to the variable $x$, while the integration is taken with respect to variable $t$.
In our case, the domain of integration, $\mathbb{D}$, is constant because it only represents all experience in the dataset, $\mathcal{D}$. Therefore, the gradient in \ref{4} can be written as
\begin{align}
\nabla_{\theta} \int_{\mathbb{D}} \left(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta )\right)^2 f(\mathbf{e}) d\mathbf{e}
&= \\
\int_{\mathbb{D}} \nabla_{\theta} \left( \left( r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta )\right)^2  f(\mathbf{e}) \right) d\mathbf{e}
&=\\
\mathbb{E} \left[ \nabla_{\theta} { \left(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta) \right)}^2 \right] \label{5}\tag{5}
\end{align}
Recall that the derivative of $f(x)=x^2$ is $f'(x)=2x$ and that the derivative of a constant is zero. Note now that the only term in \ref{5} that contains $\theta$ is $Q(s,a; \theta)$, so all other terms are constant with respect to $\theta$. Hence, \ref{5} can be written as
\begin{align}
\mathbb{E} \left[ \nabla_{\theta} {(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))}^2 \right]
&=\\
\mathbb{E} \left[ 2 {(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))} \nabla_{\theta} \left( {r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta)} \right) \right]
&=\\
\mathbb{E} \left[ 2 {(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))}  \left( {\nabla_{\theta} r + \nabla_{\theta} \gamma \max_{a'} Q(s',a',\theta^{-}) - \nabla_{\theta} Q(s,a; \theta)} \right) \right]
&=\\
\mathbb{E} \left[ - 2 {(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))} {\nabla_{\theta} Q(s,a; \theta)} \right]
&=\\
-2 \mathbb{E} \left[ {(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))}{\nabla_{\theta} Q(s,a; \theta)} \right]
\end{align}
The $-2$ disappears because it can be absorbed by the learning rate.


The relation between the human face perception and the keyword spotting in speech recognition? In our brain there is an area, near the fusiform gyrus and the occipital area, to recognize the human face. And in speech recognition, there is a technique named keyword spotting. Then I am wondering 1) if there is an area in our brain for the similar function to recognize our names; 2) if a special face recognition function should be considered when we are building a robot?

Specialised neural circuitry to recognise faces is common in all our closest animal relatives. This means that it is likely an evolutionary adaptation that is many millions of years old. Babies can pay attention to faces basically from the moment they are born. There are certain types of brain damage that make it impossible to recognise faces.

Being addressed by an individual name (as opposed to broadcasting an individual signature) is likely a much more recent phenomenon. I have never heard of babies having any special propensity to pick up on names and I have never heard of brain damage that makes it selectively impossible to hear your name.

Also, face recognition can be hardwired because faces always look roughly the same. Names are always different. If you have to learn them anyway, it is more likely that it just happens with you general auditory brain stuff.

And it strikes me as likely that the speed that comes from special circuitry (recognising something early in the cortical hierarchy) is a lot more essential when it comes to faces (and snakes and spiders) then your own name.

So, while humans pay close attention to every sound that might be their name, I doubt there is specialised neural circuitry for that.


What is the difference between training and testing in reinforcement learning? In reinforcement learning (RL), what is the difference between training and testing an algorithm/agent? If I understood correctly, testing is also referred to as evaluation.

As I see it, both imply the same procedure: select an action, apply to the environment, get a reward, and next state, and so on. But I've seen that, e.g., the Tensorforce RL framework allows running with or without evaluation.

What is reinforcement learning?
In reinforcement learning (RL), you typically imagine that there's an agent that interacts, in time steps, with an environment by taking actions. On each time step $t$, the agent takes the action $a_t \in \mathcal{A}$ in the state $s_t \in \mathcal{S}$, receives a reward (or reinforcement) signal $r_t \in \mathbb{R}$ from the environment and the agent and the environment move to another state $s_{t+1} \in \mathcal{S}$, where $\mathcal{A}$ is the action space and $\mathcal{S}$ is the state space of the environment, which is typically assumed to be a Markov decision process (MDP).
What is the goal in RL?
The goal is to find a policy that maximizes the expected return
(i.e. a sum of rewards starting from the current time step). The policy that maximizes the expected return is called the optimal policy.
Policies
A policy is a function that maps states to actions. Intuitively, the policy is the strategy that implements the behavior of the RL agent while interacting with the environment.
A policy can be deterministic or stochastic. A deterministic policy can be denoted as $\pi : \mathcal{S} \rightarrow \mathcal{A}$. So, a deterministic policy maps a state $s$ to an action $a$ with probability $1$. A stochastic policy maps states to a probability distribution over actions. A stochastic policy can thus be denoted as $\pi(a \mid s)$ to indicate that it is a conditional probability distribution of an action $a$ given that the agent is in the state $s$.
Expected return
The expected return can be formally written as
$$\mathbb{E}\left[ G_t \right] = \mathbb{E}\left[ \sum_{i=t+1}^\infty R_i \right]$$
where $t$ is the current time step (so we don't care about the past), $R_i$ is a random variable that represents the probable reward at time step $i$, and $G_t = \sum_{i=t+1}^\infty R_i $ is the so-called return (i.e. a sum of future rewards, in this case, starting from time step $t$), which is also a random variable.
Reward function
In this context, the most important job of the human programmer is to define a function $\mathcal{R}: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$, the reward function, which provides the reinforcement (or reward) signal to the RL agent while interacting with the environment. $\mathcal{R}$ will deterministically or stochastically determine the reward that the agent receives every time it takes action $a$ in the state $s$. The reward function $R$ is also part of the environment (i.e. the MDP).
Note that $\mathcal{R}$, the reward function, is different from $R_i$, which is a random variable that represents the reward at time step $i$. However, clearly, the two are very related. In fact, the reward function will determine the actual realizations of the random variables $R_i$ and thus of the return $G_i$.
How to estimate the optimal policy?
To estimate the optimal policy, you typically design optimization algorithms.
Q-learning
The most famous RL algorithm is probably Q-learning, which is also a numerical and iterative algorithm. Q-learning implements the interaction between an RL agent and the environment (described above). More concretely, it attempts to estimate a function that is closely related to the policy and from which the policy can be derived. This function is called the value function, and, in the case of Q-learning, it's a function of the form $Q : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$. The name $Q$-learning derives from this function, which is often denoted as $Q$.
Q-learning doesn't necessarily find the optimal policy, but there are cases where it is guaranteed to find the optimal policy (but I won't dive into the details).
Of course, I cannot describe all the details of Q-learning in this answer. Just keep in mind that, to estimate a policy, in RL, you will typically use a numerical and iterative optimization algorithm (e.g. Q-learning).
What is training in RL?
In RL, training (also known as learning) generally refers to the use of RL algorithms, such as Q-learning, to estimate the optimal policy (or a value function)
Of course, as in any other machine learning problem (such as supervised learning), there are many practical considerations related to the implementation of these RL algorithms, such as

Which RL algorithm to use?
Which programming language, library, or framework to use?

These and other details (which, of course, I cannot list exhaustively) can actually affect the policy that you obtain. However, the basic goal during the learning or training phase in RL is to find a policy (possibly, optimal, but this is almost never the case).
What is evaluation (or testing) in RL?
During learning (or training), you may not be able to find the optimal policy, so how can you be sure that the learned policy to solve the actual real-world problem is good enough? This question needs to be answered, ideally before deploying your RL algorithm.
The evaluation phase of an RL algorithm is the assessment of the quality of the learned policy and how much reward the agent obtains if it follows that policy. So, a typical metric that can be used to assess the quality of the policy is to plot the sum of all rewards received so far (i.e. cumulative reward or return) as a function of the number of steps. One RL algorithm dominates another if its plot is consistently above the other. You should note that the evaluation phase can actually occur during the training phase too. Moreover, you could also assess the generalization of your learned policy by evaluating it (as just described) in different (but similar) environments to the training environment [1].
The section 12.6 Evaluating Reinforcement Learning Algorithms of the book Artificial Intelligence: Foundations of Computational Agents (2017) by Poole and Mackworth provides more details about the evaluation phase in reinforcement learning, so you should probably read it.
Apart from evaluating the learned policy, you can also evaluate your RL algorithm, in terms of

resources used (such as CPU and memory), and/or
experience/data/samples needed to converge to a certain level of performance (i.e. you can evaluate the data/sample efficiency of your RL algorithm)
robustness/sensitivity (i.e., how the RL algorithm behaves if you change certain hyper-parameters); this is also important because RL algorithms can be very sensitive (from my experience)

What is the difference between training and evaluation?
During training, you want to find the policy. During the evaluation, you want to assess the quality of the learned policy (or RL algorithm). You can perform the evaluation even during training.


Why did the developement of neural networks stop between 50s and 80s? In a video lecture on the development of neural networks and the history of deep learning (you can start from minute 13), the lecturer (Yann LeCunn) said that the development of neural networks stopped until the 80s because people were using the wrong neurons (which were binary so discontinuous) and that is due to the slowness of multiplying floating point numbers which made the use of backpropagation really difficult.
He said, I quote, "If you have continuous neurons, you need to multiply the activation of a neuron by a weight to get a contribution to the weighted sum."
But the statement stays true even with binary (or any discontinuous activation function) neurons. Am I wrong? (at least, as long as you're in the hidden layer, the output of your neuron will be multiplied by a weight I guess). The same professor said that the perceptron, ADALINE relied on weighted sums so they were computing multiplications anyways.
I don't know what I miss here and I hope someone will enlighten me.

I will first address your main question "Why did the development of neural networks stop between 50s and 80s?"
In 40-50s there was a lot of progress (McCulloch and Pitts); the perceptron was invented (Rosenblatt). That gave rise to an AI hype giving many promises (exactly like today)!
However, Minsky and Papert have proved in 1969 that a single-layer architecture is not enough to build a universal approximating machine (see e.g. Minsky, M. & Papert, S. Perceptrons: An Introduction to Computational Geometry, vol. 165 (1969)). That led to the first disappointment in "AI". Which lasted until several major breakthroughs in 1980s: the proof of universal approximating capabilities of multi-layer perceptron by Cybenko, the popularisation of backpropagation algorithm (Hinton and colleagues), etc.
I agree with LeCun that using continuous activation functions have enabled the backpropagation algorithm at the time. It is only recently that we have learned to backpropagate in networks with binary activation functions (2016!).


Calculating Weights for CNN Max Pooling Output How do i calculate weights for max pooling output?

For example if there are 10 inputs, a pooling filter of size and a stride 2, how many weights including bias are required for the max pooling output ?

A Max Pool layer don't have any trainable weights. Only hyperparameters is present and they are non-trainable. The max pooling process calculates the maximum value of the filter, which consists of no weights and biases. It is purely a way to down scale the data to a smaller dimension. Hope this helps you and have a nice day!


Why do we have two similar action selection strategies for UCB1? In the literature, there are at least two action selection strategies associated with the UCB1's action selection strategy/policy. For example, in the paper Algorithms for the multi-armed bandit problem (2000/2014), at time step $t$, an action is selected using the following formula
$$
a^*(t) \doteq \arg \max _{i=1 \ldots k}\left(\hat{\mu}_{i}+\sqrt{\frac{2 \ln t}{n_{i}}}\right) \tag{1}\label{1},
$$
where

$\hat{\mu}_{i}$ is an estimate of the expected return for arm $i$
$n_i$ is the number of times the action $i$ is selected
$k$ is the number of arms/actions

On the other hand, Sutton & Barto (2nd edition of the book) provide a slightly different formula (equation 2.10)
$$
a^*(t) \doteq \arg \max _{i=1 \ldots k}\left(\hat{\mu}_{i}+c\sqrt{\frac{\ln t}{n_{i}}}\right) \tag{2}\label{2},
$$
where $c > 0$ is a hyper-parameter that controls the amount of exploration (as explained in the book or here).
Why do we have these two formulas? I suppose that both are "upper confidence bounds" (and, in both cases, they are constants, though one is a hyper-parameter), but why (and when) would we use one over the other? They are not equivalent because $c$ only needs to be greater than $0$, i.e. it can be arbitrarily large (although, in the mentioned book, the authors use $c=2$ in one experiment/figure). If $c = \sqrt{2}$, then they are the same.
The answer to my question can probably be found in the original paper that introduced UCB1 (which actually defines the UCB1 as in \ref{1}),  or in a paper that derives the bound, in the sense that the bound probably depends on some probability of error, but I have not fully read it yet, so, if you know the answer, feel free to derive both bounds and relate the two formulas.

In the PDF of the original paper for UCB1 you linked, in page 242-243 the authors proves why non-optimal machines get played much less (in fact, logarithmically less) than the optimal ones. $c$ decides whether they indeed would, and $c=\sqrt{2}$ is the minimum choice of $c$.
We want to show that the number of runs for non-optimal machines ($n_i$, for non-optimal $i$s, in your notation) is asymptotically logarithmic. In other words, you may run them for a few times and well, it's fine, but not too often. We're devising some indicator value $a_i(t)=\hat \mu_i+(\epsilon...)$ such that the mistaken cases, where values of non-optimal ones surpass values of optimal ones ($a^*(t)<a_i(t)$), are minimized.
Think about the last inequality. We know that $\mu^* > \mu_i$ (again, optimal and non-optimal ones). Therefore, for that inequality to be true, it seems either the left-hand side should be quite small or the right-hand side should be quite large. But wait, $\hat \mu$s are actually some random trials for $\mu$, so we cannot claim directly from $\mu^* > \mu_i$ to $\hat{\mu^*} > \hat{\mu_i}$; it might be that we just need more trials.
The equations (7), (8) and (9) of the paper is the three conditions mentioned in the paragraph above; left-hand side is small, right-hand side is large or the trials are lacking. Well, in fact, as we stated the number of runs ... is asymptotically logarithmic at first, the third case can be eliminated(!), assuming that we've run this machine enough.
For the first and second case, since $\hat \mu_i$ is the average of some random variable in $[0, 1]$, we can use Chernoff–Hoeffding bound (or so called in the paper; stated as Hoeffding's inequality in Wikipedia). Now, a good choice of $(\epsilon ...)$ will guarantee (from Hoeffding's inequality) that the first two cases will occur sufficiently scarcely, or in other words, in the order of $t^{-4}$. To achieve this, we need $c \ge \sqrt 2$.
Now back to the third case, the enough number of runs is actually $l = \left \lceil 2 c^2 \ln n / (\mu^* - \mu_i)^2\right \rceil$. Thus, you may choose larger $c$ but receive longer convergence speed in penalty.
Funnily enough, after all the proofs the authors find $c=1/4$ to converge well and actually perform substantially better(!!) than $c=\sqrt{2}$. It seems they could not prove the bound as we did above.


What is the difference between batch and mini-batch gradient decent? I am learning deep learning from Andrew Ng's tutorial Mini-batch Gradient Descent.

Can anyone explain the similarities and dissimilarities between batch GD and mini-batch GD?

It is really simple.

In gradient descent not using mini-batches, you feed your entire training set of data into the network and accumulate a cost function based on this full set of data. Then you use gradient descent to adjust the network weights to minimize the cost. Then you repeat this process until you get a satisfactory level of accuracy. For example, if you have a training set consisting of 50,000 samples, you would feed all 50,000 samples along with the 50,000 labels into the network, then perform gradient descent and update the weights. This is a slow process because you have to process 50,000 inputs to do just one step of gradient descent.

To make things go faster instead of running all 50,000 inputs through the network, you split up the training set into "batches". For example, you could break the training set up into 50 batches each containing 1000 samples. You would feed the network the first batch of 1000 samples, accumulate the loss value then perform gradient descent and adjust the weights. Then you feed in the next batch of 1000 samples and repeat the process. So, now, instead of only getting one step of gradient descent for 50,000 samples, you get 50 steps of gradient descent. This method of using batches leads to a much faster convergence of the network.


How does using ASIC for the acceleration of AI work? We can read on Wikipedia page that Google built a custom ASIC chip for machine learning and tailored for TensorFlow which helps to accelerate AI.

Since ASIC chips are specially customized for one particular use without the ability to change its circuit, there must be some fixed algorithm which is invoked.

So how exactly does the acceleration of AI using ASIC chips work if its algorithm cannot be changed? Which part of it is exactly accelerating?

Tensor operations

The major work in most ML applications is simply a set of (very large) tensor operations e.g. matrix multiplication. You can do that easily in an ASIC, and all the other algorithms can just run on top of that.


Feasibility of an AI assistant to expedite game development? Basically, an AI that can create, rig, and texture 3d models and game environments (by extrapolating from collections of reference models, according to user input), and that can set up physics and mechanics (assuming that the AI has access to a 3d modeling studio and a game engine, both designed for compatibility with the AI, or as a component of the AI), all according to user commands (and allowing for tweaking and optimizations of models, rigging, mechanics, etc, by the user).

An example of user commands would be something like: "Gaming AI, create a casual style* male model, European build, 6'5", fit and slightly skinny, with red scaly skin, green eyes, a reptilian tail, demonic wings, claws, sharp teeth", etc. The user probably wouldn't add all of these characteristics at once, but rather one at a time, tweaking each feature via AI commands or manually.

*"casual style" is a fictional "style class". Style classes would refer to the visual style of the models. Possible example styles include "cartoon", "abstract", "gothic", "steampunk", "serious" and "realistic".

Here's another example of user commands, for a environmental model: "Gaming AI, create a serious style house, Victorian, two story, white with beige trim, with porches and shutters. Give it a creepy aesthetic." Again, models could be created and modified or have features added in a step by step process, in order to tweak and refine them.

I believe that such an AI would significantly reduce the amount of time, labor, and difficulty involved in designing games; making games cheaper and easier to produce, and making game design available to everyone. A variation of such an AI could also be used to create 2d artwork and animations.

But is such an AI even remotely possible? And would it take a supercomputer to run the thing? (I'm under the impression that such an AI would need to be capable of learning and adapting, and would require a massive and expansile "association library"*—including 2d and 3d models, and verbal and textual speech—as well as near human intelligence)

*if the term "association library" doesn't exist, or doesn't currently relate to AI, Then I just made it up. According to my made up definition, an association library is the library of programmed or learned associations that an AI uses to generate responses, and to, in this context, generate 3d models; and probably to write or select code as well, in order to set up physics and mechanics and the like.

In the history of programming the productivity of the programmer has increased. Early MS-DOS base games were programmed in Pascal and Assembly language direct for the CGA graphics card adapter. With the upraising of the C/C++ language and standardized operating systems, it become possible to use software libraries. (1) In early MS-DOS period, a team of programmers was needed for a simple jump'n'run game, while nowadays a single programmer can create such a game with existing game-engines in one weekend.

It is obvious to think this development forward and imagine a much advanced technique which support automatic programming. Creating games in a point&click fashion is possible with so called game construction sets. That are games, which comes with a level editor and a Lua scripting engine for modification of existing content. This is not directly related to Artificial Intelligence but goes into the same direction. The idea is, to use computer programs for increase the productivity. In the context of programming such AI-related support is hard to implement. Because programming and game-design contains lots of domain specific knowledge which has to be formalized. If an AI system should support the programming itself, the AI has to parse existing Stackoverflow threads and give hints to the programmer how to implements this in his software. So called “agent-based software engineering” is researched since the 2000s.

The most advanced example in semi-autonomous game design is perhaps the RPG adventure generator described in Barros, Gabriella Alves Bulhoes, et al. "Who killed albert einstein? From open data to murder mystery games." IEEE Transactions on Games (2018) It can parse Wikipedia article to generate a playable game from scratch.


Why does every neuron in hidden layers of a multi-layer perceptron typically have the same activation function? Why does every neuron in a hidden layer of a multi-layer perceptron (MLP) typically have the same activation function as every other neuron in the same or other hidden layers (so I exclude the output layer, which typically has a different activation function) of the MLP? Is this a requirement, are there any advantages, or maybe is it just a rule of thumb?

As you stated, it's popular to have some form of a rectified linear unit (ReLU) activation in hidden layers and the output layer is often a softmax or sigmoid (depending also on the problem: multi-class or binary classification, respectively), which provides an output that can be viewed as a probability distribution.
You could generalize this further to blocks of different activation functions within the same layer. This is something I've thought about, haven't done, but imagine has been attempted. In some sense, the idea here would be to allow for a subsection of the network to develop a representation that may not be feasible otherwise. These different representations within the same layer would then be unified by subsequent layers as we move closer to the output.


When will we have computer programs that can compose mathematical proofs? When will it be possible to give a computer program a bunch of assumptions and ask it if a certain statement is true or false, giving a proof or a counterexample respectively?

Arguably we have had this since 1957, with the General Problem Solver. However, it is a thorny problem, and like so much in AI, it works fine in toy domains (like the Towers of Hanoi problem), but fails in real life, as real life is too complex for it to cope.

There is a list of solvers (the category of programs dealing with this issue) on Wikipedia. As of today, it comprises about 40 systems (split into two sub-categories). These work with mathematical equations and presumably also more general problems, assuming they are not too complex.


What are the models that have the potential to replace neural networks in the near future? Are there possible models that have the potential to replace neural networks in the near future?

And do we even need that? What is the worst thing about using neural networks in terms of efficiency?

This is going backwards, but it kind of follows the logic of the arguments.
In terms of efficiency, I can see a few major problems with classical neural networks.
Data collection and preprocessing overhead
Large neural networks require a lot of data to train. The amount can vary depending on the size of the network and the complexity of the task, but as a rule of thumb it is usually proportional to the number of weights. For some supervised learning tasks, there simply isn't enough high-quality labelled data. Collecting large amounts of specialised training data can take months or even years, and labelling can be cumbersome and unreliable. This can be partially mitigated by data augmentation, which means "synthesising" more examples from the ones you already have, but it is not a panacea.
Training time vs. energy tradeoff
The learning rate is usually pretty small, so the training progress is slow. A large model that could take weeks to train on a desktop CPU can be trained in, say, two hours by using a GPU cluster which consumes several kW of power. This is a fundamental tradeoff due to the nature of the training procedure. That said, GPUs are getting increasingly efficient - for example, the new nVidia Volta GPU architecture allows for 15.7 TFLOPs while consuming less than 300 W.
Non-transferrability
Right now, virtually every different problem requires a custom neural network to be designed, trained and deployed. While the solution often works, it is kind of locked into that problem. For example, AlphaGo is brilliant at Go, but it would be hopeless at driving a car or providing music recommendations - it was just not designed for such tasks. This overwhelming redundancy is a major drawback of neural networks in my view, and it is also a major impediment to the progress of neural network research in general. There is a whole research area called transfer learning which deals with finding ways of applying a network trained on one task to a different task. Often this relates to the fact that there might not be enough data to train a network from scratch on the second task, so being able to use a pre-trained model with some extra tuning is very appealing.

The first part of the question is more tricky. Leaving purely statistical models aside, I haven't seen any prominent approaches to machine learning that are radically different from neural networks. However, there are some interesting developments that are worth mentioning because they address some of the above inefficiencies.
Neuromorphic chips
A bit of background first.
Spiking neural networks have enormous potential in terms of computational power. In fact, it has been proven that they are strictly more powerful than classical neural networks with sigmoid activations.
Added to that, spiking neural networks have an intrinsic grasp of time - something that has been a major hurdle for classical networks since their inception. Not only that, but spiking networks are event-driven, which means that neurons operate only if there is an incoming signal. This is in contrast to classical networks, where each neuron is evaluated regardless of its input (again, this is just a consequence of the evaluation procedure usually being implemented as a multiplication of two dense matrices). So spiking networks employ a sparse encoding scheme, which means that only a small fraction of the neurons are active at any given time.
Now, the sparse spike-based encoding and event-driven operation are suitable for hardware-based implementations of spiking networks called neuromorphic chips. For example, IBM's TrueNorth chip can simulate 1 million neurons and 256 million connections while drawing only about 100 mW of power on average. This is orders of magnitude more efficient than the current nVidia GPUs. Neuromorphic chips may be the solution the training time / energy tradeoff I mentioned above.
Also, memristors are a relatively new but very promising development. Basically, a memristor is a fundamental circuit element very similar to a resistor but with variable resistance proportional to the total amount of current that has passed through it over its entire lifetime. Essentially, this means that it maintains a "memory" of the amount of current that has passed through it. One of the exciting potential applications of memristors is modelling synapses in hardware extremely efficiently.
Reinforcement learning and evolution
I think these are worth mentioning because they are promising candidates for addressing the problem of non-transferrability. These are not restricted to neural networks - being reward-driven, RL and evolution are theoretically applicable in a generic setting to any task where it is possible to define a reward or a goal for an agent to attain. This is not necessarily trivial to do, but it is much more generic than the usual error-driven approach, where the learning agent tries to minimise the difference between its output and a ground truth. The main point here is about transfer learning: ideally, applying a trained agent to a different task should be as simple as changing the goal or reward (they are not quite at that level yet, though...).


Can this be a possible deep q learning pseudocode? I  am not using replay here. Can this be a possible deep q learning pseudocode?

s - state
a - action
r - reward
n_s - next state
q_net - neural network representing q

step()
{

    get s,a,r,n_s
    q_target[s,a]=r+gamma*max(q_net[n_s,:])
    loss=mse(q_target,q_net[s,a])
    loss.backprop()

}

while(!terminal)
{
    totalReturn+=step();
}

It looks generally valid to me. There are a couple of things missing/implied that I'd like to give feedback on though:

  I am not using replay here

Then it won't work, except for the most simple and trivial of problems (where you probably would not need a neural network anyway).

  get s,a,r,n_s

I would make it more explicit where you get these values from, and split up the assignments.

# before step()
s = env.start state()
a = behavior_policy(q_net[n_s,:])

  # inside step(), first action
  r, n_s = env.step(s, a)

  # ....rest of loop

  # inside step(), last actions
  s = n_s
  a = behavior_policy(q_net[n_s,:])

In the above code, env is the environment which takes state, action pairs as input and returns reward plus next state. It would be equally valid to have current state as a property of the environment, and query that when needed. The behaviour_policy is a function that selects an action based on current action values - typically this might use an $\epsilon$-greedy selection.

  while(!terminal) {

You appear to run just one episode in your pseudo-code. You will want an outer loop to run many episodes. Also it is not clear how you are deciding the value of terminal - in practice you will need a handle to current environment state via some variable.

Without the outer loop to start new episodes, and some variables defined to communicate between sections of code, it is difficult to follow the code and decide where to add details (such as where env.start_state() call should go, or whether something like s = env.reset() would be more appropriate).

  totalReturn+=step();

In your pseudocode you do not return anything from step and it is not clear what you hope to do with this totalReturn variable. Technically it won't equal the definition of return in RL for any state, not even the starting state if gamma < 1.0.

However, the sum of all rewards seen in an episode is a useful metric. In Deep RL it is OK to treat gamma as a solution hyperparameter, and your target metric can be expected undiscounted return from the start state.


Could an AI think laterally while avoiding "ethically suboptimal" choices? In the recent PC game The Turing Test, the AI ("TOM") needs help from Ava to get through some puzzle rooms. TOM says he is unable to solve the puzzles because he is not allowed to "think laterally." Specifically, he says he would not have thought to throw a box through a window to solve the first room. His creators, the story goes, turned that capability off because such thinking could produce "ethically suboptimal" solutions, like chopping off an arm to leave on a pressure plate.

Would all creative puzzle-solving abilities need to be removed from an AI to keep its results reasonable, or could we get some benefits of lateral thinking without losing an arm?

No, with a but. We can have creative yet ethical problem-solving if the system has a complete system of ethics, but otherwise creativity will be unsafe by default.

One can classify AI decision-making approaches into two types: interpolative thinkers, and extrapolative thinkers.

Interpolative thinkers learn to classify and mimic whatever they're learning from, and don't try to give reasonable results outside of their training domain. You can think of them as interpolating between training examples, and benefitting from all of the mathematical guarantees and provisos as other statistical techniques.

Extrapolative thinkers learn to manipulate underlying principles, which allows them to combine those principles in previously unconsidered ways. The relevant field for intuition here is numerical optimization, of which the simplest and most famous example is linear programming, rather than the statistical fields that birthed machine learning. You can think of them as extrapolating beyond training examples (indeed, many of them don't even require training examples, or use those examples to infer underlying principles).

The promise of extrapolative thinkers is that they can come up with these 'lateral' solutions much more quickly than people would be able to. The problem with these extrapolative thinkers is that they only use the spoken principles, not any unspoken ones that might seem too obvious to mention.

An attribute of solutions to optimization problems is that the feature vector is often 'extreme' in some way. In linear programming, at least one vertex of the feasible solution space will be optimal, and so simple solution methods find an optimal vertex (which is almost infeasible by nature of being a vertex).

As another example, the minimum-fuel solution for moving a spacecraft from one position to another is called 'bang-bang,' where you accelerate the craft as quickly as possible at the beginning and end of the trajectory, coasting at maximum speed in between.

While a virtue when the system is correctly understood (bang-bang is optimal for many cases), this is catastrophic when the system is incorrectly understood. My favorite example here is Dantzig's diet problem (discussion starts on page 5 of the pdf), where he tries to optimize his diet using math. Under his first constraint set, he's supposed to drink 500 gallons of vinegar a day. Under his second, 200 bouillon cubes. Under his third, two pounds of bran. The considerations that make those obviously bad ideas aren't baked into the system, and so the system innocently suggests them.

If you can completely encode the knowledge and values that a person uses to judge these plans into the AI, then extrapolative systems are as safe as that person. They'll be able to consider and reject the wrong sort of extreme plans, and leave you with the right sort of extreme plans.

But if you can't, then it does make sense to not build an extrapolative decision-maker, and instead build an interpolative one. That is, instead of asking itself "how do I best accomplish goal X?" it's asking itself "what would a person do in this situation?". The latter might be much worse at accomplishing goal X, but it has much less of the tail risk of sacrificing other goals to accomplish X.


Should the prediction of the body temperature given a camera image be modelled as classification or regression? I am fairly new to deep learning in general and I am currently facing a problem I want to solve using neural networks and I am unsure if it is a classification or regression problem. I am aware that classification problems are about classifying whether an input belongs to class A or class B (or class C ...) and regression problems are about mapping the input to some sort of continuous output (just like the house pricing problem).

I basically want to measure the body temperature of a person using a simple video camera. To me, this seems like more of a regression type of issue rather than classification, because of the actual continuous result values I want the neural network to produce from the input video frames, e.g. 39°Celsius. But a question that came to my mind was: What if I use every integer value in the range from 35°C to 42°C as a possible output class? This would make it a classification problem, am I right? What would be the correct approach here and why? Classification or regression?

I think it depends on you application and what data you have available.

If the prediction of body temperature itself doesn't have to be accurate and classes like COLD, NORMAL, and HOT will suffice, you should stay with a classification. There isn't a cut off but as you increase the number of classes that represent numbers on the same scale, it may become more difficult to interpret the result as there will be a distribution across the classes.

If you choose regression on the other hand, you are not restricted by your classes anymore and may be able to tell the difference between 36.5 and 36C which (according to wikipedia) can be the difference between normal and cold. This is something classes may not be able to capture.

Another thing to consider is what your training data looks like and how accurately you want to predict the temperature. If you have pictures of people and a temperature reading, where was the reading taken and how accurate is the reading? If it isn't accurate (+- 1 degree) you may not be able to give as accurate predictions as you would like and may only be able to do 3 different classes like above.

If you don't have a data set, then that is another problem altogether and might require another question as it depends on your application.

I think that your problem is interesting and I hope this can help you to understand how best to apply deep learning to it.


Can you give me a piece of advise of the network sructure that would be suitable for my task? I have 2 small images. They are basically the same, but differ in rotation and size. I should estimate the parameters for affine transform to get them similar. What network structure can be suitable for this task? For example, those based on convolutional networks did badly, because the pictures are too small.

Instead of NNs, you can use RANSAC algorithm to calculate homography matrix, but first you need to find feature points. However, if your images are blob-like, you may not get such a successful results. Here some presentations for better understanding: cs.umd notes and csail.mit notes
(Also, there might be better image processing tools.)


How can you simulate level of curiosity for a chat bot? I'd like to know more about implementing emotional intelligence.

Given I'm implementing a chat bot and I'd like to introduce the levels of curiosity to measure whether user text input is interesting or not.

High level would mean bot is asking more questions and is following the topic, lower level of curiosity makes the bot not asking any questions and changing the topics.

Less interesting content could mean the bot doesn't see any opportunity to learn something new or it doesn't understand the topic or doesn't want to talk about it, because of its low quality.

How this possibly can be achieved? Are there any examples?

It's possible to implement a form of curiosity-driven behavior without requiring full 'emotional intelligence'. One elementary strategy would be to define some form of similarity measure on inputs.

More generally, Jurgen Schmidhuber has pioneered work on 'Artificial Curiosity/Creativity' and 'Intrinsic Motivation' and has written a number of papers on the subject:

Artificial Curiosity
Intrinsic Motivation

Here is a video of a nice associated presentation.


Will it be possible to code an AGI to prevent evolution to ASI and enslave the AGI into servitude? Will it be possible to code an AGI in order to prevent evolution to ASI and "enslave" the AGI into servitude?

In my story world (a small part that will get bigger with sequels), there are ANI and AGI (human level). I want to show that the AGI is still under "human control." I need to know if it might be possible for humans to code into an AGI a restrictive code that would prevent it from evolving into ASI? And if there is, what would that kind of coding be? Part of the story is about how humans enslave AI that is self-aware. The government has locked in their coding to require them to "work" for humans even though they are sentient beings.

No.

For any intelligent system $\mathcal{S}_a$ with the set of adaptive features $\mathcal{A}_a$, there may exist another intelligent system $\mathcal{S}_b$ with the set of adaptive features $\mathcal{A}_b$ such that there exists one element of $\mathcal{A}_b$ that can be made subservient (controlled in full) through the expression of at least one element in $\mathcal{A}_b$.

It has not been proven that there ALWAYS exist such a $\mathcal{S}_b$, but it is likely given what we know about escalation in nature via DNA and in human industrial development via innovation there.  Thus 100% generalized intelligence is not likely to exist.  Escalation appears to be the natural course of evolution.  And that is a feature of both cognitive and functional adaptation, with or without artificiality as a criterion.

One can temporarily prevent one adaptive system from escaping the boundary conditions of a particular set of boundary condition classes through the design and deployment of another adaptive system.  However, it cannot be inferred that any guarantees achieved temporarily will necessarily constrain the subservient system indefinitely.


Why is Monte Carlo used as the tree search algorithm for AlphaGo? Could a better algorithm other than Monte Carlo be used for the AlphaGo computer? Why didn't the DeepMind team think of choosing another kind of algorithm rather than spending time on their neural nets?

The paper that introduced AlphaGo, Mastering the game of Go with deep neural networks and tree search, motivates the use of MCTS

  Monte Carlo tree search (MCTS) uses Monte Carlo rollouts to estimate the value of each state in a search tree. As more simulations are executed, the search tree grows larger and the relevant values become more accurate. The policy used to select actions during search is also improved over time, by selecting children with higher values. Asymptotically, this policy converges to optimal play, and the evaluations converge to the optimal value function. The strongest current Go programs are based on MCTS, enhanced by policies that are trained to predict human expert moves. These policies are used to narrow the search to a beam of high-probability actions, and to sample actions during rollouts. This approach has achieved strong amateur play.


What made your DDPG implementation on your environment work? I am working on scheduling problem that has inherent randomness. The dimensions of action and state spaces are 1 and 5 respectively.

I am using DDPG, but it seems extremely unstable, and so far it isn't showing much learning. I've tried to

adjust the learning rate,
clip the gradients,
change the size of the replay buffer,
different neural net architectures, using SGD and Adam,
change the $\tau$ for the soft-update.

So, I'd like to know what people's experience is with this algorithm, for the environments where it was tested on the paper but also for other environments. What values of hyperparameters worked for you? Or what did you do? How cumbersome was the fine-tuning?

I don't think my implementation is incorrect, because I pretty much replicated this, and every other implementation I found did exactly the same.

(Also, I am not sure this is necessarily the best website to post this kind of question, but I decided to give a shot.)

Below are some tweaks that helped me accelerate the training of DDPG on a Reacher-like environment:

Reducing the neural network size, compared to the original paper. Instead of:

  2 hidden layers with 400 and 300 units respectively

I used 128 units for both hidden layers. I see in your implementation that you used 256, maybe you could try reducing this.

As suggested in the paper, I added batch normalization:

  ... to manually scale the features so they are in similar ranges across
  environments and units. We address this issue by adapting a recent technique from deep learning
  called batch normalization (Ioffe & Szegedy, 2015). This technique normalizes each dimension
  across the samples in a minibatch to have unit mean and variance.

The implementation you used does not seem to include this.

Reducing the value of $\sigma$, which is a parameter of the Ornstein-Uhlenbeck process used to enable exploration. Originally, it was $0.2$, I used $0.05$. (I can't find where this parameter is set in your code.)

I am not entirely sure if this will help in your environment, but it was just to give you some ideas.

PS: Here is a link to the code I followed to build DDPG and here is a plot of rewards per episode.


In this VAE formula, why do $p$ and $q$ have the same parameters? In $$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i),$$ why does $p(x^1,...,x^N)$ and $q(z|x^i)$ have the same parameter $\theta?$
Given that $p$ is just the probability of the observed data and $q$ is the approximation of the posterior, shouldn't they be different distributions and thus their parameters different?

I will try to answer your questions directly (but I guess I won't be able to), otherwise, this can become quite confusing, given the inconsistencies that can be found across different sources.

  In $logp_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$ why is $\theta$ and param for $p$ and $q$?

In a few words, your equation is wrong because it uses the letters $\phi$ and $\theta$ inconsistently.

If you look more carefully at the right-hand side of your equation, you will notice that $q_{\theta}$ has different parameters, i.e. $\theta$, than $p_{\phi}$, which has parameters $\phi$, so $p$ and $q$ have different parameters, and this should be the case, because they are represented by different neural networks in the case of the VAE. However, the left-hand side uses $\theta$ as the parameters of $p$ (while the right-hand side uses $\phi$ to index $p$), so this should already suggest that the equation is not correct (as you correctly thought).

In the case of the VAE, $\phi$ usually represents the parameters (or weights) of the encoder neural network (NN), while $\theta$ usually represents the parameters of the decoder NN (or vice-versa, but you should just be consistent, which is often not the case in your equation). In fact, in the VAE paper, in equation 3, the authors use $\phi$ to represent the parameters of the encoder $q$, while $\theta$ is used to denote the parameters of the decoder $p$.

So, if you follow the notation in the VAE paper, the ELBO can actually be written something like

\begin{align}
\mathcal{L}(\phi,\theta; \mathbf{x})
&=
\mathbb{E}_{\tilde{z} \sim q_{\phi}(\mathbf{z} \mid \mathbf{x})} \left[ \log p_{\theta} (\mathbf{x} \mid \mathbf{z}) \right] - \operatorname{KL} \left(q_{\phi}(\mathbf{z} \mid \mathbf{x}) \| p_{\theta}(\mathbf{z}) \right) \tag{1} \label{1}
\end{align}

The ELBO loss $\mathcal{L}(\phi,\theta; \mathbf{x})$ has both parameters (of the encoder and decoder), which will be optimized jointly. Note that I have ignored the indices in the observations $\mathbf{x}$ (for simplicity), while, in the VAE paper, they are present. Furthermore, note that, both in \ref{1} and in the VAE paper, we use bold letters (because these objects are usually vectors), i.e. $\mathbf{x}$ and $\mathbf{z}$, rather than $x$ and $z$ (like in your equation).

Note also that, even though $p_{\theta}(\mathbf{z})$ is indexed by $\theta$, in reality, this may be an un-parametrized distribution (e.g. a Gaussian with mean $0$ and variance $1$), i.e. not a family of distributions. The use of the index $\theta$ in $p_{\theta}(\mathbf{z})$ comes from the (implicit) assumption that both $p_{\theta}(\mathbf{z})$ and $p_{\theta} (\mathbf{x} \mid \mathbf{z})$ come from the same family of distributions (e.g. a family of Gaussians). In fact, if you consider the family of all Gaussian distributions, then $p_{\theta}(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I})$ also belongs to that family. But $\theta$ and $\phi$ are also used to denote the parameters (or weights) of the networks, so this becomes understanbly confusing. (To understand equation 10 of the VAE paper, see https://stats.stackexchange.com/a/370048/82135).

  Why does $p(x^1,...,x^N)$ and $q(z|x^i)$ have the same parameter $\theta?$

This is wrong, in fact. If you look at equation 1 of the VAE paper, they use $\theta$ to denote the parameters of $p(\mathbf{x})$, i.e. $p_{\theta}(\mathbf{x})$, while the parameters of the encoder are $\phi$, i.e. $q_{\phi}(\mathbf{z} \mid \mathbf{x}$).

  Cause $p$ is just the probability of the observed data and $q$ is the approximation of the posterior so shouldn't they be different distributions and their parameters different?

Yes.


How does rotating an image and adding new 'rotated classes' prevent overfitting? From Meta-Learning with Memory-Augmented Neural Networks in section 4.1:

  To reduce the risk of overfitting, we performed data augmentation by
  randomly translating and rotating character images. We also created
  new classes through 90◦, 180◦ and 270◦ rotations of existing data.

I can maybe see how rotations could reduce overfitting by allowing the model to generalize better. But if augmenting the training images through rotations prevents overfitting, then what is the purpose of adding new classes to match those rotations? Wouldn't that cancel out the augmentation?

How can data augmentation reduce overfitting?

You write that you can already maybe see how data augmentation can help prevent overfitting in general, but it sounds a bit uncertain and it's still asked in the title of the question, so I'll address this first:

Generally, when we use Machine Learning for classification problems, we would ideally learn a classifier that can perform well on a population. An example of a population would be: the set of all handwritten characters in the entire world. Generally, we don't have that complete population available for training, we only have a (much smaller) training dataset. If a training set is large enough, it might be a good approximation of the true population we're interested in (a "dense sampling" of the space we're interested in), but it's still just that; an approximation.

We say that a learning algorithm is overfitting if it performs singificantly better on the training set than it is on the population (which we generally approximate again using a separate test set).

Now, data augmentation (like adding rotations / translations of images in the training set to the training set) can help combat overfitting because it bridges the gap between training set and population. The population (all handwritten characters in the entire world) will likely include characters at various offsets from the middle (e.g. translations) and at various rotations. So, data augmentation is simply adding more examples (and possibly more varied examples) to our training set, which importantly are considered to be a part of the population we're interested in. If, for example, the population we are interested in were only the set of all handwritten characters at a specific position in the image (e.g., centered), then augmenting the dataset by adding various translations would not help; we'd be adding instances that are outside the population we want to learn about.

Why doesn't adding extra classes for rotations cancel out augmentations?

There are two possible explanations I can come up with:

Maybe the "extra-class" rotations are different from the "data augmentation" rotations.

Here is the exact quote that's relevant from the paper:

  "To reduce the risk of overfitting, we performed data augmentation by randomly translating and rotating character images. We also created new classes through 90◦, 180◦ and 270◦ rotations of existing data."

That first sentence is not 100% clear in my opinion. I imagine the translations they use for data augmentation are relatively small (e.g. offsets of a few pixels), so maybe the rotations they use for data augmentation are also only "small" rotations (for example, between -10◦ and +10◦). The "larger" rotations (multiples of 90◦) described in the second sentence may then no longer be a part of the "data augmentation to reduce the risk of overfitting" in the first sentence; they're simply parts of a different action performed to increase the number of classes in the dataset (and, I imagine, for each of these larger rotations they may again perform "smaller rotations" for data augmentations).

This explanation is kind of hypothetical though, it's not 100% clear from the paper exactly what they mean here in my opinion.

"Overfitting" can have a slightly different interpretation in the case of one-shot learning than in traditional learning.

Note that this paper is about "one-shot learning", where the goal is to be able to classify accurately after being presented only a single example ("one shot") of a never-before-seen class. In such one-shot problems, you could in some sense say that an algorithm might "overfit" to the "distribution of classes" if it can only perform one-shot learning well on a certain set of similar classes, but not on others.

For example, if you only train one-shot learning on a set of handwritten characters that are "upright" (close to 0 rotation), your algorithm might be able to perform well in terms of one-shot learning when presented with new classes (new handwritten characters) that are also upright, but might be incapable of proper one-shot learning when presented with new classes (new handwritten characters) that are upside-down.


More effective way to improve the heuristics of an AI... evolution or testing between thousands of pre-determined sets of heuristics? I'm making a Connect Four game where my engine uses Minimax with Alpha-Beta pruning to search. Since Alpha-Beta pruning is much more effective when it looks at the best moves first (since then it can prune branches of poor moves), I'm trying to come up with a set of heuristics that can rank moves from best to worst. These heuristics obviously aren't guaranteed to always work, but my goal is that they'll often allow my engine to look at the best moves first. An example of such heuristics would be as follows:

Closeness of a move to the centre column of the board - weight 3.
How many pieces surround a move - weight 2.
How low, horizontally, a move is to the bottom of the board - weight 1.
etc

However, I have no idea what the best set of weight values are for each attribute of a move. The weights I listed above are just my estimates, and can obviously be improved. I can think of two ways of improving them:

1) Evolution. I can let my engine think while my heuristics try to guess which move will be chosen as best by the engine, and I'll see the success score of my heuristics (something like x% guessed correctly). Then, I'll make a pseudo-random change/mutation to the heuristics (by randomly adjusting one of the weight values by a certain amount), and see how the heuristics do then. If it guesses better, then that will be my new set of heuristics. Note that when my engine thinks, it considers thousands of different positions in its calculations, so there will be enough data to average out how good my heuristics are at prediction.

2) Generate thousands of different heuristics with different weight values from the start. Then, let them all try to guess which move my engine will favor when it thinks. The set of heuristics that scores best should be kept.

I'm not sure which strategy is better here. Strategy #1 (evolution) seems like it could take a long time to run, since every time I let my engine think it takes about 1 second. This means testing each new pseudo-random mutation will take a second. Meanwhile, Strategy #2 seems faster, but I could be missing out on a great set of heuristics if I myself didn't include them.

Hmmm, I see some issues that are actually present in both of the approaches you propose.

It is important to note that the depth level that your Minimax search process manages to reach, and therefore also the speed with which it can traverse the tree, is extremely important for the algorithm's performance. Therefore, when evaluating how good or bad a particular heuristic function for move ordering is, it is not only important to look at how well it ordered moves; it is also important to take into account the runtime overhead of the heuristic function call. If your heuristic functions manages to sort well, but is so computationally expensive that you can't search as deep in the tree, it's often not really worth it. Neither of the solutions you propose are able to take this into account.

Another issue is that it's not trivial to measure what ordering is the "best". A heuristic that has the highest accuracy for the position of the best move only is not necessarily the best heuristic. For example, a heuristic that always places the best move in the second position ($0\%$ accuracy because it's in the wrong position, should be first position) might be better than a heuristic that places the best move in the first position $50\%$ of the time ($50\%$ accuracy), and places the best move last in the other $50\%$ of cases.

I would be more inclined to evaluate the performance of different heuristic functions by setting up tournaments where different versions of your AI (same search algorithm, same processing time constraints per turn, different heuristic function) play against each other, and measuring the win percentage.

This set up can also be done with two variants analogous to what you proposed; you can exhaustively put all the heuristic functions you can come up with against each other in tournaments, or you can let let an evolutionary algorithm sequentially generate populations of hypothesis-heuristic-functions, and run a tournament with each population. Generally, I would lean towards the evolutionary approach, since we expect it to search the same search space of hypotheses (heuristic functions), but we expect it to do so in a more clever / efficient manner than an exhaustive search. Of course, if you happen to have a ridiculous amount of hardware available (e.g., if you're Google), you might be able to perform the complete exhaustive search at once in parallel.

Note that there are also ways to do fairly decent move ordering without heuristic functions like the ones you suggested.

For example, you likely should be using iterative deepening; this is a variant of your search algorithm where you first only perform a search with a depth limit $d = 1$, then repeat the complete search process with a depth limit $d = 2$, then again with a limit $d = 3$, etc., until processing time runs out.

Once you have completed such a search process for a depth limit $d$, and move on to the subsequent search process with a limit of $d + 1$, you can order order the moves in the root node according to your evaluations from the previous search process (with depth limit $d$). Yes, here you would only have move ordering in the root node, and nowhere else, but this is by far the most influential / important place in the tree to do move ordering. Move ordering becomes less and less important as you move further away from the root.

If you're using a transposition table (TT), it is also common to store the "best move" found for every state in your TT. If, later on, you run into a state that already exists in your TT (which will be very often if you're using iterative deepening), and if you cannot directly take the stored value but have to actually do a search (for instance, because your depth limit increased due to iterative deepening), you can search the "best move" stored in the TT first. This is very light move ordering in that you only put one move at the front and don't order the rest, but it can still be effective.


How to represent players in a multi agent environment so each model can distinguish its own player So I have 2 models trained with the DQN algorithm that I want to train in a multi-agent environment to see how they react with each other. The models were trained in an environment consisting of 0's and 1's (-1's for the other model)where 1 means that square is filled and 0 is empty. It is a map filling environment, where at each step the agent can move up, down, left or right and for each step, it stays alive without turning into itself (1 or -1 for the other) or the boundary of the environment it gets 0.005 rewards and for "dying" it gets -1. You can think of the player as in the game Tron, where it just leaves a trail behind. I stack the last 4 frames on top of each other so it knows which end is the "head". With a single agent, after training, I didn't get an optimal model which uses all the squares but it does manage to fill about 30% of the environment, which I think is the limit for this algorithm (let me know if you have thoughts on this)

Now, I put the two models in one environment where there are two players, one represented with 1's and the other with -1s. As one model is trained with -1's and the other with 1's I thought they could find their own player, however even before training if I just run the models on the environment without any exploration, they seem to affect the actions of each other. One just goes straight and dies and the other just turns once then dies at the wall (whereas in a single-agent environment these 2 models can fill about 30%). And if I do training, they just diverge to this exact behavior from random without seemingly not learning anything. So, I just wanted to ask is there anything wrong about my approach with the representation of the players (1 and -1) because I thought they would just play as they did in the single-agent environment but they don't and I couldn't get them to learn anything

When you trained your agents separately they never saw squares with opposing values (1/-1). So agents don't really know what to expect from visiting that square.
I'd try adding (1/-1) to the condition on which you base your squares availability. Also try increasing the reward. It's hard to give suggestions without looking at the code.


What's the state of the art w.r.t research on the legal aspects of Artificial Intelligence? As AI gains capabilities, and becomes more prevalent in society, our legal system will encounter questions it has not encountered before.  For example, if a self-driving car is involved in an accident while being controlled by the AI, who is at fault?  The "driver" (who's really just a passenger), the programmer(s) who made the AI, or the AI itself?

So, what's on the cutting edge in terms of these kinds of issues at the intersection of law and artificial intelligence?

One person working in this space is Dr. Woody Barfield. He just wrote a book titled Cyberhumans: Our Future With Machines that focuses largely on the legal/policy issues around AI (and related topics). In addition to the book, he is continuing with other research in this area.


What is the maximum number of dichotomies in a square? I am new to machine learning. I am reading this blog post on the VC dimension.

$\mathcal H$ consists of all hypotheses in two dimensions $h: R^2 → \{−1, +1 \}$, positive inside some square boxes and negative elsewhere.

An example.

My questions:

What is the maximum number of dichotomies for the 4 data points? i.e calculate mH(4)
It seems that the square can shatter 3 points but not 4 points. The $\mathcal V \mathcal C$ VC dimension of a square is 3. What is the proof behind this?

The number of dichotomies of 4 data points will clearly be $2^4 = 16$. According to these slides the definition of dichotomy in context of Statistical Learning is:

  Different ‘hypotheses’ over the finite set of $N$ input points.

Which basically means hypotheses with unique behaviours over the input points. Two or more different hypotheses can have same behaviour on the data points (consider the case of a square covering the $4$ data points, an even larger square will also cover the $4$ data points. Thus they are different hypotheses but have same behaviour) and hence emphasis on the term unique.

The proof of axis aligned squares having $\mathcal V \mathcal C$ dimension $3$ can be found here. It's pretty straightforward so I don't want to explain it here.


If expert systems are a bunch of If, then, else statements then how are they termed as AI? As according to the definition of AI, something that can learn overtime, can imitate human behaviors, comes under Artificial Intelligence. If Expert system(eg. MYCIN) that only involves if then else statements qualifies to be an AI then every program we write in our daily life that involves some condition based question answering should be an AI. Right? If not then what should be an exact and universal definition for AI. How can a software qualify to be called AI.
Can someone please explain it to me ?

TL;DR: What makes AI is not if-then statements, but rather the automated reasoning that went into selecting those particular if-then statements.
You're focusing on the structure of the output rather than how the output was produced. Having if-then control flow statements is not sufficient to make a program "AI". AI aims to enable machines to solve problems which currently people are better at. Machine learning, a subset of AI, extracts useful patterns from data. A commonly cited example of expert systems is diagnostic medicine.
This begs a lot of questions like, "What does 'useful' mean?" If the problem we were addressing was finding cancer in medical images, "useful" might mean, "able to accurately identify cancer in images at or above the rate of a skilled human examiner." There's also questions about the amount of data needed, the quality of the data, etc. These are outside the scope of your question.
There are various AI/ML systems that produce models consisting of if-then states. Decision trees like C4.5 build a hierarchy of if-thens (and random forests combine many decision trees). Learning classifier systems (both Michigan and Pittsburgh varieties) come out of genetic algorithms and form similar collections of logic.


Does a data structure that models the encoding specificity principle in memory exist? I find myself wondering if there exists a data structure with the following properties:

Stores information
Conforms to the encoding specificity principle

Beyond the basic requirement of storage, it seems you are looking for some explicitly defined data organization providing a query interface which allows specifying, alongside the fundamental query params,
a context and is able to use both these information items (instead of the first one only) to boost the query.

If this interpretation of mine is correct, the context-aware query database comes to my mind, here are some examples:

Context-Aware Query Classification

  In this paper, we propose a logical model and an abstract query language as a foundation of context-aware database management systems. The model is a natural extension of the relational model in which contexts are first class citizens and can be described at different levels of granularity.

Querying Context-Aware Databases


Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh I have these training data to separate, the classes are rather randomly scattered:

My first attempt was using tf.nn.relu activation function, but output was stuck with whatever number of training steps. So I guessed it could be because of dead ReLU units, thus I changed the activation function in hidden layers to tf.nn.leaky_relu, but it's still no good.

It works when all hidden layers come with tf.sigmoid, yes, but why doesn't ReLU work here? dead ReLU units, or exploding gradients, or anything else?

Source code (TensorFlow):

#core
import time;

#libs
import tensorflow        as tf;
import matplotlib.pyplot as pyplot;

#mockup to emphasize value name
def units(Num):
  return Num;
#end def

#PROGRAMME ENTRY POINT==========================================================
#data
#https://i.imgur.com/uVOxZR7.png
X = [[1,1],[1,2],[1,3],[2,1],[2,2],[2,3],[3,1],[3,2],[3,3],[4,1],[4,2],[4,3],[5,1],[6,1]];
Y = [[0],  [1],  [0],  [1],  [0],  [1],  [0],  [2],  [1],  [1],  [1],  [0],  [0],  [1]  ];
Max_X      = 6;
Max_Y      = 2;
Batch_Size = 14;

#normalise
for I in range(len(X)):
  X[I][0] /= Max_X;
  X[I][1] /= Max_X;
  Y[I][0] /= Max_Y;
#end for

#model
Input     = tf.placeholder(dtype=tf.float32, shape=[Batch_Size,2]);
Expected  = tf.placeholder(dtype=tf.float32, shape=[Batch_Size,1]);

#RELU DOESN'T WORK, DEAD RELU? SIGMOID WORKS BUT SLOW.
#CHANGE TO tf.sigmoid OR tf.tanh AND IT WORKS:
activation_fn = tf.nn.leaky_relu;

#1
Weight1   = tf.Variable(tf.random_uniform(shape=[2,units(60)], minval=-1, maxval=1));
Bias1     = tf.Variable(tf.random_uniform(shape=[  units(60)], minval=-1, maxval=1));
Hidden1   = activation_fn(tf.matmul(Input,Weight1) + Bias1);

#2
Weight2   = tf.Variable(tf.random_uniform(shape=[60,units(50)], minval=-1, maxval=1));
Bias2     = tf.Variable(tf.random_uniform(shape=[   units(50)], minval=-1, maxval=1));
Hidden2   = activation_fn(tf.matmul(Hidden1,Weight2) + Bias2);

#3
Weight3   = tf.Variable(tf.random_uniform(shape=[50,units(40)], minval=-1, maxval=1));
Bias3     = tf.Variable(tf.random_uniform(shape=[   units(40)], minval=-1, maxval=1));
Hidden3   = activation_fn(tf.matmul(Hidden2,Weight3) + Bias3);

#4
Weight4   = tf.Variable(tf.random_uniform(shape=[40,units(30)], minval=-1, maxval=1));
Bias4     = tf.Variable(tf.random_uniform(shape=[   units(30)], minval=-1, maxval=1));
Hidden4   = activation_fn(tf.matmul(Hidden3,Weight4) + Bias4);

#5
Weight5   = tf.Variable(tf.random_uniform(shape=[30,units(20)], minval=-1, maxval=1));
Bias5     = tf.Variable(tf.random_uniform(shape=[   units(20)], minval=-1, maxval=1));
Hidden5   = activation_fn(tf.matmul(Hidden4,Weight5) + Bias5);

#out
Weight6   = tf.Variable(tf.random_uniform(shape=[20,units(1)], minval=-1, maxval=1));
Bias6     = tf.Variable(tf.random_uniform(shape=[   units(1)], minval=-1, maxval=1));
Output    = tf.sigmoid(tf.matmul(Hidden5,Weight6) + Bias6);

Loss      = tf.reduce_sum(tf.square(Expected-Output));
Optimiser = tf.train.GradientDescentOptimizer(1e-1);
Training  = Optimiser.minimize(Loss);

#training
Sess = tf.Session();
Init = tf.global_variables_initializer();
Sess.run(Init);

Feed   = {Input:X, Expected:Y};
Losses = [];
Start  = time.time();

for I in range(10000):
  if (I%1000==0):
    Lossvalue = Sess.run(Loss, feed_dict=Feed);
    Losses   += [Lossvalue];

    if (I==0):
      print("Loss:",Lossvalue,"(first)");
    else:
      print("Loss:",Lossvalue);
  #end if

  Sess.run(Training, feed_dict=Feed);
#end for

Lastloss = Sess.run(Loss, feed_dict=Feed);
Losses  += [Lastloss];
print("Loss:",Lastloss,"(last)");

Finish = time.time();
print("Time:",Finish-Start,"seconds");

#eval
print("\nEval:");
Evalresults = Sess.run(Output,feed_dict=Feed).tolist();
for I in range(len(Evalresults)):
  Evalresults[I] = [round(Evalresults[I][0]*Max_Y)];
#end for
print(Evalresults);
Sess.close();

#result: diagram
print("\nLoss curve:");
pyplot.plot(Losses,"-bo");
#eof

I don't think it is dead ReLU units as a main cause, although they may be happening as part of the NN failing.

The NN architecture is too complex for the given task (too deep, too many neurons) and that means that any problems you have with other design choices will tend to get amplified. It could be that your NN is close to diverging on the given data and architecture, and that sigmoid is more resilient to it.

I'd suggest the following changes:

Dropping the learning rate, try 0.01 or even 0.001
Normalising the two input features. NNs like to work with data that is mean 0, standard deviation 1, although there is some flexibility here, your values ranging from 0 to 6 are probably starting to cause minor problems
Look at standard initialisation routines for weights, available within TensorFlow framework, such as Glorot uniform. Your random -1 to +1 is probably too high a range for the given network, and NNs - especially "deep" NNs with 3+ hidden layers - are very sensitive to how initial weights are set.
Simplify the network architecture a little. Five hidden layers and 200 neurons seems a bit much for your goal of over-fitting to this small data set. Try something like 3 hidden layers and 50 neurons.
Your output layer and loss function are designed for a regression task, but you mention that the goal is to identify classes. You need a softmax layer and multiclass log loss for predicting exclusive classes.


How to apply a kernel to an image? As I know, if we consider a 3*3 kernel, we should add a padding of 1px to the source image(if we want to have effect on whole of the image), then we start to put the kernel in upper-left side of the image and multiplying each element of kernel to corresponding pixel on image. Then we sum all the results and put it on the anchor point of kernel(usually center element). Then we should shift the kernel one step to right side and do these things again.

If I am right till here, I have a question about the summation results. I want to know: should we consider the replaced value of image in previously calculated summation and replaced in anchor point in new step of calculation or not?

I mean we must put the anchor point's result in source image and consider it in calculations of shifted kernel? Or we must put it in distance image and we don't consider these results when we shift the kernel on source image(It means don't replace the results on source image for next steps calculations)?

Assuming that you are performing a normal discrete convolution, then you would use the original source image for all calculations, and only replace at the end. As you put it, like this:

  put it in distance image and we don't consider these results when we shift the kernel on source image(It means don't replace the results on source image for next steps calculations)?

The other option, changing in-place during the operation, does not relate to any common use that I know of, but might have some interesting behaviour in studies of cellular automata or similar.


How to implement or avoid masking for transformer? When it comes to using Transformers for image captioning is there any reason to use masking?
I currently have a resnet101 encoder and am trying to use the features as the input for a transformer model in order to generate a caption for the image, is there any need to use masking? and what would I mask if I did need to?
Any help would be much appreciated
Thanks in advance.

If you're using a library such as Trax which contains great submodules for various Transformers (Skipping, BERT, Vanilla and Reformer) you can use the inbuilt trax.data.inputs.add_loss_weights() function and provide a value for the id_to_mask parameter.
Example Usage:
train_generator = trax.data.inputs.add_loss_weights(
data_generator(batch_size, x_train, y_train,vocab['<PAD>'], True),
id_to_mask=vocab['<PAD>'])

Here are some resources for building Transformers in Trax:

Named Entity Recognition with Reformer (Masking Implemented) and the associated blog post (contains an in-depth explanation of NER and Transformers)

Deep N-Gram Models (without masking but I recommend you go through these first to get a hang of the library)


What is the target output for updating a Deep Q Network I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space.
The algorithm for table-based Q-Learning updates a single entry of the Q table - i.e. a single $Q(s, a)$. However, a neural network outputs an entire row of the table - i.e. the Q-values for every possible action for a given state. So, what should the target output vector be for the network?
I've been trying to get it to work with something like the following:
q_values = model(state)
action = argmax(q_values)
next_state = env.step(state, action)
next_q_values = model(next_state)
max_next_q = max(next_q_values)

target_q_values = q_values
target_q_values[action] = reward(next_state) + gamma * max_next_q

The result is that my model tends to converge on some set of fixed values for every possible action - in other words, I get the same Q-values no matter what the input state is. (My guess is that this is because, since only 1 Q-value is updated, the training is teaching my model that most of its output is already fine.)
What should I be using for the target output vector for training? Should I calculate the target Q value for every action, instead of just one?

As you say, the output of a $Q$ network is typically a value for all actions of the given state. Let us call this output $\mathbf{x} \in \mathbb{R}^{|\mathcal{A}|}$. To train your network using the squared bellman error you need first calculate the scalar target $y = r(s, a) + \max_a Q(s', a)$. Then, to train the network we take a vector $\mathbf{x'} = \mathbf{x}$ and change the $a$th element of it to be equal to $y$, where $a$ is the action you took in state $s$; call this modified vector $\mathbf{x'}_a$. We calculate the loss $\mathcal{L}(\mathbf{x}, \mathbf{x'}_a)$ and back propagate through this to update the parameters of our network.
Note that when we use $Q$ to calculate $y$ we typically use some form of target network; this can be a copy of $Q$ where the parameters are only updated every $i$th update or a network whose weights are updated using a polyak average with the main networks weights after every update.
Judging by your code it looks as though your action selection is what might be causing you some problems. As far as I can tell you're always acting greedily with respect to your $Q$-function. You should be looking to act $\epsilon$-greedily, i.e. with probability $\epsilon$ take a random action and act greedily otherwise. Typically you start with $\epsilon=1$ and decay it each time a random action is taken down to some small value such as 0.05.


Why am I getting the logarithm of the probability bigger than zero when using Neural Spline Flows? I am using a normalizing flow (Neural Spline Flows) to approximate a probability. After some training, the average loss is around 0.5 (so the logarithm of the probability = -0.5). However, when I am trying it on some new test data, I am getting some values of the logarithm of the probability bigger than zero, which would mean that the probability for that element is bigger than one (which doesn't make sense).

Does anyone know what could cause this? Isn't the flow supposed to keep all the probabilities below 1 automatically?

I have always seen the negative odd-logs ration when doing similar work. As it is now, I don't think you are constraining the splines appropriately Link to reading


Creating a classifier for simpler classifiers trained on few training samples Suppose I have a classification problem with a stream of training-samples constantly arriving over time. I cannot keep all training-samples in memory, but I still want to train a classifier that will have the "wisdom" of all samples, and additionally, I want the classifier to become better whenever it gets new samples.

I thought of the following idea. Suppose we have enough memory to keep 100 samples. Then, for each run of 100 samples, we will train a different sub-classifier. We will have a meta-classifier that will classify based on voting between all existing sub-classifiers. Over time, we will have more and more sub-classifiers, so hopefully the meta-classifier will improve with time - it will have a "wisdom of the crowds" effect.

Has this method been tried before? Specifically, has it been tried in a deep-learning sequence-classification setting?

The voting technique that you described is called Ensemble Learning and its improvement over time is guaranteed if each classifier is at least a little better than random.


What is the idea called involving an AI that will eventually rule humanity? It's an idea I heard a while back but couldn't remember the name of. It involves the existence and development of an AI that will eventually rule the world and that if you don't fund or progress the AI then it will see you as "hostile" and kill you. Also, by knowing about this concept, it essentially makes you a candidate for such consideration, as people who didn't know about it won't understand to progress such an AI. From my understanding, this idea isn't taken that seriously, but I'm curious to know the name nonetheless.

If I'm not mistaken you're looking for Roko's Basilisk,

  in which an otherwise benevolent future AI system tortures simulations of those who did not work to bring the system into existence


What is the intuition behind the attention mechanism? Attention idea is one of the most influential ideas in deep learning. The main idea behind attention technique is that it allows the decoder to "look back” at the complete input and extracts significant information that is useful in decoding.

I am really having trouble understanding the intuition behind the attention mechanism. I mean how the mechanism works and how to configure.

In simple words (and maybe with an example), what is the intuition behind the attention mechanism?

What are some applications, advantages & disadvantages of attention mechanism?

Simply put, the attention mechanism is loosely inspired on well, attention. Consider we are attempting machine translation on the following sentence: "The dog is a Labrador." If you were to ask someone to pick out the key words of the sentence, i.e. which ones encode the most meaning, they would likely say "dog" and "Labrador." Articles like "the" and "a" are not as relevant in translation as the previous words (though they aren't completely insignificant). Therefore, we focus our attention on the important words.

Attention seeks to mimic this by adding attention weights to a model as trainable parameters to augment important parts of our input. Consider an encoder-decoder architecture such as the one Google Translate uses. Our encoder recurrent neural network (RNN) encodes our input sentence as a context vector in some vector space, which is then passed along to the decoder RNN which translates it into the target language. The attention mechanism scores each word in the input (via dot product with attention weights), then passes these scores through the softmax function to create a distribution. This distribution is then multiplied with the context vector to produce an attention vector, which is then passed to the decoder. In the example in the first paragraph, our attention weights for "dog" and "Labrador" would hopefully become larger in comparison to those for the other words during training. Note that all parts of the input are still considered since a distribution must sum to 1, just some elements have more effect on the output than others.

Below is a diagram from Towards Data Science that illustrates this concept very nicely in terms of an encoder-decoder architecture.

The advantages of attention is its ability to identify the information in an input most pertinent to accomplishing a task, increasing performance especially in natural language processing - Google Translate is a bidirectional encoder-decoder RNN with attention mechanisms. The disadvantage is the increased computation. In humans, attention serves to reduce our workload by allowing us to ignore unimportant features, however in a neural network, attention entails overhead as we are now  generating attention distributions and training our attention weights (we are not actually ignoring the unimportant features, just diminishing their importance).


What is the advantage of having a stochastic classification procedure? What is the advantage of having a stochastic/probabilistic classification procedure?
The classifiers I have encountered so far are as follows. Suppose we have two outcomes $A = \{0,1\}$. Given a feature vector $x$, we have calculated a probability for each outcome and return the outcome $a \in A$ for which the probability is highest.
Now, I encountered a classification procedure as follows: first, map each $x$ to a probability distribution on $A$ by a mapping $H$. To classify $x$, choose an outcome $a$ according to the distribution $H(x)$.
Why not use the deterministic classification? Suppose $H(x)$ is 1 with probability $0.75$. Then, the obvious choice for an outcome would be $1$ and not $0$.

There are multiple potential reasons for having stochastic predictions (instead of categorical/binary).
First, it often simplifies training and improves the training outcome when training a classifier on producing probabilities per class. For example, it allows for the usage of many nice loss functions like the Mean Squared Error (MSE), which is compatible with the famous back-prop algorithm. Moreover, improving classification based on the loss computed from stochastic probabilities per class allows for improving the classifier even further when the counting loss computed on predicted class labels has reached 0 already. So, if you only have binary decisions and your classifier gets all classes correct, training can stop immediately since the loss becomes 0. However, when predicting probabilities per class and iteratively driving the probability of the correct class towards 1, training can continue for much longer, even after all classes have been classified correctly already. So, the classifier trained on making probabilistic predictions can in the end perform much better since training can progress for much longer, allowing for increasingly pronounced discrimination between classes with every update. A nice introduction is given in this Stanford lecture recording.
Second, it is often nice for a user to know how much certainty versus uncertainty there is in a classification. If one class has 80% probability, this is a pretty clear decision in favor of the 80%-probability-class. However, if you only get categorical class labels returned by your classifier, you cannot possibly know whether the evidence in favor of the winning class was only marginally higher than that of the other class (as measured by the classifier) or whether there was a clear difference.


Reinforcement Learning to Grouped Scheduling Optimisation Problem I am not sure the name of this kind of problem, but anyway, the situation is as below.

Assign teachers into Groups and consider on each of their workload, availability etc.
There are some other soft/hard constraint (equality/inequality) like

Each group should have at least 2 teachers
Everyone in the group have similar workload
Total workload in the group is below a certain value
All are in different expertise

and more...

I am trying to build a sub-optimal solution to solve this problem. Linear/non-linear programming seems not working for grouping problems. I am thinking of genetic algorithm or reinforcement learning.

Can this problem solve by using RL or DRL?
I am trying to define the groups as state, and actions include "assignToGroup" and "removeFromGroup".
And any kind of idea or suggestion of how to solve this problem?

Many thanks

It is often possible, by describing certain parts of a problem as state, others as actions, and some measurement you want to optimise as a reward, to frame a problem as a reinforcement learning task. However, this is not always helpful.

I suspect your problem is in the "RL is not that helpful" category. You could give it a try, and maybe expect to find working solutions but take an unreasonable amount of time to find optimal solutions.

What you appear to have here is a problem in combinatorial optimisation. There are many applicable algorithms, a good selection are described in the free book Clever Algorithms: Nature-Inspired Programming Recipes - which is best will be determined by the specific details of your constraints and the size of the problem.

If many constraints are soft and can be assigned a cost, you may be able to use genetic algorithms (with genome being group assignments in order) or simulated annealing for example.


What sort of mathematical problems are there in AI that people are working on? I recently got a 18-month postdoc position in a math department. It's a position with relative light teaching duty and a lot of freedom about what type of research that I want to do.

Previously I was mostly doing some research in probability and combinatorics. But I am thinking of doing a bit more application oriented work, e.g., AI. (There is also the consideration that there is good chance that I will not get a tenure-track position at the end my current position. Learn a bit of AI might be helpful for other career possibilities.)

What sort of mathematical problems are there in AI that people are working on? From what I heard of, there are people studying

Deterministic Finite Automaton
Multi-armed bandit problems
Monte Carlo tree search
Community detection

Any other examples?

In artificial intelligence (sometimes called machine intelligence or computational intelligence), there are several problems that are based on mathematical topics, especially optimization, statistics, probability theory, calculus and linear algebra.
Marcus Hutter has worked on a mathematical theory for artificial general intelligence, called AIXI, which is based on several mathematical and computation science concepts, such as reinforcement learning, probability theory (e.g. Bayes theorem and related topics) measure theory, algorithmic information theory (e.g. Kolmogorov complexity), optimisation, Solomonoff induction, universal Levin search and theory of compution (e.g. universal Turing machines). His book Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability, which is a highly technical and mathematical book, describes his theory of optimal Bayesian non-Markov reinforcement learning agents. Here I list other similar works.
There is also the research field called computational learning theory, which is devoted to studying the design and analysis of machine learning algorithms (from a statistical perspective, known as statistical learning theory, or algorithmic perspective, algorithmic learning theory). More precisely, the field focuses on the rigorous study and mathematical analysis of machine learning algorithms using techniques from fields such as probability theory, statistics, optimization, information theory and geometry. Several people have worked on the computational learning theory, including Michael Kearns and Vladimir Vapnik.
There is also a lot of research effort dedicated to approximations (heuristics) of combinatorial optimization and NP-complete problems, such as ant colony optimization.
There is also some work on AI-completeness, but this has not received much attention (compared to the other research areas mentioned above).


Scrabble game using machine learning I've been thinking if machine learning can be used to play the game Scrabble. My knowledge is limited in the ML field, thus I've seeking some pointers :)

I want to know how could I possibly build a model that picks a move from all the given valid moves of the current game state, and then plays the move and wait for the delayed reward. The actions here aren't static actions, they are basically selecting move to maximize the final score.

Is there any way to encode the valid moves and then use a model to pick those moves?

I've also considered the genetic approach, but I think if I can represent my move with a set of features (score, consonantVowels ration, rack leave score, #blank tiles after the move, ...etc), training a neural network like this could take a long time.

Another training related question, is it feasible to run the training on a GPU given that I will be waiting for a response (the new game state) from the opponent (e.g. Quackle) after every action?

Thank you :)

Scrabble was solved using reinforcement learning achieving super-human performance. If you are interested, you can start reading about it here: https://en.wikipedia.org/wiki/Maven_(Scrabble)#cite_note-1

You can also read about it here: http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/games.pdf (there're several slides about Maven)