What is a simplified way to explain why the AI researchers Bengio, Hinton, and Lecun, won the 2019 Turing Award? The Turing award is sometimes called Computer Sceince's Nobel Prize. This year's award goes to Bengio, Hinton, and LeCun for their work on artificial neural networks.

The actual work contributed by these authors is, of course, quite technical. It centers around the development of deep neural networks, convolutional neural networks, and effective training techniques. The lay press will tend to simplify these results to the point that they lose meaning.

I would like to have a concise, and yet still precise, explanation of their contributions to share with a lay audience. So, what is a simplified way to explain the contributions of these researchers?

I have my own ideas and will add them if no other satisfactory answer appears. For a "lay" audience, I want to assume someone who had taken a college level course in something scientific but not necessarily computer science. Explanations that are suitable for those with even less background are better still though, as long as they don't lose too much precision.

The related ACM article describes a few specific technical contributions, which led the ACM to award them.

Geoffrey Hinton
Backpropagation: In a 1986 paper, "Learning Internal Representations by Error Propagation", co-authored with David Rumelhart and Ronald Williams, Hinton demonstrated that the backpropagation algorithm allowed neural nets to discover their own internal representations of data, making it possible to use neural nets to solve problems that had previously been thought to be beyond their reach. The backpropagation algorithm is standard in most neural networks today.
Boltzmann Machines: In 1983, with Terrence Sejnowski, Hinton invented Boltzmann Machines, one of the first neural networks capable of learning internal representations in neurons that were not part of the input or output.
Improvements to convolutional neural networks: In 2012, with his students, Alex Krizhevsky and Ilya Sutskever, Hinton improved convolutional neural networks using rectified linear neurons and dropout regularization. In the prominent ImageNet competition, Hinton and his students almost halved the error rate for object recognition and reshaped the computer vision field.
Yoshua Bengio
Probabilistic models of sequences: In the 1990s, Bengio combined neural networks with probabilistic models of sequences, such as hidden Markov models. These ideas were incorporated into a system used by AT&T/NCR for reading handwritten checks, were considered a pinnacle of neural network research in the 1990s, and modern deep learning speech recognition systems are extending these concepts.
High-dimensional word embeddings and attention: In 2000, Bengio authored the landmark paper, "A Neural Probabilistic Language Model", that introduced high-dimension word embeddings as a representation of word meaning. Bengio's insights had a huge and lasting impact on natural language processing tasks including language translation, question answering, and visual question answering. His group also introduced a form of attention mechanism which led to breakthroughs in machine translation and form a key component of sequential processing with deep learning.
Generative adversarial networks: Since 2010, Bengio's papers on generative deep learning, in particular the Generative Adversarial Networks (GANs) developed with Ian Goodfellow, have spawned a revolution in computer vision and computer graphics. In one fascinating application of this work, computers can actually create original images, reminiscent of the creativity that is considered a hallmark of human intelligence.
Yann LeCun
Convolutional neural networks: In the 1980s, LeCun developed convolutional neural networks, a foundational principle in the field, which, among other advantages, have been essential in making deep learning more efficient. In the late 1980s, while working at the University of Toronto and Bell Labs, LeCun was the first to train a convolutional neural network system on images of handwritten digits. Today, convolutional neural networks are an industry standard in computer vision, as well as in speech recognition, speech synthesis, image synthesis, and natural language processing. They are used in a wide variety of applications, including autonomous driving, medical image analysis, voice-activated assistants, and information filtering.
Improving backpropagation algorithms: LeCun proposed an early version of the backpropagation algorithm (backprop), and gave a clean derivation of it based on variational principles. His work to speed up backpropagation algorithms included describing two simple methods to accelerate learning time.
Broadening the vision of neural networks: LeCun is also credited with developing a broader vision for neural networks as a computational model for a wide range of tasks, introducing in early work a number of concepts now fundamental in AI. For example, in the context of recognizing images, he studied how hierarchical feature representation can be learned in neural networks - a concept that is now routinely used in many recognition tasks. Together with Léon Bottou, he proposed the idea, used in every modern deep learning software, that learning systems can be built as complex networks of modules where backpropagation is performed through automatic differentiation. They also proposed deep learning architectures that can manipulate structured data, such as graphs.


When we create artificial life and artificial intelligence will we require it to obey human laws? Assuming mankind will eventually create artificial humans, but in doing so have we put equal effort into how humans will relate to an artificial human, and what can we expect in return? This is happening in real-time as we place AI trucks and cars on the road. Do people have the right to question, maybe in court, if an AI machine breaks a law?

For those times when AI does interact with humans, I believe that AI would be held at LEAST to the same standards humans are. The problem comes in when we ask "who is really to blame". If a self-driving car cuts you off in traffic and causes you to wreck, you can't take the AI in the car to court. Do you take the company? The programmer? The owner of the car? Some entity will likely be held responsible, the question is just which one.

As for future human-like AI, I believe my answer still remains true. Having a human level AI changes the meaning of the word "entity". If a human-like AI breaks a law, it may be because it was programmed to do so. I don't think our current legal system is ready for such cases, but it have to evolve in the future.


In YOLO, what exactly do the values associated with each anchor box represent? I'm going through Andrew NG's course, which talks about YOLO, but he doesn't go into the implementation details of anchor boxes.
After having looked through the code, each anchor box is represented by two values, but what exactly are these values representing?
As for the need for anchor boxes, I'm also a little confused about that --
As far as I understand, the ground truth labels have around 6 variables :

$P_o$ checks if it's an object or background,
$B_x$ and $B_y$ are the center coordinates
$B_h$ and $B_w$ are the height and width of the box
$C$ is the object class, which depends on how many classes you have, so you can have multiple $C$

As for creating the bounding box,
$B_h$ is divided by 2, with one half from the center points ($B_x, B_y$) to the top, and the other half to the bottom.
If we train our classifier, wouldn't the prediction boxes be close to the ground truth labels as training progresses? So, if our ground truth label has a high height, small width as boxes for some images, and low hight and large width for other images, wouldn't our classifier automatically
learn to differentiate between when to use one over the other, as it is being trained? If so then what is the use of anchor boxes? And what are those numbers representing anchor boxes representing?

Not an pro but I think I know some answers to your questions.

  If we train our classifier, wouldn't the prediction boxes be close to the ground truth labels as training progresses

I think that's what YOLO v1 did. According to Andrew NG's video the bounding boxes are introduced to solve multiple objects inside the same grid cell. And according to this post anchor boxes assignment ensures that an anchor box predicts ground truth for an object centered at its own grid center, and not a grid cell far away (like YOLO may)

  what are those numbers representing anchor boxes representing?

They are just width and height (shape). In YOLO v2 it is used to compute IOU assuming all boxes are placed at the same location (ignoring the location), you could think of it just tries to match the shape. And it uses (1-IOU) as the distance when applying the K-means clustering.


Two chatbots - One teaches another I am seeking the information for this kind of chatbot architecture : There are two chatbots. One plays the role of teacher, and another is a student who is learning. The goal is to test the student's quality, and to improve the student's ability.

I didn't find much reference. There are :

Bottester: Testing Conversational Systems with Simulated Users

And the ParlAI, a python-based platform for enabling dialog AI research has the notion of "Teacher agent", which seems to be what I am looking for.

Of course, we also have deep reinforcement learning which might be related.

I prefer to have some classical references for this approach to chatbots.
Currently, reinforcement learning is not in my consideration.

Constructing two chatbots talking to each other, like what Facebook did, is not what I want. Because in this case, both of them are student agents.

The Hidden Agenda User Simulation Model (Schatzmann/Young) describes a chatbot training design in which a user simulator assembles and executes a conversational agenda, with the direct goal being to train the target chatbot.

Perhaps you can add specificity to this design by casting the user simulator as the teacher, and creating an agenda in which it is communicating information to the (student) chatbot. The trained behavior expected is, perhaps, correct responses to topical questioning by the teacher.


What is the effect of mislabeled training data? Collecting and labeling training data for supervised learning tasks is incredibly time-consuming and costly.

For instance, let's say you wrote a script that went on Google images and got you 5000 pictures for each of 10 classes. You then use an unsupervised algorithm to cluster them. Then, you train a supervised algorithm using the labels from the scraper as ground truth. Obviously, your network will perform more poorly than one with perfectly labeled data, but is there a way to guesstimate how much?

Perhaps there are 50 mislabeled images in each class. That would most likely be better than 500 mislabeled images, but I'm wondering if there is a way to predict how much (even if it is by someone's rules of thumb or something like that).

I think the crucial point here is what you precisely mean by mislabelled. Google's image classifier will likely do a 'pretty good' job of retrieving images with the given subject included, but how strict or lenient your class requisites are is quite important. For example, if one of your classes is 'dog' there may be hundreds of images procured from scraping that could display (examples off the top of my head, but you can get even more creative):

Ancient canine fossils
Wolves
Stuffed animal dogs
Hard mode dogs (i.e. partially occluded, variable lighting schemas, background color variation, intraclass/species variation)

Additionally, your computational tool will impact this. If you're using a neural network, the above issues can to some extent be accounted for, but a linear classifier would likely have difficulty adopting a broader/flexible view of your class.


Why are tree-based models more widely used in Medical Diagnosis? In Chapter 14.4 (p. 664) of the book Pattern Recognition and Machine Learning by Bishop, it is mentioned that tree-based models are more widely used in Medical Diagnosis.
Apart from giving better performance, is there a human-centric reason for this trade-off as medical diagnosis is mainly performed by a human?

One possible reason may have something to do with the scrutability of models, as described in the first few paragraphs of this article.  It presents a case study of a hospital whose policy was to send asthma sufferers to an intensive care unit; the intensive care meant they were less likely to develop pneumonia and therefore the data showed that people with asthma were less likely to have pneumonia.

Essentially, since machine learning models learn false relationships if the data are in any way flawed, it is beneficial to be able to "debug" them.  The processes by which decision trees make their decisions, and the reasons for making them, are more readily visible than in other models - particularly neural networks - which makes errors such as the example given in the article more likely to be picked up and corrected.


Recognition of abbreviated text I want to write a program that looks at abbreviated words, then figures out what the words are. For example, the abbreviation is "blk comp", and the translation is "black computer".

In order to give it context for more ambiguous terms, I will be inputting sets of words with each request. So, if I input the set "keyboard, software, mouse, monitor", I would expect to get "black computer". On the other hand, if I input "Honda, transmission, mileage, Ford", I then would expect to get "black compact", or at least something that has anything to do with cars.

Basing on the above case scenario, what kind of an algorithm should be applied in this case?

For your first question,take a look at using a skip grab model to find what the abbreviated text is. The skip gram model turns a word into a vector which allows it to be processed by other machine learning algorithms. Or , alternatively you can do some really cool addition and subtraction problems with the resulting vectors. With the skip gram model in your case you could generate a vector for your input and then compare it with other skip gram vectors and once you have found a near perfect match then that is the unabbreviated word.

You could also look at using sparse distributed representations of words to do this. This approach is similar to the skip gram model except that instead of a vector with maybe 500 values a sparse representation may contain thousands of binary digits of which only a couple are positive or 1.

If you would like to look at this approach take a look at cortical.io which has free API that you can use. As for your second question I reccomend that you use a deep neural network in combination with the skip gram model to produce your output.


Can you provide some pseudocode examples of what constitutes an AI? After years of learning, I still can't understand what is considered to be an AI. What are the requirements for an algorithm to constitute Artificial Intelligence? Can you provide pseudocode examples of what constitutes an AI?

AI is not a simple term. There are different types, ranging from the most simplistic rule-based AI to black-box AI's so complicated it's unreasonable for a human to understand exactly what they're doing.

There's no pseudocode that if used in a program automatically constitutes it as an AI. It's not that black and white. But I can give examples:

Here's a rule-based chess AI that forfeits if it's too far behind, and plays aggressively if it's far enough ahead.

if player.score - my.score > 10:
    forfeit
elif my.score - player.score > 10:
    agressive = True

for each piece of my.pieces:
    for each square of board.squares:
        if noThreats(square) and agressive is True:
            move(piece, square)
            return

This is considered an "AI" because it feigns intelligence - appearing to have a true understanding of chess while simply following a set of rules, making it an "Artificial" Intelligence.

Here's another more complicated AI:

decisionNet = NeuralNetwork(64 inputs, 2 outputs)

choice = decisionNet(board.squares) // Returns a chess square with one of my pieces and desitnation
move(choice)

This uses a neural network to make the decision, which could have been trained on a bunch of example games or against itself. Due to this "training phase", humans can't understand precisely what the network is doing without extensive effort, so it's an even gives an even more convincing understanding of chess. But if we want, we could still understand the nuances of this network, and show it doesn't possess an intelligence, it again only feigns it.

I should mention that virtually any code that has an if statement can be considered AI. The examples I provided are just easier to pass off as understanding a very complicated concept (chess), as opposed to, say, verifying a user login. They both have the same fundamentals, it's just one appears more complicated on the surface than the other.


Why is GLIE Monte-Carlo control an on-policy control? In slide 16 of his lecture 5 of the course "Reinforcement Learning", David Silver introduced GLIE Monte-Carlo Control.

But why is it an on-policy control? The sampling follows a policy $\pi$ while improvement follows an $\epsilon$-greedy policy, so isn't it an off-policy control?

In this case, $\pi$ has always been an $\epsilon$-greedy policy. In every iteration, this $\pi$ is used to generate ($\epsilon$-greedily) a trajectory from which the new $Q(s, a)$ values are calculated. The last line in the "pseudocode" tells you that the policy $\pi$ will be a new $\epsilon$-greedy policy in the next iteration. Since the policy that is improved and the policy that is sampled are the same, the learning method is considered an on-policy method.
If the last line was $\mu \leftarrow \epsilon\text{-greedy}(Q)$, it would be an off-policy method.


Issues with and alternatives to Deep Learning approaches? Over the last 50 years, the rise/fall/rise in popularity of neural nets has acted as something of a 'barometer' for AI research.

It's clear from the questions on this site that people are interested in applying Deep Learning (DL) to a wide variety of difficult problems.

I therefore have two questions:

Practitioners - What do you find to be the main obstacles to
applying DL 'out of the box' to your problem?
Researchers - What
techniques do you use (or have developed) that might help address
practical issues? Are they within DL or do they offer an
alternative approach?

I have very little experience with ML/DL to call myself either practitioner, but here is my answer on the 1st question:

At its core DL solves well the task of classification. Not every practical problem can be rephrased in terms of classification. Classification domain needs to be known upfront. Although the classification can be applied to any type of data, it's necessary to train the NN with samples of the specific domain where it'll be applied. If the domain is switched at some point, while keeping the same model (NN structure), it'll have to be retrained with new samples. Furthermore, even the best classifiers have "gaps" - Adversarial Examples can be easily constructed from a training sample, such that changes are imperceptible to human, but are misclassified by the trained model.


What exactly is averaged when doing batch gradient descent? I have a question about how the averaging works when doing mini-batch gradient descent.

I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-batch gradient descent, do I have to:

forward propagate
calculate error
calculate all gradients

...repeatedly over all samples in the batch, and then average all gradients and apply the weight change?

I thought it would work that way, but recently I have read somewhere that you basically only average the error of each example in the batch, and then calculate the gradients at the end of each batch. That left me wondering though, because, the activations of which sample in the mini-batch am I supposed to use to calculate the gradients at the end of every batch?

It would be nice if somebody could explain what exactly happens during mini-batch gradient descent, and what actually gets calculated and averaged.

Introduction

First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't completely answer the original question. In fact, I leave some unanswered questions at the end (that I will eventually answer).

The gradient is a linear operator

The gradient operator $\nabla$ is a linear operator, because, for some $f : \mathbb{R} \rightarrow \mathbb{R} $ and $g: \mathbb{R} \rightarrow \mathbb{R}$, the following two conditions hold.

$\nabla(f + g)(x) = (\nabla f)(x) + (\nabla g)(x),\; \forall x \in \mathbb{R}$
$\nabla(kf)(x) = k(\nabla f)(x),\; \forall k, x \in \mathbb{R}$

In other words, the restriction, in this case, is that the functions are evaluated at the same point $x$ in the domain. This is a very important restriction to understand the answer to your question below!

The linearity of the gradient directly follows from the linearity of the derivative. See a simple proof here.

Example

For example, let $f(x) = x^2$, $g(x) = x^3$ and $h(x) = f(x) + g(x) = x^2 + x^3$, then $\frac{dh}{dx} = \frac{d (x^2 + x^3)}{d x} = \frac{d x^2}{d x} + \frac{d x^3}{d x} = \frac{d f}{d x} + \frac{d g}{d x} = 2x + 3x$.

Note that both $f$ and $g$ are not linear functions (i.e. straight-lines), so the linearity of the gradients is not just applicable in the case of straight-lines.

Straight-lines are not necessarily linear maps

Before proceeding, I want to note that there are at least two notions of linearity.

There's the notion of a linear map (or linear operator), i.e. which is the definition above (i.e. the gradient operator is a linear operator because it satisfies the two conditions, i.e. it preserves addition and scalar multiplication).
There's the notion of a straight-line function: $f(x) = c*x + k$. A function can be a straight-line and not be a linear map. For example, $f(x) = x+1$ is a straight-line but it doesn't satisfy the conditions above. More precisely, in general, $f(x+y) \neq f(x) + f(y)$, and you can easily verify that this is the case if $x = 2$ and $y=3$ (i.e. $f(2+3) = 6$, $f(2) = 3$, $f(3) = 4$, but $f(2) + f(3) = 7 \neq f(2+3)$.

Neural networks

A neural network is a composition of (typically) non-linear functions (let's ignore the case of linear functions), which can thus be represented as $$y'_{\theta}= f^{L}_{\theta_L} \circ f^{L-1}_{\theta_{L-1}} \circ \dots \circ f_{\theta_1},$$ where

$f^{l}_{\theta_l}$ is the $i$th layer of your neural network and it
computes a non-linear function
${\theta_l}$ is a vector of parameters associated with the $l$th layer
$L$ is the number of layers,
$y'_{\theta}$ is your neural network,
$\theta$ is a vector containing all parameters of the neural network
$y'_{\theta}(x)$ is the output of your neural network
$\circ $ means the composition of functions

Given that $f^l_{\theta}$ are non-linear, $y'_{\theta}$ is also a non-linear function of the input $x$. This notion of linearity is the second one above (i.e. $y'_{\theta}$ is not a straight-line). In fact, neural networks are typically composed of sigmoids, ReLUs, and hyperbolic tangents, which are not straight-lines.

Sum of squared errors

Now, for simplicity, let's consider the sum of squared error (SSE) as the loss function of your neural network, which is defined as

$$
\mathcal{L}_{\theta}(\mathbf{x}, \mathbf{y}) = \sum_{i=1}^N  \mathcal{S}_{\theta}(\mathbf{x}_i, \mathbf{y}_i) = \sum_{i=1}^N (\mathbf{y}_i - y'_{\theta}(\mathbf{x}_i))^2
$$
where

$\mathbf{x} \in \mathbb{R}$ and $\mathbf{y} \in \mathbb{R}$ are vectors of inputs and labels, respectively
$\mathbf{y}_i$ is the label for the $i$th input $\mathbf{x}_i$
$\mathcal{S}_{\theta}(\mathbf{x}_i, \mathbf{y}_i) = (\mathbf{y}_i - y'_{\theta}(\mathbf{x}_i))^2$

Sum of gradients vs gradient of a sum

Given the gradient is a linear operator, one could think that computing the sum of the gradients is equal to the gradient of the sums.

However, in our case, we are summing $\mathcal{S}_{\theta}(\mathbf{x}_i, \mathbf{y}_i)$ and, in general, $\mathbf{x}_i \neq \mathbf{x}_j$, for $i \neq j$. So, essentially, the SSE is the sum of the same function, i.e. $S_{\theta}$, evaluated at different points of the domain. However, the definition of a linear map applies when the functions are evaluated at the same point in the domain, as I said above.

So, in general, in the case of neural networks with SSE, the gradient of the sum may not be equal to the sum of gradients, i.e. the definition of the linear operator for the gradient doesn't apply here because we are evaluating every squared error at different points of their domains.

Stochastic gradient descent

The idea of stochastic gradient descent is to approximate the true gradient (i.e. the gradient that would be computed with all training examples) with a noisy gradient (which is an approximation of the true gradient).

How does the noisy gradient approximate the true gradient?

In the case of mini-batch ($M \leq N$, where $M$ is the size of the mini-batch and $N$ is the total number of training examples), this is actually a sum of the gradients, one for each example in the mini-batch.

The papers Bayesian Learning via Stochastic Gradient Langevin Dynamics (equation 1) or Auto-Encoding Variational Bayes (in section 2.2) use this type of approximation. See also these slides.

Why?

To give you some intuition of why we sum the gradients of the error of each input point $\mathbf{x}_i$, let's consider the case $M=1$, which is often referred to as the (actual) stochastic gradient descent algorithm.

Let's assume we uniformly sample an arbitrary tuple $(\mathbf{x}_j, \mathbf{y}_j)$ from the dataset $\mathcal{D} = \{ (\mathbf{x}_i, \mathbf{y}_i) \}_{i=1}^N$.

Formally, we want to show that

\begin{align}
\nabla_{\theta} \mathcal{L}_{\theta}(\mathbf{x}, \mathbf{y})
&=
\mathbb{E}_{(\mathbf{x}_j, \mathbf{y}_j) \sim \mathbb{U}}\left[ \nabla_{\theta} \mathcal{S}_{\theta} \right] \label{1} \tag{1}
\end{align}

where

$\nabla_{\theta} \mathcal{S}_{\theta}$ is the gradient of $\mathcal{S}_{\theta}$ with respect to the parameters $\theta$
$\mathbb{E}_{(\mathbf{x}_j, \mathbf{y}_j) \sim \mathbb{U}}$ is the expectation with respect to the random variable associated with a sample $(\mathbf{x}_j, \mathbf{y}_j)$ from the uniform distribution $\mathbb{U}$

Under some conditions (see this), we can exchange the expectation and gradient operators, so \ref{1} becomes
\begin{align}
\nabla_{\theta} \mathcal{L}_{\theta}(\mathbf{x}, \mathbf{y})
&=
\nabla_{\theta} \mathbb{E}_{(\mathbf{x}_j, \mathbf{y}_j) \sim \mathbb{U}}\left[  \mathcal{S}_{\theta} \right]
\label{2} \tag{2}
\end{align}
Given that we uniformly sample, the probability of sampling an arbitrary $(\mathbf{x}_j, \mathbf{y}_j)$ is $\frac{1}{N}$. So, equation \ref{2} becomes
\begin{align}
\nabla_{\theta} \mathcal{L}_{\theta} (\mathbf{x}, \mathbf{y})
&=
\nabla_{\theta} \sum_{i=1}^N \frac{1}{N} \mathcal{S}_{\theta}(\mathbf{x}_i, \mathbf{y}_i) \\
&=
\nabla_{\theta} \frac{1}{N}  \sum_{i=1}^N \mathcal{S}_{\theta}(\mathbf{x}_i, \mathbf{y}_i)
\end{align}

Note that $\frac{1}{N}$ is a constant with respect to the summation variable $i$ and so it can be taken out of the summation.

This shows that the gradient with respect to $\theta$ of the loss function $\mathcal{L}_{\theta}$ that includes all training examples is equivalent, in expectation, to the gradient of $\mathcal{S}_{\theta}$ (the loss function of one training example).

Questions

How can we extend the previous proof to the case $1 < M \leq N$?
Which conditions need exactly to be satisfied so that we can exchange the gradient and the expectation operators? And are they satisfied in the case of typical loss functions, or sometimes they aren't (but in which cases)?
What is the relationship between the proof above and the linearity of the gradient?

In the proof above, we are dealing with expectations and probabilities!

What would the gradient of a sum of errors represent? Can we still use it in place of the sum of gradients?


What is the relation between online (or offline) learning and on-policy (or off-policy) algorithms? In the context of RL, there is the notion of on-policy and off-policy algorithms. I understand the difference between on-policy and off-policy algorithms. Moreover, in RL, there's also the notion of online and offline learning.
What is the relation (including the differences) between online learning and on-policy algorithms? Similarly, what is the relation between offline learning and off-policy algorithms?
Finally, is there any relation between online (or offline) learning and off-policy (or on-policy) algorithms? For example, can an on-policy algorithm perform offline learning? If yes, can you explain why?

The concepts of on-policy vs off-policy and online vs offline are separate, but do interact to make certain combinations more feasible. When looking at this, it is worth also considering the difference between prediction and control in Reinforcement Learning (RL).

Online vs Offline

These concepts are not specific to RL, many learning systems can be categorised as online or offline (or somewhere in-between).

Online learning algorithms work with data as it is made available. Strictly online algorithms improve incrementally from each piece of new data as it arrives, then discard that data and do not use it again. It is not a requirement, but it is commonly desirable for an online algorithm to forget older examples over time, so that it can adapt to non-stationary populations. Stochastic gradient descent with back-propagation - as used in neural networks - is an example.
Offline learning algorithms work with data in bulk, from a dataset. Strictly offline learning algorithms need to be re-run from scratch in order to learn from changed data. Support vector machines and random forests are strictly offline algorithms (although researchers have constructed online variants of them).

Of the two types, online algorithms are more general in that you can easily construct an offline algorithm from a strictly online one plus a stored dataset, but the opposite is not true for a strictly offline algorithm. However, this does not necessarily make them superior - often compromises are made in terms of sample efficiency, CPU cost or accuracy when using an online algorithm. Approaches such as mini-batches in neural network training can be viewed as attempts to find a middle ground between online and offline algorithms.

Experience replay, a common RL technique, used in Deep Q Networks amongst others, is another in-between approach. Although you could store all the experience necessary to fully train an agent in theory, typically you store a rolling history and sample from it. It's possible to argue semantics about this, but I view the approach as being a kind of "buffered online", as it requires low-level components that can work online (e.g. neural networks for DQN).

On-policy vs Off-Policy

These are more specific to control systems and RL. Despite the similarities in name between these concepts and online/offline, they refer to a different part of the problem.

On-policy algorithms work with a single policy, often symbolised as $\pi$, and require any observations (state, action, reward, next state) to have been generated using that policy.
Off-policy algorithms work with two policies (sometimes effectively more, though never more than two per step). These are a policy being learned, called the target policy (usually shown as $\pi$), and the policy being followed that generates the observations, called the behaviour policy (called various things in the literature - $\mu$, $\beta$, Sutton and Barto call it $b$ in the latest edition).

A very common scenario for off-policy learning is to learn about best guess at optimal policy from an exploring policy, but that is not the definition of off-policy.
The primary difference between observations generated by $b$ and the target policy $\pi$ is which actions are selected on each time step. There is also a secondary difference which can be important: The population distribution of both states and actions in the observations can be different between $b$ and $\pi$ - this can have an impact for function approximation, as cost functions (for e.g. NNs) are usually optimised over a population of data.

In both cases, there is no requirement for the observations to be processed strictly online or offline.

In contrast to the relationship between online and offline learning, off-policy is always a strict generalisation of on-policy. You can make any off-policy algorithm into an equivalent on-policy one by setting $\pi = b$. There is a sense in which you can do this by degrees, by making $b$ closer to $\pi$ (for instance, reducing $\epsilon$ in an $\epsilon$-greedy behaviour policy for $b$ where $\pi$ is the fully greedy policy). This can be desirable, as off-policy agents do still need to observe states and actions that occur under the target policy - if that happens rarely because of differences between $b$ and $\pi$, then learning about the target policy will happen slowly.

Prediction vs Control

This can get forgotten due to the focus on search for optimal policies.

The prediction problem in RL is to estimate the value of a particular state or state/action pair, given an environment and a policy.
The (optimal) control problem in RL is to find the best policy given an environment.

Solving the control problem when using value-based methods involves both estimating the value of being in a certain state (i.e. solving the prediction problem), and adjusting the policy to make higher value choices based on those estimates. This is called generalised policy iteration.

The main thing to note here is that the prediction problem is stationary (all long-term expected distributions are the same over time), whilst the control problem adds non-stationary target for the prediction component (the policy changes, so does the expected return, distribution of states etc)

Combinations That Work

Note this is not about choice of algorithms. The strongest driver for algorithm choice is on-policy (e.g. SARSA) vs off-policy (e.g. Q-learning). The same core learning algorithms can often be used online or offline, for prediction or for control.

Online, on-policy prediction. A learning agent is set the task of evaluating certain states (or state/action pairs), and learns from observation data as it arrives. It should always act the same way (it may be observing some other control system with a fixed, and maybe unknown policy).
Online, off-policy prediction. A learning agent is set the task of evaluating certain states (or state/action pairs) from the perspective of an arbitrary fixed target policy $\pi$ (which must be defined to the agent), and learns from observation data as it arrives. The observations can be from any behaviour policy $b$ - depending on the algorithm being used, it may be necessary to have $b$ defined as well as $\pi$.
Offline, on-policy prediction. A learning agent is set the task of evaluating certain states (or state/action pairs), and is given a dataset of observations from the environment of an agent acting using some fixed policy.
Offline, off-policy prediction. A learning agent is set the task of evaluating certain states (or state/action pairs) from the perspective of an arbitrary fixed target policy $\pi$ (which must be defined to the agent), and is given a dataset of observations from the environment of an agent acting using some other policy $b$.
Online, on-policy control. A learning agent is set the task of behaving optimally in an environment, and learns from each observations as it arrives. It will adapt its own policy as it learns, making this a non-stationary problem, and also importantly making its own history of observations off-policy data.
Online, off-policy control. A learning agent is set the task of behaving optimally in an environment. It may behave and gain observations from a behaviour policy $b$, but learns a separate optimal target policy $\pi$. It is common to link $b$ and $\pi$ - e.g. for $\pi$ to be deterministic greedy policy with respect to estimated action values, and for $b$ to be $\epsilon$-greedy policy with respect to the same action values.
Offline, on-policy control. This not really possible in general, as an on-policy agent needs to be able to observe data about its current policy. As soon as it has learned a policy different to that in the stored dataset, then all the data becomes off-policy to it, and the agent has no valid source data. In some cases you might still be able to get something to work.
Offline, off-policy control. A learning agent is set the task of learning an optimal policy from a store dataset of observations. The observations can be from any behaviour policy $b$ - depending on the algorithm being used, it may be necessary to have $b$ defined as well as $\pi$.

As you can see above, only one combination offline, on-policy control, causes a clash.

There is a strong skew towards online learning in RL. Approaches that buffer some data, such as Monte Carlo control, experience replay or Dyna-Q do mix-in some of the traits of offline learning, but still require a constant supply of new observations plus forget older ones. Control algorithms imply non-stationary data, and these require online forgetting behaviour from the estimators - another online learning trait.

However, mixing in a little "offline" data in experience replay can cause some complications for on-policy control algorithms. The experience buffer can contain things that are technically off-policy to the latest iteration of the agent. How much of a problem this is in practice will vary.


Can the C4.5 algorithm learn a GOAP model? Goal-oriented action planning (GOAP) is a well-known planning technique in computer games. It was introduced to control the non-player characters in the game F.E.A.R. (2005) by creating an abstract state model. Similar to STRIPS planning, a GOAP model contains an action name, a precondition and an effect. The domain knowledge is stored in these subactions.

The bottleneck of GOAP is, that before the planner can bring the system into the goal state. the action model has to be typed in. Usually, the programmer defines actions like "walk to", "open the door", "take the object", and identifies for each of them the feature set for the precondition and the effect.

In theory, this challenging task can be simplified with a decision tree learning algorithm. A decision tree stores the observed features in a tree and creates the rules on its own with inductive learning. A typical example of the C4.5 algorithm is to find a rule like "if the weather is sunny, then play tennis". Unfortunately, the vanilla tree learning algorithm doesn't separate between different actions.

Is it possible to modify C4.5 algorithm such that the GOAP actions, like "walk to", "open the door", etc., are connected to individual rules?

You're going to run straight into the qualification problem, the frame problem and ramification problem on these. Plus you'll need to get your head round what an action is and means.

Qualification Problem

A more formal "STRIPS" would be Situation Calculus (logical equivalency demonstrated by Reiter), which can be defined in FOL or SOL, depending on who you talk to. In Situation Calculus you have your actions and you define when they are possible (S is the current Situation):

poss(open_door, S) → holds(door_position(closed) ^ door_lock(unlocked), S).

This is true, when it is possible to open the door it's both closed and unlocked. The difficulty arises when we try to flip that implication around:

poss(open_door, S) ← holds(door_position(closed) ^ door_lock(unlocked), S).

This is not true, because there must also be nothing blocking the door, the door can't be wedged too tightly into the door frame, and a whole bunch of other stuff that we might not think of. This is the qualification problem, which we get around in STRIPS by ignoring it and just flipping the implication.

In your case you'll end up learning a whole bunch of irrelevant qualifications:

 poss(open_door, S) ← holds(door_position(closed) ^ door_lock(unlocked) ^ weather(sunny) ^ grass(green), S).

You'd need some way to trim them down, by observing gameplay for your data you'll only learn the conditions when players choose to do an action, not when it is possible to do that action: the old correlation vs causation problem. You can immediately prune things that aren't fluents, but as far as I'm aware there's no good solution to this without letting the computer simulate to try actions in subsets of the fluents that hold in-order to prune.

Frame Problem

The result of an action in the language of Situation Calculus is that it causes some fluent to hold. So in our example we have a door_position fluent, we can say that opening the door causes door_position(open) to hold, but what about some other action, such as making tea? We know it has no effect on the position of the door, but for the computer to know that we need to tell it:

holds(door_position(open), S) → holds(door_position(open), do(make_tea, S)).

This creates far too many axioms. Reiter has a working solution to this for Golog in his book "Knowledge in Action". In your case it will manifest as missing axioms, i.e. not knowing what the result of an action will be for some fluent. Try beginning with the assumption that an action has no effect unless proven otherwise.

Ramification Problem

The ramification problem is very similar to the frame problem, it means you might not attribute all of the effects of an action to that action. Opening a door can cause changes in air-pressure through a house that can cause another door to slam, such an effect is easy to miss. For this you'd need a sensitive algorithm that can correctly recognise rare events that are important, which is a big ask. The solution is likely to be to ignore it.

Closely related and not usually a problem for such systems, you could also attribute effects to an action incorrectly. How will you be able to distinguish what the action caused vs occurred simultaneously? In some Mario game data it could likely be induced that jumping causes a turtle to move closer to Mario, again correlation vs causation and needing a method to distinguish. The solution here is likely to come from increasing the volume of data.

Definition and Granularity Problem

OK, it's probably possible that with enough data and a sensitive training algorithm you can overcome the above problems. The definition one is merely a conceptual one and easily overcome.

Humans tend to define an action by it's name, say for example "turning on a light". However, in Situation Calculus the action is defined by when it is possible and the result of that action, the name is merely a label. So "turning on a light" where the bulb is broken and when it is not may share the same label, but they are different actions.

Furthermore, in STRIPS and Situation Calculus we're defining "types" or "classes" of actions, the actual execution is an "instance" of that "class". Your problem is to derive the abstract from the concrete.

For you this problem will arise in terms of defining the granularity, as each instance of an action is possible in the unique situation that it occurred in, and caused the unique situation that came after it.

How do you group these actions together to abstract the most similar ones? How do you know how many actions there are to create these groups? If it's not based on a limited set of user-inputs to some computer game, then you might want to draw some ideas from the research into clustering, particularly heuristics of how many clusters to use.

Alternative to a decision tree

If I were to tackle this problem (fund a Post-Doc for me and I'd be tempted), I'd first investigate ILP as a method to learn the relations using Golog as the foundation of my action axioms. Then I'd look at clustering based on a space where each fluent was a dimension. I imagine it would be a rather useful tool, even if it merely created a pretty accurate template for manual review.


Is there any difference between a control and an action in reinforcement learning? There are reinforcement learning papers (e.g. Metacontrol for Adaptive Imagination-Based Optimization) that use (apparently, interchangeably) the term control or action to refer to the effect of the agent on the environment at each time step.

Is there any difference between the terms control or action or are they (always) used interchangeably? If there is a difference, when is one term used as opposed to the other?

The term control likely comes from the field of optimal control theory, which is related to reinforcement learning.

There's no difference. As they too often do, ML researchers take concepts from other disciplines, conveniently forget to cite sources and change the terminology, leading to much confusion. RL is a textbook example (pun intended). Optimal control researchers have been studying very similar problems long before RL ones, and used standard symbols and terms ($x$ for states, $u$ for controls). Then RL researchers came and changed just about everything.

See the paper A Tour of Reinforcement Learning: The View from Continuous Control (2018), by Benjamin Recht, which discusses reinforcement learning from a control and optimization perspective.

See also this tweet https://twitter.com/beenwrekt/status/1134536093980864514?s=21 (by Benjamin Recht) regarding the presentation of Sham Kakade.


What is the difference between 'prediction' and 'control' problem in the context of Reinforcement Learning? What is the difference between the term 'prediction'/value estimation in RL as compared to the 'control' problem?

Are there scenarios in RL where the problem cannot be distinctly categorised into the aforementioned problems and is a mixture of the problems?

Examples where the problem cannot be easily categorised into one of the aforementioned problems would be nice.

Prediction is the problem of predicting any feature of the environment. In reinforcement learning, the typical feature is the reward or return, but this doesn't have to be always the case. See Multi-timescale nexting in a reinforcement learning robot (2011) by Joseph Modayil et al.
Control is the problem of estimating a policy. Clearly, the term control is related to control theory. In fact, the term control is often used as a synonym for action. See Is there any difference between a control and an action in reinforcement learning?. Similarly, the term controller is also used as a synonym for agent. For example, in the paper Metacontrol for Adaptive Imagination-Based Optimization (2017) by Jessica B. Hamrick et al. the term meta-controller is used to refer to an agent. A controlled system can also refer to the environment.
Section 14.1 of the book Reinforcement learning: an introduction (2nd edition) provides more details about the distinction between prediction and control and how this distinction is related to psychological concepts.


How should the values of the filters of a CNN change? I wrote a convolutional neural network for the MNIST dataset with Numpy from scratch. I am currently trying to understand every part and calculation.
But one thing I noticed was the "just positive" derivative of the ReLU function.

My network structure is the following:

(Input 28x28)
Conv Layer (filter count = 6, filter size = 3x3,  stride = 1)
Max Pool Layer (Size 2x2) with RELU
Conv Layer (filter count = 6, filter size = 3x3,  stride = 1)
Max Pool Layer (Size 2x2) with RELU
Dense (128)
Dense (10)

I noticed, when looking at the gradients, that the ReLU derivative is always (as it should be) positive. But is it right that the filter weights are always decreasing their weights? Or is there any way they can increase their weight?

Whenever I look at any of the filter's values, they decreased after training. Is that correct?

By the way, I am using stochastic gradient descent with a fixed learning rate for training.

The weights of the filters do not always and necessarily decrease. Consider the extreme case when you initialise them to $-\infty$ and you want to approximate a function different than the one the CNN represents initially with all weights set to $-\infty$. You will have to increase one or more weights.


How can I start learning mathematics for machine learning? I am an Android programmer. Now, I would like to learn machine learning. I know it requires a mathematical background, like statistics, probability, calculus and linear algebra. However, I am a bit lost. Where should I start from? Can someone provide me a road map for how to learn the mathematical background required for machine learning?

You should begin from Dr Andrew Ng machine learning course on Coursera. It's probably the most popular course for newcomers in machine learning. It's a free course.

You should also grab "Elements of Statistical Learning" ebook PDF. It's a free book.

You may want to focus on:

Regression
Cross validation
Bias-variance tradeoff
Decision surface
Gradient descent

And more...


Loss function for choosing a subset of objects I'm trying to train a neural net to choose a subset from some list of objects.
The input is a list of objects $(a,b,c,d,e,f)$ and for each list of objects the label is a list composed of 0/1 - 1 for every object that is in the subset, for example $(1,1,0,1,0,1)$ represents choosing $a,b,d,f$. I thought about using MSE loss to train the net but that seemed like a naive approach, is there some better loss function to use in this case?

The choice of the loss function depends primarily on the type of task you're tackling: classification or regression. Your problem is clearly a classification one since you have classes to which a given input can either belong or not. More specifically, what you're trying to do is multi-label classification, which is different from multi-class classification. The difference is important to stress out and it consists in the format of the target labels.
# Multi-class --> one-hot encoded labels, only 1 label is correct

[1,0,0], [0,1,0], [0,0,1]

# Multi-label --> multiple labels can be correct

[1,0,0], [1,1,0], [1,1,1], [0,1,0], [0,1,1], [0,0,1]

MSE is used when continuous values are predicted for some given inputs, therefore it belongs to the loss functions suitable for regression and it should not be used for your problem.
Two loss functions that you could apply are Categorical Cross Entropy or Binary Cross Entropy. Despite being both based on cross-entropy, there is an important difference between them, consisting of the activation function they require.

Binary Cross Entropy

$$L(y, \hat{y})=-\frac{1}{N} \sum_{i=0}^{N}\left(y * \log \left(\hat{y}_{i}\right)+(1-y) * \log \left(1-\hat{y}_{i}\right)\right)$$
Despite the name that suggests this loss should be used only for binary classification, this is not strictly true and actually, this is the loss function that conceptually is best suited for multi-label tasks.
Let's start with the binary classification case. We have a model that returns a single output score, to which the sigmoid function is applied in order to constrain the value between 0 and 1. Since we have a single score, the resulting value can be interpreted as a probability of belonging to one of the two classes, and the probability of being to the other class can be computed as 1 - value.
What if we have multiple output scores, for example, 3 nodes for 3 classes? In this case, we still could apply the sigmoid function, ending up with three scores between 0 and 1.

The important aspect to capture is that since the sigmoid treat each output node independently, the 3 scores would not sum up to 1, so they will represent 3 different probability distributions rather than a unique one. This means that each score after the sigmoid represents a distinct probability of belonging to that specific class. In the example above, for example, the prediction would be true for the two labels with a score higher than 0.5 and false for the remaining label. This also means that 3 different loss have to be computed, one for each possible output. In practice, what you'll be doing is to solve n binary classification problems where n is the number of possible labels.

Categorical Cross Entropy

$$L(y, \hat{y})=-\sum_{j=0}^{M} \sum_{i=0}^{N}\left(y_{i j} * \log \left(\hat{y}_{i j}\right)\right)$$
In categorical cross-entropy we apply the softmax function to the output scores of our model, to constrain them between 0 and 1 and to turn them into a probability distribution (they all sum to 1). The important thing to notice is that in this case, we end up with a unique distribution because, unlike the sigmoid function, the softmax consider all output scores together (they are summed in the denominator).

This implies that categorical cross-entropy is best suited for multi-class tasks, in which we end up with a single true prediction for each input instance. Nevertheless, this loss can also be applied also for multi-label tasks, as done in this paper. To do so, the authors turned each target vector into a uniform probability distribution, which means that the values of the true labels are not 1 but 1/k, with k being the total number of true labels.
# Example of target vector tuned into uniform probability distribution
[0, 1, 1] --> [0, .5, .5]
[1, 1, 1] --> [.33, .33, .33]

Note also that in the above-mentioned paper the authors found that categorical cross-entropy outperformed binary cross-entropy, even this is not a result that holds in general.
Lastly, the are other loss that you could try which differ functions rather than cross-entropy, for example:

Hamming-Loss

It computes the fraction of wrong predicted labels
$$\frac{1}{|N| \cdot|L|} \sum_{i=1}^{|N|} \sum_{j=1}^{|L|} \operatorname{xor}\left(y_{i, j}, z_{i, j}\right)$$

Exact Match Ratio

Only predictions for which all target labels were correctly classified are considered correct.
$$ExactMatchRatio,\space M R=\frac{1}{n} \sum_{i=1}^{n} I\left(Y_{i}=Z_{i}\right)$$


How to use Machine Learning with simple games? I built a simple HTML game. In this game the goal is to click when the blue ball is above the red ball. If you hit, you get 1 point, if you miss, you lose 1 point. With each hit, the blue ball moves faster. You can test the game here.

Without using machine learning, I would easily solve this problem by just clicking when the X, Y of the blue ball was on the X, Y of the red ball. Regardless of the time, knowing the positions of the 2 elements I could solve the problem of the game.

  However, if I wanted to create an AI to solve this problem, could I?
  How would it be? I'd really like to see the AI randomly wandering
  until it's perfect.

My way to solve the problem

I click many times and watch score. If score down, add to bad_positions. If actual position in bad_positions, not click. At first he misses many times, then starts to hit eternally. This is machine learning? Deep learning? Just a bot?

var bad_positions = [];
function train(){
  var pos = $ball.offset().left;
  var last_score = score;
  if (!bad_positions.includes(pos)) {
   $('#hit').click();
    if (score < last_score){
      bad_positions.push(pos)
    }
  }
}

You have implemented a simple contextual bandit solver, which is a machine learning algorithm. A few details may be different from a full implementation, but the key elements are:

A choice of actions (click hit or don't click hit)
A reward signal that can be observed after each action (+1 for a hit, 0 for nothing happens, -1 for an attack which misses)
An observable state which affects the reward achievable (the position of the blue ball). For a contextual bandit, the state is not influenced by the action taken. This is true here.
One thing that is different about your problem from a classic contextual bandit is that the next state is predictable from the current state (whilst in a pure bandit problem it should be entirely random). However, that's not too important to your problem here, and your solver is definitely following a contextual bandit approach.
Your solver tests the score from trying different actions in each state, and narrows down the best action to take in each state. Your implementation is simple and "greedy" for a contextual bandit solver. A more typical solution would maintain an average result for each action and have a rule for how to explore actions in each state, so it could test whether results were reliable (this is very helpful with non-deterministic scenarios where bandit solvers are more often used).

  With each hit, the blue ball moves faster

Unless you somehow limit the reaction time of the agent, this is not relevant to how you write the solver. You could change the rules affecting the agent to make it relevant in the same way as it would be for a human, e.g. deciding to click means the click happens 0.1 seconds later, and the state can include observations of position just now and several 0.02 seconds going back.

In general, if you want to take this further, with more complex games and still learning how to control agent actions, you could look at simple reinforcement learning agents, such as Q-learning. If you are interested in the underlying theory of agents like this, then a good (and free) introductory text is Sutton & Barto "Reinforcement Learning: An Introduction"


What are preferences and preference functions in multi-objective reinforcement learning? In RL (reinforcement learning) or MARL (multi-agent reinforcement learning), we have the usual tuple:

(state, action, transition_probabilities, reward, next_state)

In MORL (multi-objective reinforcement learning), we have two more additions to the tuple, namely, "preferences" and "preference functions".

What are they? What do we do with them? Can someone provide an intuitive example?

In MORL the reward component is a vector rather than a scalar, with an element for each objective. So if we are using a multiobjective version of an algorithm like Q-learning, the Q-values stored for each state-action pair will also be vectors.
Q-learning requires the agent to be able to identify the greedy action in any state (the action expected to lead to the highest long-term return). For scalar rewards this is easy, but for vector values it is more complicated as one vector may be higher for objective 1, while another is higher for objective 2, and so on.
We need a means to order the vector values in terms of how well they meet the user's desired trade-offs between the different objectives. That is the role of the preference function and preferences. The function defines a general operation for either converting the vector values to a scalar value so they can be compared, or for performing some sort of ordering of the vectors (some types of orderings such as lexicographic ordering can't readily be defined in terms of scalarisation). So, for example, our preference function might be a weighted sum of the components of the vector. The preferences specify the parameters of the preference function which define a specific ordering (i.e. based on the needs of the current user). So, in the case of a weighted sum for the preference function, the preferences would be specified in terms of the values of the weights.
The choice of preference function can have implications for the types of solutions which can be found, or for whether additional information needs to be included in the state in order to ensure convergence.
I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a useful introduction to this area)

Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67-113.


What does it mean if classification error is equal between two networks but the MSE is different? I'm experimenting with training a feedforward neural network using a genetic algorithm and I've done a few tests using both the mean squared error and classification error functions as fitness heuristic in the GA.

When I use MSE as error function, my GA tends to converge around an MSE of 0.1 (initial conditions have an MSE of around 0.9). Testing system accuracy with this network gives me 95%+ for both training and testing data.

But, when I use classification error as my heuristic, my GA tends to converge around when the MSE is about 0.3. System accuracy is still around the same at 95%+.

  My question is, if you had two networks, one showing an MSE of 0.1 and one an MSE of 0.3, but both perform approximately the same in terms of accuracy, what can I deduce from the differences in MSE?

In other words: which network is "better", even if the accuracy is the same? Does a lower MSE mean anything below a certain amount? I could train my network for 100x as many generations and get a better MSE but not necessarily a better accuracy. Why?

For some context:

When the MSE is approximately 1.5 (epoch 250), the accuracy seems to match when the MSE is approximately 2.0 (epoch 50). Why does the accuracy not increase despite MSE decreasing?

MSE just measures the squared difference between actual and target values. It can still correctly classify the values, but perhaps not with the same confidence - leading to a higher loss (e.g. an output of 0.77 vs 0.98 when the target is 1). In terms of which is better, I wouldn't know without the specifics of your problem. It is possible the higher loss could be more robust since it is less likely to have overfitted on the data, yet achieves the same accuracy.


What are the differences between Q-Learning and A*? Q-learning seems to be related to A*. I am wondering if there are (and what are) the differences between them.

Q-learning and A* can both be viewed as search algorithms, but, apart from that, they are not very similar.
Q-learning is a reinforcement learning algorithm, i.e. an algorithm that attempts to find a policy or, more precisely, value function (from which the policy can be derived) by taking stochastic moves (or actions) with some policy (which is different from the policy you want to learn), such as the $\epsilon$-greedy policy, given the current estimate of the value function. Q-learning is a numerical (and stochastic optimization) algorithm that can be shown to converge to the optimal solution in the tabular case (but it does not necessarily converge when you use a function approximator, such as a neural network, to represent the value function). Q-learning can be viewed as a search algorithm, where the solutions are value functions (or policies) and the search space is some space of value functions (or policies).
On the other hand, A* is a general search algorithm that can be applied to any search problem where the search space can be represented as a graph, where nodes are positions (or locations) and the edges are the weights (or costs) between these positions. A* is an informed search algorithm, given that you can use an (informed) heuristic to guide the search, i.e. you can use domain knowledge to guide the search. A* is a best-first search (BFS) algorithm, which is a family of search algorithms that explore the search space by following the next best location according to some objective function, which varies depending on the specific BFS algorithm. For example, in the case of A*, the objective function is $f(n) = h(n) + g(n)$, where $n$ is a node, $h$ the heuristic function and $g$ the function that calculates the cost of the path from the starting node to $n$. A* is also known to be optimal (provided that the heuristic function is admissible)


Will we be able to build an artificial intelligence that feels empathy? Nowadays, robots or artificial agents often only perform the specific task they have been programmed to do.

Will we be able to build an artificial intelligence that feels empathy, that understands the emotions and feelings of humans, and, based on that, act accordingly?

Let us describe a very simple system that does something we could label as empathyc.

A chatbot answers "I am sorry to hear that. What happened?" when we type "I feel bad", and it replies "I am glad to hear that. Fancy some music?" when we type "I feel good".

Somehow, it perceives a human emotion, and acts accordingly.

Planes fly but they do not fly as birds. Similarly, we can expect artificial empathy from an AI, not necessarily natural empathy as we feel it.


Adversarial Game playing using RL I asked a question relating to TicTacToe playing in RL. From the answer it seems to me a lot is dependent on the opponent (rightly so, if we write down the Expectation Equations). My question is (in the context of TicTacToe or Chess):

How to make the RL player a perfect player (TicTactoe) or an expert (chess). As far as TTT is concerned when playing against a perfect player an RL will become perfect conditioned on the clause that the opponent is perfect. So will this hold true if the same RL algo, with its learned values are used to play some other lesser perfect players?
The question can be extended to the scenario, can a RL player with pre-trained values (assume from a perfect or expert opponent) be used in any scenario with best results?

Note: The problem is more severe in chess, since experts will use some kind of opening moves which will not match with say a random player and thus finding values for those states becomes a problem, since we have not encountered it during training time.

Footnote: Any resources on Game Playing RL is appreciated.

I would say a good way to make a good agent would be making it play against itself. As you go through several episodes, with a good exploration and exploitation balance, both will gradually learn and converge to Q*(s,a).´

  So will this hold true if the same RL algo, with its learned values
  are used to play some other lesser perfect players?

As long as the states that are played  (or approximations if you are using Function approximation methods) were simulated enough times during  training, it will play well against any kind of opponent.

If you are training against a completely perfect opponent and you are not using function approximation, I believe you could get to an incomplete Q(s,a) table and as such not be able to predict the best play when facing certain states.


Why is the learning rate generally beneath 1? In all examples I've ever seen, the learning rate of an optimisation method is always less than $1$. However, I've never found an explanation as to why this is. In addition to that, there are some cases where having a learning rate bigger than 1 is beneficial, such as in the case of super-convergence.
Why is the learning rate generally less than 1? Specifically, when performing an update on a parameter, why is the gradient generally multiplied by a factor less than 1 (absolutely)?

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition
$$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$
where $a_t$ is the learning rate at iteration $t$, does not hold (given that a number bigger than $1$ squared becomes a bigger number), so stochastic gradient descent is not generally guaranteed to converge to a minimum [1] (although the condition \ref{1} is a sum from $t=0$ to $t=\infty$, but, of course, we only iterate for a finite number of iterations). Moreover, note that, if the learning rate is bigger than $1$, you are essentially giving more weight to the gradient of the loss function than to the current value of the parameters (you give weight $1$ to the parameters).
This is probably the main reason why the learning rate is usually in the range $(0, 1)$ and there are methods to decay the learning rate, which can be beneficial (and there are several explanations of why this is the case [2]).


How should I label the classes in RNA? I have a project, which is the keyboard biometrics of users.

suppose I have 3 users,
I do not know how to label in two types of class, (+ 1, -1).

If I want to verify the identity to user1, my idea of class designation would be:

       TIMES                LABEL
user 1
9.4  9.2  1.0  3.4  0.5      1
9.4  9.2  1.0  3.4  0.5      1
9.4  9.2  1.0  3.4  0.5      1
9.4  9.2  1.0  3.4  0.5      1
9.4  9.2  1.0  3.4  0.5      1

user 2
0.1  3.2  1.0  1.2  1.7      -1
3.4  1.2  3.0  1.1  2.8      -1
2.4  2.2  3.0  1.6  2.9      -1
1.4  3.2  2.0  2.6  3.6      -1
3.4  0.2   3.0  2.7  3.5     -1

user N
0.2  1.4  4.5  3.7  2.9      -1
9.2  1.5  7.6  2.6  2.6      -1
9.3  1.6  7.5  2.9  3.4      -1
9.8  3.8  6.6  2.8  2.5      -1
9.8  2.8  1.7  3.8  1.6      -1

but as my system has more and more users classes -1 will be too many compared to classes +1,
How should I label the classes?

From a purely dataset-development perspective I would just label the classes with numbers starting from 0 (ID of the first person) to N (ID of the last person.

During training however, what you do with those classes will vary depending on the type of architecture you are training.  For example, if you are building a neural-network classifier you could simply have multiple output nodes.  When running backprop with a label for the ith person you could put a 1 on the ith output layer and -1 on the rest of them.  This is similar to this answer as well

Whatever you end up doing, try not to label your data to the benefit of the algorithm you're trying to run unless you really know what you're doing.  You don't want to accidentally lose information or make your training data less interpretable and harder to deal with in the long run.


What is the difference between simple reflex and model-based agents? What is the difference between simple reflex and model-based agents?

What is the role of the internal state, in the case of model-based agents?

A simplex reflex agent takes actions based on current situational experiences.

For example, if you set your smart bulb to turn on at some given time, let's say at 9 pm, the bulb won't recognize how the time is longer simply because that's the rule defined it follows.

A simple reflex agent doesn't compute complex computational problems nor exhibit intelligence.

A model-based agent takes actions based on historical situational experiences. It performs actions basing on its internal state.

The internal state of the model-based agent acts as a "knowledge base", which allows actions to be performed by representing of unobserved aspects of the current state depending on percept history nor historical situational experiences.

For instance, a research-based planetary system can predict the planet's climatical conditions or weather basing on some geo-information knowledge base.


Why would the learning rate curve go backwards? I'm working on recognizing the numbers 3 and 7 using the MNIST data set. I'm using cnn_learner() function from fastai library.
When I plotted the learning rate, the curve started going backward after a certain value on X-axis. Can someone please explain what does it signify?

I have not used fastai library but this also happens on tensorboard when you have more than one training being recorded on the same plot.
Looking at the picture, I think this is a very special type of graph because for a single LR value you have 2 loss values associated. Put in other words, you have the same LR value for different loss values. My guess it that some time-dependent issue is messing things here.
Another intuition that may help to solve the issue is representing data differently. If you rotate the graph 90º left-wise you could see how the LR is evolving with different loss values. LR should be decreasing along with loss value, but in this case it is not like that either. So review how you are setting the LR too!
My list of to-check would be:

Check how you update LR according to the loss value
Check the LR recorder instance is only being used by one training at a time
Try plotting the value of LR (y) against your loss values (x)
Try plotting the value of LR (y) against the current epoch number (x)

Hope it helps


What's the difference between RMSE and Euclidean distance, and when to use a custom loss? I'm searching for a loss function that fits my project. Actually, I have two questions, but they are in the same direction. I take a look at the definition of the root mean squared error (RMSE) and the Euclidean distance and they look the same to me. That's why I want to know the difference between the two. What would be the difference if I use RMSE as a loss function or the euclidean distance?

The second question is how to search for a loss function. I mean I know it depends on the problem and common things are MSE for regression and cross-entropy for classification, but let's say I have a specific problem, how do I search for a loss function?

I also saw that some people use a custom loss function and most of the deep learning frameworks allow us to define a custom loss function, but why would I want to use a custom one? How do I get the intuition that I need a custom loss function?

To be more specific, I'm doing a project where I need to reduce the GPS error of a vehicle. I have some vehicle data and my neural network will try to predict the longitude and latitude, so it's a regression problem. That's why I thought that the Euclidean distance would make sense as a loss function, right? Now, somehow MSE also makes sense to me because it is getting the difference between prediction and ground truth. Does this make sense to you as a professional ML engineer or data scientist? And if there would be a custom loss function that you can use, what would you suggest and why?

For the first question, RMSE and Euclidean distance have no difference, not that i know of.

For the second question, you only need the common loss function for normal tasks.

MSE is a common loss function used in linear regression tasks as well as loss function similar in nature like the RMSE. For classification tasks, Cross Entropy Loss is preferred. For logistic regression, use Binary Cross Entropy loss. See this for details:

  Cross-entropy loss, or log loss, measure the performance of a
  classification model whose output is a probability value between 0 and
  1. It is preferred for classification, while mean squared error (MSE) is one of the best choices for regression. This comes directly from
  the statement of your problems itself. In classification you work with
  a very particular set of possible output values thus MSE is badly
  defined.

  To better understand the phenomena it is good to follow and understand
  the relations between

  Cross-entropy

  Logistic regression (binary cross-entropy)

  Linear regression (MSE)

  You will notice that both can be seen as a maximum likelihood
  estimator (MLE), simply with different assumptions about the dependent
  variable.

  When you derive the cost function from the aspect of probability and
  distribution, you can observe that MSE happens when you assume the
  error follows Normal Distribution and cross-entropy when you assume
  binomial distribution. It means that implicitly when you use MSE, you
  are doing regression (estimation) and when you use CE, you are doing
  classification.

Source: https://intellipaat.com/community/2015/why-is-the-cross-entropy-method-preferred-over-mean-squared-error-in-what-cases-does-this-doesnt-hold-up

For a custom loss function, there is examples like the triplet loss and cases where you need to optimize two loss at once, then you need a custom loss function. For the case of triplet loss, it is used in one-shot learning tasks like face recognition. There, face images are fed through a CNN to get embeddings about description of the face. In the training phase there is three images per batch. One anchor, one positive (which is the same person as anchor
) and one negative (different person from anchor). The loss function, triplet loss, maximizes the distance between the anchor embedding and the negative embedding. In reverse, it also minimize the distance between the anchor embedding and the positive embedding. The loss function is like this:

As you can see, there is two components here, and they are subtracted to maximize the second component and minimize the first. The alpha value is to make the loss positive as the gradient descent optimizes the loss to be as near to zero as possible.

The case of style transfer also introduces a custom loss function. Here is the loss function:

Style transfer basically alters the input image to be styled like the style image of the system.

The loss function consist of style loss and content loss. The content loss decreases the distance of the output image and the style loss decreases the distance of the output image and the style in an artistic sense. The second loss makes the output image imitate the style of the the style image.

In both scenario a custom loss function is used. In both cases, the custom loss function is used to optimize two losses together, either positively or negatively. A weight can also be introduced to weight the two losses as shown in the second case. However the second case introduce a custom loss function. That leads to the second use of custom loss function. They can be used for specific tasks and optimize a specific goal. The first use for custom loss function can be explored but the second requires careful engineering and research.

So in short, custom loss function are used for either combining two loss or a specific task.

Hope I can help you.


If neurons are only defined for values between 0 and 1, how does ReLU differ from the identity? I'm struggling to understand the underlying mechanics of CNNs so any help is appreciated. I have a network with a ReLU activation function which does perform signifigantly better than one with sigmoid. This is expected as ReLU solves the vanishing gradient problem. However, my understanding was the reason we implement nonlinearities is to separate data which cannot be separated linearly. But if ReLU is linear for all values we care about it shouldn't work at all?

Unless, of course, neurons are defined for negative values but then my question becomes "why does ReLU solve the vanishing gradient problem at all?", since the derivative of ReLU for x<0 = 0

You are right about the fact that we use nonlinearity for classes that cant be seperated by a straight line. If you think about curves,as we know from the calculus, you can approximate a nonlinear function with lines. If you have infinite amount of lines you can mimic exactly the same function. When you increase or decrease the weight of some neuron you essentially increase or decrease the length of the line it represents. And ReLU solves the vanishing gradient problem so they are superior to sigmoid in my opinion too.


Suicide Predictor and Locator Suicide is on the increase in my country and most victims tend to leave early traces from text messages, social media accounts, search engine queries. So I came up with the idea to develop an AI system with the following features:

Ability to read text messages searching for suicide trigger words
Read chats also for the same trigger words
Incorporate synchronization of words from software/browsers on recent words typed
Learning algorithm to predict the next action after taking notes of the use of these suicide trigger words
Ability to access any cellphone, Android, computer, websites

Is this feasible or not feasible?

This question seems to be specifically about your idea so I'll answer as such - any general 'can digital activity be used to predict suicide attempts' is probably too broad for this site.
As I see it you want to use what a person types, across all types of digital media, to get some measure of their risk of suicide attempt.
Two key questions in any machine learning problem

Can we create variables as predictors?
Is the data there? i.e. do we have labelled data already

As far as creating variables goes a lot of people don't realise that machine learning (or 'AI') isn't about dumping a load of unedited data into an algorithm and letting it figure it out - 90% of the work is in making those variables into something sensible. In this case it may involve a fair bit of psychology. Perhaps you look at changes in message length - has someone gone from writing essays to short answers? (I'm not a psychologist, that may well be way off the mark). The use of particular words or phrases may come into it - you would need to do a fair bit of research into the differences between those who attempted suicide and those who didn't (perhaps ranking words by frequency and see if there are distinct differences).
This brings us to our second question (or perhaps it should be the first) 'is the data there'? You want to predict suicide risk from digital activity so you'll need the digital activity of all different types of people with a label of whether or not they attempted suicide (and when - no point collecting the data two years after their attempt). Do you think this dataset exists? I'm sceptical - the level of detail you would need (both into someone's digital activity and their personal mental state) is unlikely to be recorded.
It isn't to say I think its an idea to be thrown out, only that you need to be realistic about the amount of effort required to do something like this. You would need to carry out research to collect this data, need to plan out your ideas before hand to know what you would like to collect (the worst thing is to find, after a year of data collection, that you didn't monitor a variable you would have now found vital).


ANN’s single output representing value Can ANN with only one neuron in output layer be trained in a way that output neuron’s value (0-1) can be representation of some real value, like for example height.

In other words,can neural network given the inputs predict the height of a person by outputing values from 0 to 1. Zero being the 50 cm and 1 being 250cm.Or it will always gravitate to 0 or 1?Can it predict a height of 150cm (0.5) ?

Yes, this is possible. The only restriction on the range of output values is due to the activation function, if there is one. If, for example, the activation function is a sigmoid, the output would be restricted to values between 0 and 1. You can of course choose an activation function best suited to your problem, even if that means having none.

In your case, if you really want the output of your network to be between 50 and 250, you could use a scaled and shifted sigmoid as your activation: 200*sig(w*x+b)+50.

But, as was pointed out in comments below, having a single output neuron may not be the best architecture for this problem.


What are the AI technologies currently used to fight the coronavirus pandemic? The ongoing coronavirus pandemic of coronavirus disease 2019 (COVID-19), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), as of 29 September 2020, has affected many countries and territories, with more than 33.4 million cases of COVID-19 have been reported and more than 1 million people have died. The live statistics can be found at https://www.worldometers.info/coronavirus/ or in the World Health Organization (WHO) site. Although countries have already started quarantines and have adopted extreme countermeasures (such as closing restaurants or forbidding events with multiple people), the numbers of cases and deaths will probably still increase in the next weeks.
Given that this pandemic concerns all of us, including people interested in AI, such as myself, it may be useful to share information about the possible current applications of AI to slow down the spread of SARS-CoV-2, to help infected people or people in the healthcare sector that have been uninterruptedly working for hours to attempt to save more lives, while putting at risk their own.
What are the existing AI technologies (e.g. computer vision or robotics tools) that are already being used to tackle these issues, such as slowing down the spread of SARS-CoV-2 or helping infected people?
I am looking for references that prove that the mentioned technologies are really being used. I am not looking for potential AI technologies (i.e. research work) that could potentially be helpful. Furthermore, I am not looking for data analysis tools (e.g. sites that show the evolution of the spread of coronavirus, etc.)

According to the Baidu Research's blog post How Baidu is harnessing the power of AI in the battle against coronavirus (12-03-2020), there are already some artificial intelligence tools or algorithms being used to fight the coronavirus.
Given that I cannot confirm that these AI tools and algorithms I will mention are really being used in practice, I will only quote the parts of the blog post that potentially answer my original question.
To give some context, similar to HIV viruses, the virus that is causing the coronavirus pandemic, SARS-CoV-2 is capable of rapidly mutating, making vaccine development and virus analysis difficult.
AI-powered and non-contact infrared sensor system

Baidu has developed several tools that are effective in building awareness and screening populations, including an AI-powered, non-contact infrared sensor system that provides users with fast multi-person temperature monitoring that can quickly detect a person if they are suspected of having a fever, one of the many symptoms of the coronavirus. This technology is currently being used in Beijing's Qinghe Railway Station to identify passengers who are potentially infected where it can examine up to 200 people in one minute without disrupting passenger flow.

AI-powered pneumonia screening and lesion detection system

By leveraging PaddlePaddle and the semantic segmentation toolkit PaddleSeg, LinkingMed has developed an AI-powered pneumonia screening and lesion detection system, putting it into use in the hospital affiliated with XiangNan University in Hunan Province. The system can pinpoint the disease in less than one minute, with a detection accuracy of 92% and a recall rate of 97% on test data sets.

Automated HealthMap system

The Boston Children's Hospital used an automated HealthMap system that scans online news and social media reports for early warning signs of outbreaks, which led to the initial awareness that COVID-19 was spreading outside China.

Autonomous vehicles carry out non-contact tasks

Access to health care and resources at a moment's notice is vital for battling the spread of the coronavirus. Autonomous vehicles are playing a useful role in providing access to necessary commodities for health-care professionals and the public alike by delivering goods in infected areas and disinfecting hospitals, effectively minimizing person-to-person transmission and alleviating the shortage of medical staff.
Apollo, Baidu's autonomous vehicle platform, partnered with a local self-driving startup called Neolix to deliver supplies and food to the Beijing Haidian Hospital.


Why is this simple neural network not training? I have created a Tf.Sequential model which outputs 1 for numbers bigger then 5 and 0 otherwise:

const model = tf.sequential();
model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
const xs = tf.tensor2d([[1], [2], [3], [4], [6], [7], [8], [9]]);
const ys = tf.tensor2d([[0], [0], [0], [0], [1], [1], [1], [1]]);
model.fit(xs, ys);
model.predict(xs).print();

With 5 hidden neurons, not even the right trend is detected. Sometimes all the number are too low, or the outputs decrease even if the inputs increase or the outputs are too high.

I later thought that the best way to do this is to have 2 neurons, where 1 is for the input and the other applies a sigmoid function to the input. The weight and bias should easily be adjusted to make the ANN work.

const model = tf.sequential();
model.add(tf.layers.dense({ units: 1, activation: 'sigmoid', inputShape: [1]}));
model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
const xs = tf.tensor2d([[1], [2], [3], [4], [6], [7], [8], [9]]);
const ys = tf.tensor2d([[0], [0], [0], [0], [1], [1], [1], [1]]);
model.fit(xs, ys);
model.predict(xs).print();

Sometimes, this ANN does detect the right trend (the higher the input, the higher the output), but still, the results are never correct and are usually simply too high, always providing an output too close to 1.

How do I make my ANN work, and what have I done wrong?

Edit:

This is the code I'm using now, same problem as before:

const AdadeltaOptimizer = tf.train.adadelta();

const model = tf.sequential();
model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
model.compile({loss: 'meanSquaredError', optimizer: AdadeltaOptimizer});
const xs = tf.tensor1d([1, 2, 3, 4, 5, 6, 7, 8, 9]);
const ys = tf.tensor1d([0, 0, 0, 0, 0, 1, 1, 1, 1]);
model.fit(xs, ys, {
epochs: 2000,
});
model.predict(xs).print();

tf.losses.meanSquaredError(ys, model.predict(xs)).print();

You should try bigger learning rate / more epochs.

What you see is basically output of randomly initialized weights. Default learning rate for SGD is 0.001, it's definitely not enough for weights to change quick enough with single batch epochs and 9 examples.

I never used TF.js, but in keras (that's look very similar) you can set it like this

sgd = keras.optimizers.SGD(lr=1)
model.compile(sgd, 'mse')

model.fit(xs, ys, epochs=200)

Another approach would be to use adaptive learning rate optimizer, for example adadelta. It still requires quite a lot of epochs, but much less, then SGD with standard learning rate and you don't need to set LR manually.

model.compile('adadelta', 'mse')
model.fit(xs, ys, epochs=2000)

Here you can read how different optimizers work and converge

Edit:
Another thing with tf.js (and javascript code in general) is it doesn't guarantee order of operations. So, in your case it prints evaluation before it finishes training.

This code should perform fine. Take note that we evaluate model in then clause.

const model = tf.sequential();
model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
model.compile({loss: 'meanSquaredError', optimizer: 'adadelta'});
const xs = tf.tensor1d([1, 2, 3, 4, 5, 6, 7, 8, 9]);
const ys = tf.tensor1d([0, 0, 0, 0, 0, 1, 1, 1, 1]);
model.fit(xs, ys, {epochs: 1000}).then(h => {
   console.log("Loss: " + h.history.loss[0]);
   model.predict(xs).print();
});

You can also use async/await approach, for example as it described in this answer


Can translational invariance of CNNs be unwanted if object is likely in certain positions? Various texts on using CNNs for object detection in images talk about how their translation invariance is a good thing. Which makes sense for tasks where the object could be anywhere in the image. Let's say detecting a kitten in household images.

But let's say, you already have some information about the likely position of the object of interest in the image. For example, for detecting trees in a dataset of images of landscapes. Here in most cases, the trees are going to be in the bottom half of the image while in some cases they might be at the top (because it's on a hill or whatever). So you want your neural network to learn that information -- that trees are likely connected to the bottom part of the image (ground). Is this possible using the CNN paradigm?

Thank you

According to your example:

Trees will likely be in the bottom half of the image. Still, you will not know whether there will be one, two or five trees. Thanks to translation invariance property of CNN's, each tree will activate filters responsible for tree detection. You still need to handle those few exceptions where trees are on hill.

To achieve better results in this particular case, you might want to consider some kind of focus mechanism, that will try to get rid of unwanted part of picture, in case when there are (for example) no hills. Take a look at Spatial Transformer Networks. During training it learns to predict spatial transformation (for example zoom) that will help "main" classifier to predict class of image.


Is there an equivalent model to the Hidden Markov Model for continuous hidden variables? I understand that Hidden Markov Models are used to learn about hidden variables $z_i$ with the help of observable variables $\xi_i$. On Wikipedia, I read that while the $\xi_i$'s can be continuous (say Gaussian), the $z_i$'s are discrete. Is this necessary, and why? Are there ways in which I could extend this to continuous domains?

Kalman filter is what you're looking for.
According to Wikipedia:

The Kalman filter may be regarded as analogous to the hidden Markov model, with the key difference that the hidden state variables take values in a continuous space (as opposed to a discrete state space as in the hidden Markov model).


Finding the right questions to increase accuracy in classification Lets say I have a list of 100k  medical cases from my hospital, each row = patient  with symptoms (such as fever , funny smell, pain etc.. ) and my labels are medical conditions such as Head trauma, cancer , etc..

The patient come and say "I have fever" and I need to predict his medical condition according to the symptoms.According to my data set I know that both fever and vomiting goes with condition X. So i would like to ask him if he is vomiting to increase certainty in my classification.

What is the best algorithmic approach to find the right question (generating question from my data set of historical data). I thought about trying active learning on the features but I am not sure that it is the right direction.

The problem you're trying to address can, in some sense, be viewed as a Feature Selection problem. If you look for literature using only those words, you're not going to find what you're looking for though. In general, "Feature Selection" simply refers to the problem where you already have a large amount of features, and you're simply deciding to select which ones to keep and which ones to throw away (because they're not informative or you don't have the processing power to try training with all features for example).

I'd recommend looking around for a combination of "Feature Selection" and "Cost-Sensitive". This is because, in your case, there are costs associated with selecting features; values may be costly to obtain for some features. Searching for this combination leads to publications which look to be interesting for you, such as:

Cost-sensitive feature selection using random forest: Selecting low-cost subsets of informative features
Cost-sensitive Dynamic Feature Selection
Cost-Effective Feature Selection and
Ordering for Personalized Energy Estimates
and probably much more...

I cannot personally vouch for any of those techniques since I've never used them, but those papers certainly look relevant for your problem.

When you're looking around for more literature, terms like "cost", "cost-based", maybe "budgeted" are crucial to include. If you don't include those, you're just going to get papers on problems like:

Feature Selection: given a set of features/columns, which ones am I going to use across all samples/instances/rows?
Feature Extraction: given data (typically without clear human-defined features, like images, sound, etc.), how am I going to extract relevant features from this?
Active Learning: given a bunch of samples without labels but feature values already assigned, which one would I like an oracle/human expert/etc. to have a look at so that they can tell me what the true label is?

Those kinds of problems all do not really appear to be relevant in your case. Active Learning may be somewhat interesting in that it is about trying to figure out which rows would be valuable to learn from, whereas your problem is about which columns would be valuable to learn from. There does seem to be a connection there, Active Learning techniques might to some extent be able to inspire techniques for your problem, but just that; inspire, they likely won't be 100% directly applicable without additional work.


How to handle changing goals in a DQN? I created a virtual 2D environment where an agent aims to find a correct pose corresponding to a target image. I implemented a DQN to solve this task. When the goal is fixed, e.g. the aim is to find the pose for position (1,1), the agent is successful. I would now like to train an agent to find the correct pose while the goal pose changes after every episode. My research pointed me to the term "Multi-Objective Deep Reinforcement Learning". As far as I understood, the aim here is to train one or multiple agents to achieve a policy approximation that fits all goals.
Am I on the right track or how should I deal with different goal states?

The simplest thing you can do is to add data regarding the target pose to the state vector. This will allow any generalisations that the agent learns that apply to similar poses to be used directly.
Clearly in normal use, where the target pose remains fixed during the episode, then that part of the state information will not change either during the episode. You will also need to train with a large variety of target poses - so training will take longer.
Multi-Objective Deep Reinforcement Learning is slightly different in that it attempts to resolve prioritising between multiple sub-goals. It would also be a more complex solution, whilst augmenting the state vector should allow you to continue using a very similar DQN set up as you have already.