Can stochastic gradient descent be properly used in any sample based learning algorithm in Reinforcement Learning? Assuming we use an MSE cost function of the form
$$ \sum_s\mu(s)(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2 =  E_{\mu(s)}[(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2])$$
The Stochastic Gradient Descent is used to approximate the true update algorithm, which looks like this
$$\theta_{t+1} = \theta_{t} - \frac{\eta_t}{2}\nabla_{\theta}(E_{\mu(s)}[(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2])$$
to this
$$\theta_{t+1} = \theta_{t} - \frac{\eta_t}{2}\nabla_{\theta}(U_t-\hat{V}(S_t,\theta_t))^2$$
where, for simplicity, $U_t$ represents an unbiased estimate of the true value function $V_{\pi}(s_t)$. This expression is the source of many learning algorithms used in reinforcement learning.
One of the conditions for SGD requires that samples used for updating the parameters must be I.I.D according to the distribution $\mu(s)$. However, in both on-policy and off-policy learning methods, updates at each time-step are based on trajectories generated. Since, along a trajectory, the state $s_{t+1}$ depends on the state at ${s_t}$, this means that the sample used to update $\theta_t$ and $\theta_{t+1}$ are not independent. Many, if not all, sample-based learning algorithms used in RL rely on using SGD, such as the Gradient Monte Carlo Algorithm but I've not really seen anywhere that mentions these algorithms have the "issue" that I mention so I feel like I'm missing something. More formally,

My Question: Does the fact that parameter updates are not I.I.D mean we can't really use stochastic gradient descent AT ALL in learning algorithms, and, if so, why then do these algorithms "work"?

As far as I know, this question applies equally to all forms of parameterised function approximation that are used with learning algorithms (tabular functions*, linear functions and non-linear functions). But, if anyone knows a special reason as to whether these cases should be treated separately could they make clear why
*I understand that when learning algorithms with tabular functions, there exists theory beyond SGD that ensures convergence, however, I'm not entirely sure what this theory is and whether if this makes them exempt, so if anyone knows whether or not it does make them exempt could they also make this clear!

Edit:
It has been highlighted in the comments that replay buffers have been used to resolve the issue of correlated sampling in cases such as DQN and variants of it. This implies correlated sampling is an issue in these cases. Aside from this, I've not heard of replay buffers being used elsewhere (correct me if I'm wrong), so why are replay buffers needed with this off-policy NN approach but not in other learning algorithms given that they all suffer from the issue of correlated sampling.

First I will address the issue of Tabular methods. These do not use SGD at all. Although the updates are very similar to an SGD update there is no gradient here and so we are not using SGD. Many Tabular methods are proven to converge, for instance the paper by Chris Watkins titled "Q-Learning" introduces and proves that Q-learning converges. Also you include tabular methods as being parameterised function approximators. This is not true. Tabular methods maintain an estimate of the value function in a look-up table for each state-action pair and there is no function approximator being used.
Now for non-tabular methods, i.e. Deep Reinforcement Learning. Here we are using SGD only to optimise the parameters of the networks (assuming of course that we are using NN's as our function approximators, but this is a fair assumption if you read the literature). This is why off-policy methods are typically preferred because they can use a replay buffer which allows the use of data from any past trajectory. When using a replay buffer we sample random past experiences which de-correlates the data and allows the i.i.d. assumption to hold when using SGD.
If we were to use an on-policy algorithm, in theory SGD may not converge to any local optima because we are violating the i.i.d. assumption. However, all this means is that we are not guaranteed to converge. For instance, I have run an experiment using REINFORCE (an on-policy learning algorithm) using NN's as function approximators and they were able to obtain an optimal policy. However, this was for a very simple environment using modestly sized networks, so this is likely why they were able to be trained using non-i.i.d. data. A more in depth question/answer as to why NN's require i.i.d. data can be found here.
To address your edit, Replay Buffers are not required but if we can use them, then why would we not? It helps to maintain the i.i.d. assumption and so it helps us obtain a local optima. If we did not use them then there would not be the guarantee that it would converge. As an aside, Replay Buffers are typically used because they make the algorithm have a greater sample efficiency - this means we can obtain an optimal policy using much less data.
If you are wondering why don't on-policy methods use a replay buffer, the answer is because they are on-policy. The actions used in updates must be taken according to our current policy that we are learning the value functions for (or are optimising the policy of in Policy Gradient Methods). This is not the case in off-policy algorithms - e.g. in Q-learning we are learning the value functions of the greedy policy but we follow a different policy that allows for exploration.


How can one intuitively understand generative v/s discriminative models, specifically with respect to when each is useful? I'm trying to gain some intuition beyond definitions, in any possible dimension. I'd appreciate references to read.

The intution that I have about these is that generative are "from abstract to concrete" whereas discriminative models are "from concrete to abstract".

For example: Detecting if a photo has a cat or not is about going from the photo i.e concrete to the abstract concept of a cat. Whereas generating a photo of a cat given some abstract properties about the cat is going from abstract to concrete.


How should I implement the state transition when it is a Gaussian distribution? I am reading this paper Anxiety, Avoidance and Sequential Evaluation and is confused about the implementation of a specific lab study. Namely, the authors model what is called the Balloon task using a simple MDP for which the description is below:

My confusion is the following sentence:

...The probability of this bad transition was modeled using normal density function, with parameters $N(16, 0.5)$

But the fact that this is a continuous, normal distribution makes me stumped. In MDP's, usually there is a nice, discrete transition matrix and so there is no ambiguity as to how to implement it. For instance, if they said the transition to a bad state is modeled by a Bernoulli random variable with parameter $p,$ then it is clear how to implement it. I would do something like:
def step(curr_state, curr_action):
   if uniform random variable(0,1) < p:
      next_state = bad state

But they are using a normal random variable for this "bad" transition, so how do I implement this?

I figured this out by going to the author's publicly available github code. It turned out  the authors were just generating the transition probability $p$ from $\mathcal{N}(\mu,\sigma^2)$ at the beginning of each episode for some reason. Answering it myself for the sake of not leaving this question unanswered.


How is an architecture composed of a second model that validates the first one called in machine learning? I have a mix of two deep models, as follows:

if model A is YES --pass to B--> if model B is YES--> result = YES
if model A is NO ---> result = NO

So basically model B validates if A is saying YES. My models are actually the same, but trained on two different feature sets of same inputs.

What is this mix called in machine learning terminology? I just call them master/slave architecture, or primary/secondary model.

Not in terms of models, but there is a terminology called 'Hierarchical learning', wherein if your model has a task to classify disease, then, If it detects a presence of a disease (disease/ no disease), then it proceeds to further classify a disease(class A/B/C/...). Else it does not proceed.
This technique of hierarchical learning is very common amongst supervised learning tasks.

Now according to your question, you have two models and I assume that they have different tasks and provide a binary outcome(yes/no). Here, you can call it as 'Multitask learning', where the output of task1 is given to task2 for processing.
If task1 detect the presence of disease, then task2 classifies disease into various classes / or segment it / localize it etc.


How to manage large amounts of image data for training? Right now, I am trying to synthesize training images for a CNN and due to the nature of the application, there is a finite number of sample images to learn from.

From other research, I expect to be using about 200,000 training images at a resolution of 1280*720, which with 3 channel at 8 bits will take about 550 GB to save uncompressed. This number can and probably will rise in the future, meaning more memory that I will need to provide.

I imagine that there are applications that required even more training data with higher complexity and that there are solutions to handling that such as compression techniques and the like.

My question: Are there solutions for the memory management beyond compressing the images with JPEG and such besides generating and instantly consuming the pictures without saving them to permanent memory?

I suggest you fine-tune an existing model. Knowledge transfer models in many image processing tasks are now open sourced and you can build your model on top of them. Also, knowledge transfer models are trained on a large datasets and can quickly converge to your case-study with a little of task-specific extra training.

  This way you will use few data to tune the model which leads to less
  memory use and less training time. You will also take advantage from a
  ready-to-use architecture and get accurate results.

Depending on your case-study you can choose from this list of computer vision pre-trained models.


Is better to spend parameters on weights or bias? If a neural network has a limited number of neuron parameters to find, -let's say only 1000 parameters-, it is generally better to spend the parameters on weights or neuron bias?

For example, if each neuron has 2 weights and one bias, it uses 3 parameters per neuron, so only 333 neurons would be available.

But if each neuron uses no bias parameter, then 500 neurons are available with 1000 parameters.

I'm concerned with overfiting by using too many parameters, so I want to minimize the number of parameters meanwhile maximizing the quality of the result.

First of all, your estimates are a bit off. If you have 300 neurons, you won't have just 2 weights per neuron, but much more, assuming full connectivity

Bias isn't just an extra parameter to fit, it is an important adjustable parameter that sets the offset of the separating hyperplane represented by each neuron. Think of the simple equation $ax+b$, there's no way to shift the line unless you use the $b$ (bias) part.

This would be especially important for small number of nodes and classification tasks (think perceptrons etc)


Why do you not see dropout layers on reinforcement learning examples? I've been looking at reinforcement learning, and specifically playing around with creating my own environments to use with the OpenAI Gym AI. I am using agents from the stable_baselines project to test with it.

One thing I've noticed in virtually all RL examples is that there never seems to be any dropout layers in any of the networks. Why is this?

I have created an environment that simulates currency prices and a simple agent, using DQN, that attempts to learn when to buy and sell. Training it over almost a million timesteps taken from a specific set of data consisting of one month's worth of 5-minute price data it seems to overfit a lot. If I then evaluate the agents and model against a different month's worth of data is performs abysmally. So sounds like classic overfitting.

But is there a reason why you don't see dropout layers in RL networks? Is there other mechanisms to try and deal with overfitting? Or in many RL examples does it not matter? e.g. there may only be one true way to the ultimate high score in the 'breakout' game, so you might as well learn that exactly, and no need to generalise?

Or is it deemed that the chaotic nature of the environment itself should provide enough different combinations of outcomes that you don't need to have dropout layers?

Dropout essentially introduces a bit more variance. In supervised learning settings, this indeed often helps to reduce overfitting (although I believe there dropout is also already becoming less.. fashionable in recent years than in the few years before that; I'm not 100% sure though, it's not my primary area of expertise).

In Reinforcement Learning, additional variance is not really what we're looking for. There already tends to be a large amount of variance in the learning signals that we get, and this variance already tends to be a major issue for learning stability and/or learning speed. For example:

Randomness in action selection leads to variance in returns that we observe
There may be randomness inherent to the environment itself, leading to extra variance in our observations (some environments are nondeterministic)
Unlike Supervised Learning settings, in Reinforcement Learning we often actually use our own predictions as a part of our loss function / training signal. For example, in temporal-difference learning (like Q-learning / DQN), the target that we update towards looks like $r + \max_{a'} Q(s', a')$. In that term, only the $r$ is a ground-truth observation (like we would use in supervised learning), and the other term is our own prediction. During a learning process, those latter parts (our own predictions) are changing over time. This is a "moving target'' problem, which can be viewed as additional variance in our learning signals.

Many important parts of Deep RL algorithms (without which our training processes empirically turn out to destabilize and break down) are very much tailored towards reducing that variance. For example, Target Networks in DQN were introduced specifically to reduce the moving target problem. From this point of view, it's not surprising that if we were to add more artificial variance through other means again (such as dropout), that this would hurt performance / destabilize learning.

  Is there other mechanisms to try and deal with overfitting? Or in many RL examples does it not matter? e.g. there may only be one true way to the ultimate high score in the 'breakout' game, so you might as well learn that exactly, and no need to generalise?

In the majority of current (Deep) Reinforcement Learning research, overfitting is indeed not viewed as a problem. The vast majority of RL research consists of training in one environment (for example Cartpole, or Breakout, or one particular level in Pacman, or navigating in one specific maze, etc.), and either constantly evaluating performance during that learning process, or evaluating performance after such a learning process in the same environment.

If we were to compare that evaluation methodology to what happens in supervised learning... we are basically evaluating performance on the training set*. In supervised learning, this would be absolutely unacceptable, but in RL it is very much treated as acceptable and more rule than exception. Some say this is simply a problem in current RL research, something that needs to change. It could also be argued that it's not necessarily a problem; if we really are able to train the agent in precisely the same environment that we wish to deploy it in later... well, then what's the problem with it overfitting to that environment?

So, when we're using the evaluation methodology described above, indeed we are overfitting to one specific environment, but overfitting is good rather than bad according to our evaluation criteria. It is clear that this methodology does not lead to agents that can generalize well though; if you consistently train an agent to navigate in one particular maze, it will likely be unable to navigate a different maze after training.

*Note: the truth, in my opinion, is slightly more nuanced than that we are really "evaluating on the training set" in RL. See, for example, this nice thread of tweets: https://twitter.com/nanjiang_cs/status/1049682399980908544

  I have created an environment that simulates currency prices and a simple agent, using DQN, that attempts to learn when to buy and sell. Training it over almost a million timesteps taken from a specific set of data consisting of one month's worth of 5-minute price data it seems to overfit a lot. If I then evaluate the agents and model against a different month's worth of data is performs abysmally. So sounds like classic overfitting.

Note that your evaluation methodology described here indeed no longer fits the more "common" evaluation methodology. You have a problem with concept drift, with nonstationarity in the environment. This means overfitting may be a problem for you.

Still, I'm not sure if dropout would help (it's still additional variance which may hurt). First and foremost, you'd want to make sure that there's some way to keep track of the time / month in your inputs, such that you'll at least have a chance of learning a policy that adapts itself over time. If you have a clear, solid boundary between "training phase" and "evaluation phase", and you know that concept drift occurs across that boundary (you know that your environment behaves differently in the training phase from the evaluation phase)... you really don't have much hope of learning a policy only from experience in the training phase that still performs well in the evaluation phase. I suspect you'll have to get rid of that clear, solid boundary. You'll want to keep learning throughout the evaluation phase as well. This enables your learning algorithm to actually collect experience in the changed environment, and adapt to it.


Do GANs come under supervised learning or unsupervised learning? Do GANs come under supervised learning or unsupervised learning?

My guess is that they come under supervised learning, as we have labeled dataset of images, but I am not sure as there might be other aspects in GANs which might come into play in the determination of the class of algorithms GAN falls under.

The terms Supervised Learning and Unsupervised Learning predate the invention of the application of artificial networks to a generative and discriminative network pair, which was the first popular generative topology.  The existence of labeling is the key distinction between the two.  Even partial labeling indicates supervision, as odd as that jargon is, since the supervisor does no learning and the labels are constants.

Based on the original description of the discriminative network in a GAN, that it consumes examples without labels, GANs are a type of unsupervised learning.  That fact does not eliminate the use of labels as part of an extension of the original design that generates based on some labeled element in the examples or the use of other labels to indicate the fitness of each image for a class into which generated images are expected to fall.


Should I always start from the same start state in reinforcement learning? In an episodic training of an RL agent, should I always start from the same initial state or I can start from several valid initial states?

For example, in a gym environment, should my env.reset() function always resets me to the same start state or it can start from different states at each training episode?

It depends on the task the agent is trying to learn and of course on the environment constrains.

In an Atari game agents have a pre-fixed starting point because that's part of the games rules, so I would say that this is enough of a justification to make each simulation start from that starting point. Moreover, you have to pay attention to the kind of reward function you're using, for example (a really dumb one just to give a grasp of the concept) if you're rewarding the agent depending on how close it gets to the ending point rather than how far it goes from the starting point, the agent might end up having huge rewards only because it respawned close to end point, which would be an artefact and not a fair reward for a good action choice.
Other situations in which it does not really make sense to select a random starting point might be dialogue systems, in which you know that a conversation start with greetings, therefore it would not be logical to make an agent starting with a random question or sentence (the space in these case would be made of different dialogue acts).

There are anyway situations in which the environment allows a random selection of the start location of the agent. In this paper for example, an agent was trained to escape from a maze, and both, the exit point and the initial location of the agent where randomly selected at each iteration. This was partially due to research reasons, the authors were analysing the possibility to train policies more complex than a simple 'space memorisation' one but anyway, conceptually there's nothing wrong in this situation in selecting a random starting point. Other tasks could be following or escaping from a specific object, again, as long as the reward function is properly designed, even with a random starting location at the end the agent would anyway learn to move faster when close to the target (toward or in the opposite direction depending on the task). Actually, in this situation I think that a random initial location would have potential benefits, like avoiding the agents to learn biases due to external biases of the environment (like parts of space with less random obstacles than others).


Which online AI masters degrees do you recommend me? I have a computer science degree and have done AI courses and projects. I want to study AI in a good university, probably online because in my country there are no good universities.

Which  online AI masters degrees do you recommend me? Do you recommend me to try to study a "normal" master's degree instead of an online one?

If you aren't looking to be accredited, here are some good options:
Stanford
Berkeley
University of Toronto
metacademy
If the links are dead, a google search on the relavant course code should yield results.
If you are looking for accreditation, then you might want to try:
Coursera
But note that in my opinion, the class materials from the links above should be of better quality.


How exactly is Monte Carlo counterfactual regret minimization with external sampling implemented? I have read many papers, such as this or this, explaining how external sampling works, but I still don't understand how the algorithm works.
I understand you divide $Q$, which is the set of all terminal histories into subsets $Q_1,..., Q_n$.
What is the probability of reaching some subset $Q_i$? Is it just the product of chance probability, the opponent's probability, and my probability?
As I understand it, the sampling only occurs in the opponent's information sets. How does that work? If there are two players, player 1 strategy is based on what strategy I use.
What happens after you have determined a subset $Q_i$ you want to sample? How many times do you iterate over the subset $Q_i$?
I have searched around and I cannot find any Python code that uses external sampling, but plenty of papers that give formulas, but do not explain the algorithm in detail. So, a Python example of MC-CFR external sampling would probably make it a lot easier for me to understand the algorithm.

External sampling and outcome sampling are two ways of defining the sets $Q_1, \dots, Q_n$. I think your mistake is that you think of the $Q_i$ as fixed and taken as input in these shampling schemes. It is not the case.
In external sampling, there is as many sets $Q_{\tau}$ as there are pure strategies for the opponent and the chance player (a pure strategy is a deterministic policy). Think of it as "the set of terminal nodes that I can reach if my opponent and chance play in this fixed way".
Sampling a set $Q_{\tau}$ thus means sampling a pure strategy for the opponent and chance node. An alternative is to sample on the fly the opponent's policy $\sigma^{-i}$ (and chance) when needed. It gives identical probabilities for q(z), while does not correspond to a full definition of a deterministic policy $\tau$ (which would need a definition on all the information nodes).
There are several implementations on Github, one can be found here: https://github.com/bakanaouji/cpp-cfr/blob/master/RegretMinimization/Trainer/Trainer.cpp


Why is Adam trapped in bad/suspicious local optima after the first few updates? In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write

To further analyze this phenomenon, we visualize the histogram of the absolute value of gradients on a log scale in Figure 2. We observe that, without applying warmup, the gradient distribution is distorted to have a mass center in relatively small values within 10 updates. Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the first few updates.

Here is figure 2 from the paper.

Can someone explain this part?

Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the first few updates.

Why is this true?

The authors describe their belief in Section 3:

Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima.

Diving further, in section 3.2:

Adam uses the exponential moving average to calculate the adaptive learning rate.  For gradients $\{g_1,\dots,g_t\}$, their exponential moving average has a larger variance than their simple average.  Also, in the early stage ($t$ is small), the difference of the exponential weights of $\{g_1,\dots,g_t\}$ is relatively small (up to $1−β^{t−1}_2$).

It seems like the root issue is that exponential moving average, while great with many data samples, have too large of a variance with few data samples. It is this variance that sometimes allow the gradient descend in a bad optima.


Training a neural network to output the conditional probability of an event when the training data output is only binary I have a dataset with hundreds of thousands of training examples. There are 27 input variables and one output variable which is always a 0 or a 1, based on whether an event happened or not.

My network therefore has 27 inputs and 1 output. I want the network's output to be a confidence guess of how likely the event is to happen, for example if the output is 0.23 then that represents that the network thinks the event has a 23% chance of happening.

I am using back propagation to train the neural network. It does appear to work well and the network outputs a higher number when the event is more likely and a lower number when the event is less likely.

Would it be a valid concern that my training data only has 0 or 1 values as outputs, when this is not truly what I want the network to output?

My concern comes from the fact that back propagation attempts to reduce the square of the error between the network's output, and the value of the output in the training data, which is always a 0 or a 1. Because it is the square of the error it is trying to reduce, I'm concerned that it's probability output may not be a linear mapping to the true probability of the event happening based on the 27 inputs it is seeing.

Is this a valid concern? And are there any techniques I can use to get a neural network to output a linear confidence guess between 0 and 1 when my test data only has outputs of 0 or 1?

I am using the sigmoid activation function for all of my neurons, would there be a better choice of activation function for this problem?

Edit: Thanks to Xpector's answer, I now understand that not all back propagation aims to reduce the square of the error, it depends on the loss function used. I am including a part of the back propagation code I have used here which calculates the error:

var neuronOutput = layerOutputs[i];
var error = (neuronOutput - desiredOutput[i]);
errors[i] = error * Maths.SigmoidDerivative(neuronOutput);

This is from an open source RProp implementation. I am not sure what loss function is being used here.

To sort the terms a bit, back propagation (repeatedly applying the chain rule of differentiation) is an implementation of stochastic gradient descent, which is an implementation of optimization. It is not necessarily the square of the error that is reduced by optimization - it is up to us to choose the loss function.

The loss function should ideally mirror the utility function (what you are truly want to optimize), but it needs a useful derivative. For example, the 0-1 loss hasn't one.

I think that what is called linear confidence guess in the question is the conditional probability of the event, given the input. Then the binary cross entropy would be a valid choice of a loss function.

From deeplearningbook.org

  The negative log-likelihood allows the model to estimate the conditional probability of the classes, given the input, and if the model can do that well, then it can pick the classes that yield the least classiﬁcation error in expectation.


What type of AI would you recommend for this complex problem? I'm trying to develop a kind of AI that will assist in debugging a large software system while we run test cases on it. The AI can get access to anything that the developers of the system can, namely log files, and execution data from trace points. It also has access to the structure of the system, and all of the source code.

The end goal of this AI is to be able to detect runtime errors during execution, and locate the source of these errors.

I was considering making use of a deep neural network, where the input would be the execution data and log output. Using this input it would be able to verify whether the current version of the system we are running is functional, or non-functional. The problems with this approach is that the system it would be evaluating would be constantly changing as it gets developed, so the only training material the NN would have is from the last stable version of the system (and even that could have some errors). Additionally, producing test cases for the system off of which we could train the NN would be very time consuming, and would defeat the purpose of using the NN in the first place.

I would like to know what AI design you think would be suitable for this task. Please let me know if you would like any other information relevant to the problem. As far as I can tell, nothing quite like this has been done before.

It's probably worth mentioning that my team has a some extremely powerful machines on which we can run the AI.

You are trying to solve a variant of The Halting Problem, which is the problem of detecting whether a computer program is going to stop, or run forever.

The Halting Problem is incomputable, which means it is not possible to write a computer program that solves it. It is easy to see that your problem is also incomputable. If you could predict whether a program would generate an error, then for any program X that someone wants to solve the halting problem for, we could write a new program:

run(X)
Error("X finished running").

and use your algorithm to determine whether X would finish running or not. Since the halting problem is known to be incomputable, this means your problem must be incomputable too.

That's not to say all is lost though. Formal verification is a field that uses some AI techniques (mostly reasoning-based, but I think there's some machine learning now too) to try to solve this problem for some programs. It can't work for every program though.


Is this overfitting avoidable? I am trying a modification of Mobilenet in which I add feedback from the softmax layer into the early layers (to implement this I put a second net after the first, which receives connections from the softmax layer of the first, the pretrained weights being non trainable). The idea was to mimic the massive feedback projections in the brain, which presumably could help object recognition by enhancing specific filters and inhibiting others.

I took the pretrained network from Keras and started to retrain it on Imagenet. I noticed that the training accuraccy increased right in the first epoch. My computer is very slow thus I cannot train for too long, an epoch takes 3.5 days. So after an epoch I tried the validation set, but instead the accuracy went down to almost half that of the pretrained values.

My question is if this is and obvious case of overfitting. That is, will continued training increase the accuracy of the training set at the expense of the validation set, or is this a normal behavior expected at the initial stages of training, so that if I keep training for a few more epochs I could expect the validation set accuracy to go eventually up? Any ideas that could help are welcomed.

an epoch takes 3.5 days

First of all, use colab to iterate quickly, it offers unlimited 12 hours of free GPU.

  to retrain it on Imagenet.

That said, we could use complex models without being afraid of overfitting due to large training size.

  will continued training increases the accuracy of the training set at the expense of the validation set

In many cases that's the case.

  The idea was to mimic the massive feedback projections in the brain.

My suggestion is to read the third part of the deeplearning book which includes Representation Learning, Structured Probabilistic Models for Deep Learning, Monte Carlo Methods, Confronting the Partition Function, Approximate Inference, Deep Generative Models.

  Part III is the most important for a researcher—someone who wants to understand
  the breadth of perspectives that have been brought to the field of deep
  learning, and push the field forward towards true artificial intelligence.


What is the difference between on and off-policy deterministic actor-critic? In the paper Deterministic Policy Gradient Algorithms, I am really confused about chapter 4.1 and 4.2 which is "On and off-policy Deterministic Actor-Critic".

I don't know what's the difference between two algorithms.

I only noticed that the equation 11 and 16 are different, and the difference is the action part of Q function where is $a_{t+1}$ in equation 11 and $\mu(s_{t+1})$ in equation 16. If that's what really matters, how can I calculate $a_{t+1}$ in equation 11?

The twist here is that the $a_{t+1}$ in (11) and the $\mu(s_{t+1})$ in (16) are the same and actually the $a_t$ in the on-policy case and the $a_t$ in the off-policy case are different.

The key to the understanding is that in on-policy algorithms you have to use actions (and generally speaking trajectories) generated by the policy in the updating steps (to improve the policy itself). This means that in the on-policy case $a_i = \mu(s_{i})$ (in equations 11-13).

Whereas in the off-policy case you can use any trajectory to improve your value/action-value functions, which means that the actions $a_t$ can be generated by any distribution $a_t~\pi(s_t,a_t)$. In (16) the algorithm explicitly states however that the action-value function ($Q^w$) has to be evaluated at $\mu_{s_{t+1}}$ (just like in the on-policy case) and not at $a_t$ which was the actual action in the trajectory generated by policy $\pi$.


How does policy evaluation work for continuous state space model-free approaches? How does policy evaluation work for continuous state space model-free approaches?

Theoretically, a model-based approach for the discrete state and action space can be computed via dynamic programming and solving the Bellman equation.

Let's say you use a DQN to find another policy, how does model-free policy evaluation work then? I am thinking of Monte Carlo simulation, but that would require many many episodes.

How does policy evaluation work for continuous state space model-free approaches? ... Let's say you use a DQN to find another policy, how does model-free policy evaluation work then?

Policy evaluation is the process of determining state-value $v_{\pi}(s)$ or action-value $q_{\pi}(s, a)$ functions for the current policy. In the context of continuous state and action spaces without a model of the environment, policy evaluation must incorporate the agent's past experience instead of the model dynamics and will generally use a function approximator such as a neural network to estimate the action-values. Many popular approaches apply online updates to the function approximator; e.g., DQN combines Temporal-Difference targets and gradient descent to change the weights of the neural network and the resultant action-value estimates. Since

we gradually change the weights of the neural network at each gradient descent step,
the estimated action-values are solely dependent on the weights of the neural network,
the current policy is solely dependent on the estimated action-values (e.g. DQN takes the action with largest action-value),

then policy evaluation (updating the estimated action-value function to better match the true action-value function under the current policy) and policy improvement (greedily changing the current policy based on the new estimated action-value function) occur simultaneously at each gradient descent step. In DQN, a gradient descent step occurs at each time step.

  I am thinking of Monte Carlo simulation, but that would require many many episodes.

After changing the action-value function, we may get a new policy. We assume that the action-value function of the old policy is similar to that of the new policy (although it is not guaranteed) because the change in weights of the neural network was small. Therefore, we use the estimated action-value function of the old policy as an initial estimate of the action-value function of the new policy. Specifically, we use the same neural network with the same weights as the initial approximation. This is computationally convenient, as it prevents the need to start the next policy evaluation update from scratch (e.g. with a Monte Carlo simulation over a painfully large number of episodes).

  Theoretically, a model-based approach for the discrete state and action space can be computed via dynamic programming and solving the Bellman equation.

This same technique of using the estimated action-values of the old policy as the initial estimates for the action-values of the new policy is employed by some Dynamic Programming methods such as value iteration, albeit with known dynamics. Generalized Policy Iteration (GPI) is the notion of letting policy evaluation and policy iteration interact on whatever granularity deemed necessary for the problem at hand. A consequence of adopting the GPI paradigm is the choice to halt policy evaluation before convergence of the action-value function. Many deep reinforcement learning algorithms take this to an extreme and perform policy evaluation and policy improvement simultaneously during a single gradient descent step. For reference, Chapter 4 of Sutton and Barto provides a brief summary of these ideas.


How is the Markovian property consistent in reinforcement learning based scheduling? In Reinforcement Learning, an MDP model incorporates the Markovian property. A lot of scheduling applications in a lot of disciplines use reinforcement learning (mostly deep RL) to learn scheduling decisions. For example, the paper Learning Scheduling Algorithms for Data Processing Clusters, which is from SIGCOMM 2019, uses Reinforcement Learning for scheduling.

Isn't scheduling a non-Markovian process, or am I missing some points?

Lot of real tasks are in reality not markovian, but it doesn't mean you can't try to train an agent on these tasks. It's like saying "we assume variable x to be normally distributed", you just assume that you can condition the probability distributions on the present state of the environment hoping that the agent will learn a good policy. In fact for most applications the challenge is to frame a problem in order to make it as most plausibly markovian as possible (by compressing some important past information into a present state of the environment).

It is pretty common to just give for granted the Markov property, for example in NLP hidden Markov Models are used a lot for sequential tasks like entities detection, this of course leads to well known issues like high error rate on long sentences (the more you look into the future the more the error rate).

Note also that a Markov model can be of first order (probabilities conditioned only on the present state):

$P(W_{t+1} = w | W_{t}, W_{t-1},W_{t-2} ..) = P(W_{t+1} = w | W_{t})$

but they can also be of higher orders (for example second order if conditioning on the present state plus one step in the past):

$P(W_{t+1} = w | W_{t}, W_{t-1},W_{t-n} ..) = P(W_{t+1} = w | W_{t},W_{t-1})$

Of course the more steps in the past you condition the more quickly the problem becomes intractable, that's why almost always first order models are used.

EDIT

I add some comments on the scheduling paper as suggested by nbro.

So, here I would say that the most stryking aspect that makes the process look impossible to describe as an MDP is the presence of dependencies between jobs. Since I might need the result of certain jobs1 before to process  another certain job2, there is surely no way that a time step t could be independent on time step t-1 (I need to know which job I processed in order to know which processes I can process or not).

Here, the trick they use is to learn these dependencies between jobs thanks to a graph network, represented by the DNN in the deep reinforcement learning framework. So, what the agent has to learn is to select a tuple of two actions: "which output (i) a stage designated to be scheduled next, and (ii) an upper limit on the number of executors to use for that stage’s job". The information used to make this selection is the deep representation computed by the graph network on the decency graph of the jobs to schedule. In this sense, since the network is able to represent the 'temporal' relationships between jobs in the present state, this allows to assume that the choice of the next action tuple do not depend on previous states. Hope this is useful and make some sense to you.


Should I call the error "validation error" or "test error" during cross validation? I'm using 10-fold cross validation on all models. Here you can see both plots:

Since I am using k-fold cross validation, is it okay to name it "validation error vs training error" or "test error vs training error" would be better?

The expression "validation error vs training error" is likely more appropriate because the data you use during cross validation that is not the training data is often considered the validation data.

The test data is the data you use to test your model after having performed e.g. cross-validation. The test dataset should be independent of both the training and validation datasets.

This is just a convention, and some people use the terms "validation data" and "test data" interchangeably, because, in a way, during cross-validation you are also "testing" (or, to disambiguate, "validating") the model.

See also this answer: https://stats.stackexchange.com/a/401702/82135.


How to use 'Canny/Watershed' algorithm's output as an input for Image Classification Model I have a very silly problem in hand. I have implemented 2 methods which give me the mask to separate the objects from the background. What I get from one method is the object encapsulated in the red Contour or boundary and the other one makes the background Red.

I am using Keras to classify Trash. I wanted to use the output or the EXTRACTED object as an input to the CNN model. Now I do not see any difference in the images and output. All there is extra boundary around the object and I fail to understand how it can help my model.

I could add an extra alpha channel to second method to make the background transparent but Keras' ImageDataGenerator do not work with RGBA images. What shoul dI do to improve the model?

First and foremost, I have to say that this could (and likely will) be a very hard task. Neural networks (NNs) have excelled at computer vision tasks identifying everything from textures to complex objects but what you are trying to do goes beyond that. We (humans) identify trash using the context as much as the object. An object on a table and the same object in a trash bin can look identical but the context tells us which one is trash. Also, anything can be trash. Trash is not an object, its a state.

Having said all that, it sounds like an interesting project and it would be a very useful model so I'll do my best to explain how you would go about trying this. As per your comment, you have outlined the two steps you need.

Identify and extract the trash object from the image.
Classify the type of waste

This could be accomplished using a single model but I'll explain how to do it with two models to make clear what is being done.

Identify and extract trash

The objective of the first model is to identify and extract the region of interest (the part of the image containing the trash). This is an image segmentation task typically accomplished using an R-CNN - some guides explaining how they work can be found here, here or here. These methods use supervised learning which requires masks delineating the object of interest to be used as the ground truth. A mask is a binary image the same size as your input image where each pixel in the image representing your positive class (trash) is set to 1 and all others are set to 0.

The output of your Canny/Watershed algorithms could be used as the masks to train the model, however your model will only ever be as good as your masks. Therefore, you might as well use your Canny/Watershed algorithms to accomplish task 1. If the masks generated by your algorithm are not of sufficient accuracy you will need to find another way to generate your masks - maybe even doing it manually.

An approximate rule of thumb for object detection with NNs is that you need 1,000 representative images per class, where class implies a specific object. In this case, unless you are attempting to identify very specific items of trash, you would likely need 100s of thousands of images to obtain a high degree of accuracy.

Classify the type of waste

This task should be easier, making the assumption that the segmented image from the first step is always trash. By assigning a label to each segmented image (paper, metal, glass, cardboard, etc) it becomes a normal multi-class classification task which are well documented online with lots of explainations and tutorials.

Single model

These tasks could be combined into a single model by modifying the masks in step 1. Instead of using a single binary mask, you could create a multichannel mask of shape m x n x o where m x n is your input image size and o is the number of kinds of waste you are attempting to identify. Each channel is a binary mask for a given type of waste. Therefore, it becomes a matter of not just identifying and segmenting trash but of segmenting each type of trash separately. Needless to say the complexity of the model would be a lot higher and so would the process to create the masks.


Neural network does not give out the required out put? Made a neural network using tensor flows that was supposed matches an Ip to one of the 7 type of vulnerabilities and gives out what type of vulnerability that IP has.

    model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(50, activation=tf.nn.relu),
  tf.keras.layers.Dense(7, activation=tf.nn.softmax)
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(xs, ys, epochs=500)

The output of print(model.predict([181271844])) when this command is executed should be one of the numbers from 1 to 7 but the out put its gives is

  [[0.22288103 0.20282331 0.36847615 0.11339897 0.04456346 0.02391759
    0.02393949]]

I can't seem to figure out what the problem is.

The numbers you are seeing as output are a probability vector. This is a common output format for multi-class classification models.

In this case, you can interpret the vector as saying:

22% chance of class 1
20% chance of class 2
37% chance of class 3
11% chance of class 4
4% chance of class 5
2% chance of class 6
2% chance of class 7

If you want to get a concrete label out of this, the easiest choice is to compute and return the index of the maximum element.


Reinforcement learning with PPO: rewards decreasing I'm trying to train a PPO agent and my average rewards graph looks like this.  Could this indicate that it's stuck at a local maximum?  Do I need to promote exploring by increasing the entropy or does this look like a bug with my implementation?  Also, my action space is continuous.  Thanks!

Hyperparameters:

Learning rate = 0.01

Entropy Coefficient = 0.01

Value Function Loss Coefficient = 0.5

Gamma/Discount Factor = 0.995

MiniBatch Size = 512

Epochs = 3

Clip Epsilon = 0.1

I had the same problem where the reward kept decreasing and started to search for answers in the forum.

I let the model trained while I search. As the model trained, the reward started to increase. You can see the tensorboard graph for rewards in validation time.

.

The fall continued until around 100k~ steps and did not change a lot for 250k~ steps.
After 350k~ th step, it slowly started to increase. Without knowing your number of steps trained, I would suggest training for more steps.

Also, I read about this (Reward first decreasing and then increasing) in an RL paper, if I find it I will mention it here.


Are Computer Vision and Digital Image Processing part of Artificial Intelligence? There are some fields of Computer Vision that are similar to Artificial Intelligence. For example, pattern recognition and path tracking. Based on these similarities, can we say that the Computer Vision is a part of Artificial Intelligence?

Artifical Intelligence is a rather fuzzily-defined field. It includes lots of subject areas. Sometimes that's because they replicate some behaviour or mental ability of creatures used to solve problems, and sometimes it is because they were the focus of "classic" AI, and have remained classified as part of AI. These problems include navigation, searching, solving combinatorial or logic puzzles, complex systems control, communication. Many of the problems can include converting visual input to something more useful to the rest of the system.

Computer Vision by comparison appears more tightly-defined. However, it quickly becomes less so when you try to consider what the eventual outputs should be. To what extent does a computer vision system need to "understand" the visual inputs in order for other system components to make sense of them, and use them correctly? Here are a couple of examples to illustrate my point:

Image captioning

For instance, work in automated captioning effectively includes natural language models. The input to the system is an image. The computer vision half of that system does not output a class or direct measurement of something inside the image. Instead it outputs a vector encoding of the description of the whole image - something that the language model can decode into an English sentence.

The vision model and language model are trained together, so this encoding can be very loosely considered an internal "ideagram" that represents the contents of the image and allows the vision and language models to share representations.

Agents using vision systems for decision-making

Another interesting example is recent work on Deep Q Networks, used to give superhuman performance when playing some Atari games. This used a neural network architecture (CNN) already known to be good for image classification problems, and used it to directly connect the pixel output from a game to a system which learned optimal behaviour. This is more clearly work in AI - creating a learning agent that works across a general set of problems - but at its core is a CV system.

In this case, the input is four consecutive images (although there are variations which learn from the stream of individual frames), and the output is a vector which scores the value of taking one of the possible controller actions (left, right, up, down, fire etc). Variations on the system directly output which action would be preferred from the CNN.

You could take this further and require that a vision system produce some kind of explainable, transferable, shared world model for other systems to interact with. An embryonic version of that approach is effectively how the captioning system works. It is not at all clear whether such an imagined design is strictly in the field of CV or some other part of AI.

Whether such outputs are considered part of computer vision or more generally part of AI is less of a pure scientific classification issue than an opinionated or political one (e.g. due to funding either an AI research team or a CV one).

The similarities between current computer vision approaches and ongoing work in AI, such as heavy use of deep learning models, would suggest that CV is strongly related to AI. Intuitively I would say that CV is part of AI, perhaps a key specialism of it, roughly equivalent in scope to Natural Language Processing. However, the practical outcome of that is only important to people who have labelled themselves "AI researcher" or "CV engineer" etc.

I also think the same arguments apply to other studies of artificial perception, most notably speech recognition.


What is a "generalized" machine learning algorithm? What does it mean when it is said that Machine Learning algorithm results can be "generalized"?

I don't understand what "generalized" algorithms, routines or functions are.

I have searched dictionaries and glossaries, and cannot find an explanation. Also, if anyone can tell me where a good source for this type of thing is? I am writing about AI and ML.

Actually you have used 2 terminologies there:

The first one is that  Machine Learning algorithm results can be "generalised". This refers to how well your trained Machine Learning model will perform on previously unseen data (test set or implemented on field). This is particularly not easy as data trends may change over time resulting in loss of accuracy. There are various methods to implement this like (having a cross validation set and a test set, which comes under the broad scheme of k-fold cross validation)
The second you mentioned is '"generalised" algorithms, routines or functions'. Most Machine Learning algorithms can be applied to a broad range of problems. For example the training of a NN is generally done by backprop which is universally applied to all NN's. Similarly, you can use CNN to find features of local interest (i.e. local dependencies) in anything that can be represented in a pictorial form (strings of DNA). Also combinations of CNN and RNN are being used to solve many problems. Thus, only a basic generalised algorithm is being applied to a lot of problems. NOTE: I have never seen any one use it in this context, but practically it happens.

Here are a few resources for general reading purposes (not mathematical):

Overfitting and Underfitting With Machine Learning Algorithms
Generalization and Overfitting


Understanding the derivation of the first-order model-agnostic meta-learning According to the authors of this paper, to improve the performance, they decided to

  drop backward pass and using a first-order approximation

I found a blog which discussed how to derive the math but got stuck along the way (please refer to the embedded image below):

Why
 disappeared in the next line.
How come  (which is an Identity matrix)

Update: I also found another math solution for this. To me it looks less intuitive but there's no confusion with the disappearance of  as in the first solution.

$\nabla_{\theta_{i-1}} \theta_{i-1} = \mathbf{I}$ in a similar way that $\frac{d f}{dx} = 1$ for $f(x) = x$. Strictly speaking, $\mathbf{I}$ should be a vector of $1s$ with the same dimensionality as $\theta_{i-1}$, but they are probably abusing notation here and putting such a vector as the diagonal elements of a matrix. Alternatively (actually, the most likely reason!), they are computing the partial derivative of $\theta_{i-1}^j$ with respect to $\theta_{i-1}^k$, for all $k$, for all $j$, which will make up an identity matrix.

Regarding your first question, $\nabla_{\theta} \theta_{0}$ probably becomes 1, but I am not familiar enough with the math of this paper to tell you why. Maybe it's because $\nabla_{\theta} \theta_{0}$ actually means $\nabla_{\theta_0} \theta_{0}$. I would need to dive into it.


Will artificial intelligence cause mass unemployment? Everyone is afraid of losing their job to robots. Will or does artificial intelligence cause mass unemployment?

The nuanced, boring answer is that it depends on your definition of AI. Most people wouldn't say that the rule-based systems designed in the 70's are AI. The amazing leaps in machine learning are almost taken for granted as well (think about how normal speech and facial recognition have become). This is known as the AI effect; when we become accustomed to the technology, it loses it's 'magical aspect' and is thus no longer labelled as AI.

Since AI is so diverse and difficult to define, the question becomes incredibly abstract. Did Siri cause all secretaries to become unemployed? Did TurboTax replace all accountants? Some parts of AI will affect jobs, or even make them redundant yes. On the other hand, it will give rise to new jobs as well. It is therefore impossible to generalize it as 'AI will cause massive unemployment'.

This is not a new phenomenon, however, it has been part of the human economy ever since the industrial revolution (probably even before that, but I am not a historian). The invention of the car crippled the horse-and-wagon industry, but it brought along new jobs as well.


Are there any learning algorithms as powerful as "deep" architectures? This article suggests that deep learning is not designed to produce the universal algorithm and cannot be used to create such a complex systems.

First of all it requires huge amounts of computing power, time and effort to train the algorithm the right way and adding extra layers doesn't really help to solve complex problems which cannot be easily predicted.

Secondly some tasks are extremely difficult or impossible to solve using DNN, like solving a math equations, predicting pseudo-random lists, fluid mechanics, guessing encryption algorithms, or decompiling unknown formats, because there is no simple mapping between input and output.

So I'm asking, are there any alternative learning algorithms as powerful as deep architectures for general purpose problem solving? Which can solve more variety of problems, than "deep" architectures cannot?

Have you read the book The Master Algorithm: by Pedro Domingos?

He discusses the present day machine learning algorithms... Their strengths, weaknesses and applications...

Deep Neural Network
Genetic Algorithm
Bayesian Network
Support Vector Machine
Inverse Deduction


Recognition of small objects I'm currently implementing an Android app for street sign recognition. My solution works quite well for the GTSRB dataset, since it provides a labeled test set of centered images. However, it doesn't scale up to more realistic scenarios like for images in the GTSDB, where the signs only take up some pixels. Is it still recommended to downsample the image to 224x224?

There is a big difference between the 2 datasets. When you train on the GTSRB dataset, your model performs image recognition, while for the GTSDB dataset you need to perform object detection. These are 2 related but different tasks (with detection being considerably harder).

You can consider using a sliding window approach, or use some prior attention model that will learn to extract the area of the signs for later classification.

I don't think that it is recommended to down sample images of 1360x800 pixels to 224x224.
(224x224)/(1360x800)=0.046. You are loosing 96% of the information (including your street signs which might reduced to nothing), granted most of it is not interesting, but simple down-sampling doesn't get to choose what to forfeit.


What does "class-level discriminative feature representation" mean in the paper "Semi-Supervised Deep Learning with Memory"? I am reading the paper Semi-Supervised Deep Learning with Memory (2018) by Yanbei Chen et al.  The topic is the classification of images using semi-supervised learning. The authors use a term on page 2 in the middle of the page that I am not familiar with. They write:

The key to our framework design is two-aspect: (1) the class-level discriminative feature representation and the network inference uncertainty are gradually accumulated in an external memory module; (2) this memorised information is utilised to assimilate the newly incoming image samples on-the-fly and generate an informative unsupervised memory loss to guide the network learning jointly with the supervised classification loss

I am not sure what the term discriminative feature representation means.
I know that a discriminative model determines the decision boundary between the classes, and examples include: Logistic Regression (LR), Support Vector Machine (SVM), conditional random fields (CRFs) and others.
Moreover, I know that, in machine learning, feature learning or representation learning is a set of techniques that allows a system to automatically discover the representations needed for feature detection or classification from raw data.
Any insights on the definition of this term much appreciated.

Discriminative models give the probability of an element in the feature space $x \in X$ belonging to a class $y \in Y$, i.e. $p(Y|X)$, where $Y$ is the set of classes in a classification problem.
The discriminative feature representation in this context means the feature map/s  which are the output/s of the convolutional layers of the backbone (the convolutional layers of the neural network) which (presumably) differ on the basis of their class, and can be used to discriminate which class the original image belongs to, in the case of image classification.


Q-values represent expected return after taking action $a$ in state $s$, so they do tell you how good it is to take an action in the specific state. Better actions will have larger Q-values. Q-values can be used to compares actions but they are not very meaningful in representing performance of the agent since you have nothing to compare them with. You don't know the actual Q-values so you can't conclude if your agent is approximating well those Q-values or not.

Better performance metric would be the average reward per episode/epoch, or average reward in last $N$ timesteps for continuing tasks. If your agent is improving its performance then it's average reward should be increasing. You said that you have rewards per state and that some of them represent more important criteria then others. You could plot the average reward per episode by doing some kind of weighted linear combination of criteria rewards
\begin{equation}
\bar R = \bar R_1 w_1 + \bar R_2 w_2 + \bar R_3 w_3 + \bar R_4 w_4
\end{equation}
where $\bar R_i$ is the average episode reward for criteria $i$.That way you can provide more importance to some specific criteria in your evaluation.


Do all expert trajectories have the same starting state in apprenticeship learning? In the apprenticeship learning algorithm described by Ng et al. in Apprenticeship Learning via Inverse Reinforcement Learning, they mention that expert trajectories come in the form of $\{s_0^i, s_1^i\, ...\}_{i=1}^m$. However, they also mentioned that $s_0 $ is drawn from distribution D. Do all expert trajectories then have to have the same starting state? Why is it not possible to compute the feature expectation based on a single trajectory?

All right, I figured it out. trajectories need not have the same starting state because the distribution of $s_0$ is drawn from a distribution D (mentioned in the paper). Had been confused because many of the code implementations on github focus on trajectories starting from the same state.

Hope this helps everyone !


Transformers - is the self attention matrix softmax output (layer 1) symmetric? Let's assume,  that we embedded a vector of length 49 into a matrix using 512-d embeddings. If we then multiply the matrix by it transposed version we receive a matrix of 49 by 49. Which is symmetric. Let's also assume we do not add the positional encoding and we only have only one attention head in the first layer of the transformer architecture. What would the result of the softmax on this 49 by 49 matrix look like? Is it still symmetric, or is the softmax correctly applied for each line of the matrix, resulting in a non symmetric matrix? My guess would be that the matrix should not be symmetric any more. But I'm unsure about that.
I ask this, to verify if my implementation is wrong or not, and what the output should look like. I have seen so many sophisticated and different implementation of the transformer architecture with different frameworks, that I can't answer this question for myself right now (confusion). I still try to understand the basic building blocks of the transformer architecture.

I compared my results visually to a second implementation known to be working - "The annotated transformer".  I compared the pytorch calculation results of the attention-method to my implementation results.
The answe is - the softmax is applied row by row. Therefore the resulting matrix p-attn is not equal to its transposed version.


How does backpropagation work on a custom loss function whose components have magnitudes of different orders? I want to use a custom loss function which is a weighted combination of l1 and DSSIM losses. The DSSIM loss is limited between 0 and 0.5 where as the l1 loss can be orders of magnitude greater and is so in my case. How does backpropagation work in this case? For a small change in weights, the change of the l1 component would obviously always be far greater than the SSIM component. So, it seems that only l1 part will affect the learning and the SSIM part would almost have no role to play. Is this correct? Or I am missing something here. I think I am, because in the DSSIM implementation of Keras-contrib, it is mentioned that we should add a regularization term like a l2 loss in addition to DSSIM (https://github.com/keras-team/keras-contrib/blob/master/keras_contrib/losses/dssim.py); but I am unable to understand how it would work and how the SSIM would affect the backpropagation being totally overshadowed by the large magnitude of the other component. It will be a great help if someone can explain this. Thanks.

Don't know if you have this doubt anymore, but this would be helpful for those who are facing similar problems-
You will need to find the correct weights with which you add these two loses by hyperparameter search. That is, find the best $\lambda$ for the loss-
$$
L = Loss_1 + \lambda(Loss_2)
$$
Here $Loss_1$ and $Loss_2$ can be any losses. Here, we take them as SSIM and L1-regularization losses, respectively.
You can keep the gradient flow of the regularization loss below some percentage of that of $Loss_1$ by choosing the correct value of gradient clipping for a hyperparameter. Note that by setting this hyperparameter to low, you may even impede its performance(the exact opposite of the case mentioned here). For this specific case, L1 regularization has a constant gradient, equal to the hyperparameter itself. So, by keeping it around 10% of the max gradient(or 10% of the max loss generally works as well), we should not face these kinds of problem.


How is gradient being calculated in Andrej Karpathy's pong code? I was going through the code by Andrej Karpathy on reinforcement learning using a policy gradient. I have some questions from the code.

Where is the logarithm of the probability being calculated? Nowhere in the code I see him calculating that.
Please explain to me the use of dlogps.append(y - aprob) line. I know this is calculating the loss, but how is this helping in a reinforcement learning environment, where we don't have the correct labels?
How is policy_backward() working? How are the weights changing to the loss function mentioned above? More specifically, what's dh here?

logp seen in code is actually logit p which has this story behind:

Given a probability p, the corresponding odds are calculated as p / (1 – p). For example if p=0.75, the odds are 3 to 1: 0.75/0.25 = 3.
The logit function is simply the logarithm of the odds: logit(x) = log(x / (1 – x)).

Sigmoid near logp is like follows:

The inverse of the logit function is the sigmoid function. That is, if you have a probability p, sigmoid(logit(p)) = p.

Source: [1]

In reinforcement learning we know in the end of game, if taken actions were successful or not. Then before next round we can adjust the gradients. From your link (commentary section):

For example in Pong we could wait until the end of the game, then take the reward we get (either +1 if we won or -1 if we lost), and enter that scalar as the gradient for the action we have taken (DOWN in this case). In the example below, going DOWN ended up to us losing the game (-1 reward). So if we fill in -1 for log probability of DOWN and do backprop we will find a gradient that discourages the network to take the DOWN action for that input in the future (and rightly so, since taking that action led to us losing the game).

In the very same commentary section (later) there is a pic, and explanation of what h is. Unfortunately, you have to check it by yourself, pic was not compatible to be attached here. By describing the pic I could say h is equal to weights in hidden layer and in gradient case the dh is the derivative of h.

Roughly speaking the backpropagation is correcting the weights backwards the network after the round is done. More thorough explanation is in the mentioned comments section.
Sources:
[1] https://www.google.com/amp/s/nathanbrixius.wordpress.com/2016/06/04/functions-i-have-known-logit-and-sigmoid/amp/


Sentiment analysis does not handle neturals I'm writing some financial tools,  I've found highly performant models for question and answering but when it comes to sentiment analysis I haven't found anything that good.  I'm trying to use huggingface:
from transformers import pipeline
classifier = pipeline('sentiment-analysis')
print(classifier("i'm good"))
print(classifier("i'm bad"))
print(classifier("i'm neutral"))
print(classifier("i'm okay"))
print(classifier("i'm indifferent"))

Which returns results

[{'label': 'POSITIVE', 'score': 0.999841034412384}]

[{'label': 'NEGATIVE', 'score': 0.9997877478599548}]

[{'label': 'NEGATIVE', 'score': 0.999396026134491}]

[{'label': 'POSITIVE', 'score': 0.9998164772987366}]

[{'label': 'NEGATIVE', 'score': 0.9997762441635132}]

The scores for all of the neutral words come up very high in a positive or negative direction,  I would of figured the model would put the score lower.
I've looked at some of the more fine-tuned models yet they seem to perform the same.
I would assume there would be some pretrained models which could handle these use cases.  If not, How can I find neutral sentiments?

Yes, there is. You can try Spacy. Here you go.
import spacy
from spacytextblob.spacytextblob import SpacyTextBlob

nlp = spacy.load('en_core_web_sm')
spacy_text_blob = SpacyTextBlob()
nlp.add_pipe(spacy_text_blob)

text = "i'm good"
doc = nlp(text)
print(doc._.sentiment.polarity) # 0.7

text = "i'm bad"
doc = nlp(text)
print(doc._.sentiment.polarity) # -0.6999999999999998

text = "i'm neutral"
doc = nlp(text)
print(doc._.sentiment.polarity) # 0.0

text = "i'm okay"
doc = nlp(text)
print(doc._.sentiment.polarity) # 0.5

text = "i'm indifferent"
doc = nlp(text)
print(doc._.sentiment.polarity) # 0.0


Why some neural network models in the 1980s shown as circuit models I am familiar with the currently popular neural network models that have weights and are trained with backpropagation and gradient descent.
However, I came across a different type of neural network popular in the 1980s and 1990s. The Hopfield network is one of the most classic. These neural network models different in the following ways:

They do not have parameter weights and bias to train or to learn from data.
They used a circuit diagram to present the model.
The model can be simplified as an ODE system and has a Lyapunov function as objective.

Please take a look at these two papers in the 1980s:

Neurons with graded response have computational properties like those of two state neurons (J.J. Hopfield)

Neural Networks for Non-linear Programming (M.P Kennedy & L.O Chua)

In the early days of neural networks the theorists and practitioners were educated in mathematics, psychology, neurophysiology, electrical engineering, and neurobiology. Computer science was still in its infancy. The first neural networks were modeled as electrical circuits.
There is evidence of this in the 1943 paper by Warren McCulloch and Walter Pitts [1], and a 1956 paper by Rochester et al. [2].
The latter paper uses terms such as 'circuits' and 'switching'.  One idea in the paper is explained in terms of a "Eccles-Jordan Flip Flop circuit" although there are no drawings.  Nathanial Rochester had designed the IBM 701k [3] and "led the first effort to simulate a neural network" [4]
Brain structure was discussed in terms of 'neural circuits' as early as 1937 [5].
I am not sure when the first electrical circuit diagram appeared in publication, but it makes sense that early neural network designers, would have thought of their implementation as such.
References:

[1] McCulloch, W. S., and Walter Pitts, W. (1943) A Logical Calculus of the Ideas Immanent in Nervous Activity. https://jontalle.web.engr.illinois.edu/uploads/498-NS.21/McCulloch-Pitts-1943-neural-networks-ocr.pdf
[2] Rochester, N., Holland, J., Haibt, L., & Duda, W. (1956). Tests on a cell assembly theory of the action of the brain, using a large digital computer. IRE Transactions on Information Theory, Information Theory, IRE Transactions on, IRE Trans. Inf. Theory, 2(3), 80–93. https://doi-org/10.1109/TIT.1956.1056810
[3] Brief History of Neural Networks
https://www.linkedin.com/pulse/data-science-term-neural-networks-70-year-old-anchit-sharma/?trk=public_profile_article_view
[4] This Data Science term - Neural Networks is 70-Year-Old Technology!
https://medium.com/analytics-vidhya/brief-history-of-neural-networks-44c2bf72eec:
[5] Papez circuit, https://en.wikipedia.org/wiki/Papez_circuit


How to deal with random weights initialization in hyperparameters tuning? In the step of tuning my neural networks I often encounter a problem that every time I train the exact same network, it gives me different final error due to random initialization of the weights. Sometimes the differences are small and negligible, sometimes they are significant, depending on the data and architecture.

My problem arises when I want to tune some parameters like number of layers or neurons, because I don't know if the change in final error was caused by recent changes in network's architecture or it is simply effect of the aforementioned randomness.

My question is how to deal with this issue?

I don't think you can.

Say a NN with 3 layers gives an accuracy of 95.3% and another NN with 4 layers gives an accuracy of 95.4%. Then there is no guarantee that the 4 layer NN is better than the 3 layered NN. Since with different initial values the 3 layer NN might perform better.

You could run multiple times and probabilistically say that this is better, but this is computational intense.


Why L2 loss is more commonly used in Neural Networks than other loss functions? Why L2 loss is more commonly used in Neural Networks than other loss functions?
What is the reason to L2 being a default choice in Neural Networks?

I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE):
MSE:

L2 loss is continuously-differentiable across any domain, unlike L1 loss.  This makes training more stable and allows for gradient-based optimization, as opposed to combinatorial optimization.
Using L2 loss (without any regularization) corresponds to the Ordinary Least Squares Estimator, which, if you're able to invoke Gauss-Markov assumptions, can lead to some beneficial theoretical guarantees about your estimator/model (e.g. that it is the "Best Linear Unbiased Estimator").  Source: https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem.

L2 Regularization:

Using L2 regularization is equivalent to invoking a Gaussian prior (see https://stats.stackexchange.com/questions/163388/why-is-the-l2-regularization-equivalent-to-gaussian-prior) on your model/estimator.  If modeling your problem as a Maximum A Posteriori Inference (MAP) problem, if your likelihood model (p(y|x)) is Gaussian, then your posterior distribution over parameters (p(x|y)) will also be Gaussian.  From Wikipedia: "If the likelihood function is Gaussian, choosing a Gaussian prior over the mean will ensure that the posterior distribution is also Gaussian" (source: https://en.wikipedia.org/wiki/Conjugate_prior).

As in the case above, L2 loss is continuously-differentiable across any domain, unlike L1 loss.


Counting people in an image of a crowd What are some good approaches that I can use to count the no. of people in a crowd. Tracking each person individually is obviously not an option. Any good approaches or some references to research papers would be very helpful.

Here a convolutional neural network (cNN) based approach is presented:
Image Crowd Counting Using Convolutional Neural Network and Markov Random Field

This blog post has more of a tutorial character and also present an approach based on CNNs
Counting Crowds and Lines


Why is description logic decidable but first order logic is not decidable? Description logic is a fragment of first order logic, but description logic is decidable and first order logic not decidable. Why is that? what is the role of variables in first order logic to make it undecidable?

This has to do with the fact that you can define arithmetic inside the axiomatic system or not. In description logic you cannot speak about arithmetic sentences and in first order logic you do.

if you look at the proof of incompleteness you will understand this in depth. This demonstration depends on an arithmetic coding of statements, and this representation is fundamental to obtain the conclusion.


If two perfect chess AI's played each other, would it always be a stalemate or would white win for an inherent first-move advantage? In the circumstances of two perfect AI's playing each other, will white have an inherent advantage? Or can black always play for a stalemate by countering every white strategy?

This relates to the concept of "solved games". In general, two player turn-based games with perfect information - of which chess is an example - can result in all three possible outcomes: a forced win for white, a forced win for black, or a forced draw.

The short, although unsatisfactory answer is that chess is not solved, and it is not clear whether it can be. There is generally thought to be an advantage to white for the first move, so likely results are considered to be a forced win for white, or a forced draw.

No current AI attempts to "solve" chess, although some of the techniques such as MCTS might be adapted theoretically to find a solution, the available computing power to run that search to completion from the start positions is too low by a few orders of magnitude.


What should the input and output of the Q-network be in the case of an ordinal action space? I recently started looking into implementations of the DQN algorithm (e.g. TensorFlow) in some more detail. All the implementations that I found use a network that gives an output for each possible action (e.g. if you have three possible actions you will have three output units in your network). This makes a lot of sense from a computational standpoint and seems to work fine if you are dealing with categorical action spaces (e.g "left" or "right").
However, I am currently working with an action space that I discretized and the actions have an ordinal meaning (e.g. you can drive left or right in 5-degree increments). I assume that the action-value function has some monotonicity in the action component (think driving 45 degrees to the left instead of 40 will have a similar value).
Am I losing information on the similarity of actions, if I use a network that has an output unit for each possible action?
Are there implementations of the DQN available in which actions are used as network inputs?

Yes it is possible to use the action as input to neural network in DQN. For discrete actions represented as one-hot encoded features, the difference is minor:

If all actions are in the output, your neural network function is $f(s): \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|} = [\hat{q}(s,a_1), \hat{q}(s,a_2), \hat{q}(s,a_3) ...]$, and you take the maximum value from the output vector as the greedy action.

If the action is provided as an input argument, your neural network function is $f(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \hat{q}(s,a)$, and to find the maximum value you construct and run a mini-batch over all possible values of $a$ for the given state.

Also see the answer to this question: Why does Deep Q Network outputs multiple Q values?
In your case, you would like to take advantage of similar values of $a$ because you expect that to work well with the approximation. As you correctly suggested, this will only work with the second approach using action as an input. So, use the steering angle, normalised into a suitable range for input to a neural network, as an input. Every time that you need to find $\text{max}_a Q(s,a)$ for the Q-learning algorithm, you must construct a mini-batch of the current state concatenated with each of the discrete steering angles that you want to consider as actions in the DQN, and run the current (or target) neural network forward.
If you want to go further and use a continuous action space, you will need to change which reinforcement learning method you are using. The various policy gradient and actor-critic approaches, such as REINFORCE, A3C, DDPG etc can cope with continuous actions, because they drop the need to find $\text{max}_a Q(s,a)$, which becomes impractical for very large action spaces.


Is the final model scaling done on the full training set? We have our training set and our test set. When we scale our data we "fit" the scaler transform to the training set and then we scale both the training set and test set using this scaler object. Using splitting and cross-validation techniques, one can use the training set as training and validation. Finally, reporting on the test set.
Now, if I want to use a model in a real-life environment, it's common to use the entire dataset (training and test) to train our already optimized model to obtain a final ready for the production model.
My question is regarding scaling. Should we fit the scaler to the entire set and then scale? Or can we simply append the scaled training set and scaled test set (both have been scaled using the training set's scaling parameters)?
I am making use of sklearn.preprocessing.PowerTransformer. Using "Yeo-Johnson's" power transform and also standardizing the data.

The short answer is yes.
When you merge the test set into the train set, you try to squeeze available data till the last drop. The cons and pros of this approach have been considered in other questions in the network 1, 2. But if you decided to go for it, there is no point to not use the whole dataset for the scaling transformation, as the trend "more data leads (generally) to better models" is valid to the scaling to the same extent it's valid for the model itself.


Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does? I was looking at the Bellman equation, and I noticed a difference between the equations used in policy evaluation and value iteration.
In policy evaluation, there was the presence of $\pi(a \mid s)$, which indicates the probability of choosing action $a$ given $s$, under policy $\pi$. But this probability seemed to be omitted in the value iteration formula. What might be the reason? Maybe an omission?

You appear to comparing the value table update steps in policy iteration and value iteration, which are both derived from Bellman equations.
Policy iteration
In policy iteration, a policy lookup table is generated, which can be arbitrary. It usually maps a deterministic policy $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$, but can also be of the form $\pi(a|s): \mathcal{A} \times  \mathcal{S} \rightarrow \mathbb{R} = Pr\{A_t = a |S_t =s\}$. Policy iteration then alternately evaluates then improves that policy, with the improvement always being to act greedily with respect to expected return. Because the policy function can be arbitrary, and also the current value estimates during evaluation might not relate to it directly, the function $\pi(s)$ or $\pi(a|s)$ needs to be shown.
Typically with policy iteration, you will see this update rule:
$$V(s) \leftarrow \sum_{r,s'} p(r,s'|s,\pi(s))(r + \gamma V(s'))$$
The above rule is for evaluating a deterministic policy, and is probably more commonly used. There is no real benefit in policy iteration to working with stochastic policies.
For completeness, the update rule for an arbitrary stochastic policy is:
$$V(s) \leftarrow \sum_a \pi(a|s) \sum_{r,s'} p(r,s'|s,a)(r + \gamma V(s'))$$
Value iteration
In value iteration, the current policy to evaluate is to always take the greedy action with respect to the current evaluations. As such, it does not need to be explicity written, because it can be derived from the value function, and so can the terms in the Bellman equation (specifically the Bellman equation for the optimal value function is used here, which usually does not refer the policy). What you would typically write for the update step is:
$$V(s) \leftarrow \text{max}_a \sum_{r,s'} p(r,s'|s,a)(r + \gamma V(s'))$$
However, you can write this out as if there was a policy table:
$$\pi(s) \leftarrow \text{argmax}_a \sum_{r,s'} p(r,s'|s,a)(r + \gamma V(s'))$$
$$a \leftarrow \pi(s)$$
$$V(s) \leftarrow \sum_{r,s'} p(r,s'|s,a)(r + \gamma V(s'))$$
This is not the usual way to implement it though, because of the extra maximum value search required to identify the action. In simple value iteration it does not matter what the interim action choices and policies actually are, and you can always derive them from the value function if you want to know.
Other value-based methods
You will find other algorithms that drive the current policy direct from a value function, and when they are described in pseudo-code they might not have an explicit policy function. It is still there, only the Bellman update is easily calculated directly from the value function, so the policy is not shown in the update step. Descriptions of SARSA and Q-learning are often like that.


Are there any load balancing approaches for employee cases based on complexity score? In working with a social services agency that provides a continuum of programs across the behavioral health and child welfare spectrum the need to adequately manage individual worker and total program caseloads has become increasingly difficult due to burgeoning demand.

Each program has already developed a profile for ranking the complexity of a client based a select number of factors (i.e. diagnosis, previous encounters, results of assessment instruments, etc.).

What we are trying to develop now is an approach to act as a load balancer for new cases to ensure that individual workers' caseloads are balanced with the complexity of cases. We can do this manually to some degree but if there was a way to automate and take the human bias out of it, that is the goal.

Are there are any algorithms or adaptable approaches that have been used to do such a task (even in other sectors)?

It sounds like you're looking at the Partition Problem. https://en.wikipedia.org/wiki/Partition_problem

The task of slicing one set into N sets so that each set is equal or as close to equal as possible.

Obtaining an exact solution is NP-hard (you can't do much better than trying all combinations), however you can get an approximate answer in polynomial time.

Greedy approach:

Create N sets, each initially empty
Sort all items in descending order (by their complexity score)
for each item add it to the set with the lowest sum

If N is large, you may want to put the sets into a min heap/priority queue.


Which evaluation methods can I use for image segmentation? I implemented an image segmentation pipeline and I trained it on the DICOM dataset. I compared the results of the model with manual segmentation to find the accuracy. Is there other methods for evaluation?

See:

  Martin Thoma: A Survey of Semantic Segmentation, Section III

Subsection A is about metrics and B is about datasets.

Metrics include: accuracy, IoU, frequency weighted IoU, F-beta score, speed, ...


Formula for expected rewards for state–action–next-state triples as a three-argument function While reading about reinforcement learning, I have come across the following expression for expected rewards in terms of a summation, the denominator of which I am not able to account for.

The formula given is:

According to what I understand, the formula should have been correct without the denominator (that I have highlighted). How is this formula correct?

Source of image: Andrew G and Sutton's book on RL.

Expectation of reward after taking action $a$ in state $s$ and ending up in state $s'$ would simply be
\begin{equation}
r(s, a, s') = \sum_{r \in R} r \cdot p(r|s, a, s')
\end{equation}
The problem with this is that they do not define probability distribution for rewards separately, they use joint distribution $p(s', r|s, a)$ which represents probability for ending up in state $s'$ with reward $r$ after taking action $a$ in state $s$. This probability can be separated in 2 parts using product rule
\begin{equation}
p(s', r|s, a) = p(s'|s, a)\cdot p(r|s', s, a)
\end{equation}
which represent probability for getting to state $s'$ from $(s, a)$ and then probability for getting reward $r$ after ending up in $s'$.
If we define reward expectation through joint distribution we would have
\begin{align}
r(s, a, s') &= \sum_{r \in R} r \cdot p(s', r|s, a)\\
&= \sum_{r \in R} r \cdot p(s'|s, a) \cdot p(r|s', s, a)
\end{align}
but this would not be correct since we have this extra $p(s'|s, a)$ so we divide everything by it to get expression with only $p(r|s', s, a)$. So, in the end we have

\begin{equation}
r(s, a, s') = \sum_{r \in R} r \frac{p(r, s'|s, a)}{p(s'|s, a)}
\end{equation}


How much the dialects recognition and speech recognition are relevant? In this tutorial, they build a speech recognition model to classify a one-second audio clip as one of ten predefined words. Suppose that we modified this problem as the following: Given an Arabic dataset, we aim to build a dialects recognition model to classify a two-second audio clip as one of $n$ local dialects using ten predefined sentences. I.e. for each of these ten sentences, there are $x$ different phrases and idioms which refer to the same meaning$^*$. Now how can I take advantage of the mentioned tutorial to solve the modified problem?

$*$ The $x$ different phrases and idioms for each sentence are not predefined.

The tutorials you link are not much relevant, there are already existing implementations of your exact problem.

You can use https://github.com/swshon/dialectID_e2e, there are many other similar implementations on github.


Why are the value functions sometimes written with capital letters and other times with lower-case letters? Why are the state-value and action-value functions are sometimes written in small letters and other times in capitals? For instance, why in the Q-learning algorithm (page 131 of Barto and Sutton's book but not only), we the capitals are used $Q(S, A)$, while the Bellman equation it is $q(s,a)$?

In the Sutton and Barto book $q(s,a)$ is used to denote the true expected value of taking action $a$ in state $s$, whereas capital $Q(s,a)$ is used to denote an estimate of $q(s,a)$. However, there is likely to be a lot of inconsistency in the literature as each author has their own preference on how to denote things. I would encourage you to consider whether the value you are reading is to denote an estimate or the true value.


What are the available exploration strategies for continuous action space scenarios in RL? I'm building a deep neural network to serve as the policy estimator in an actor-critic reinforcement learning algorithm for a continuing (not episodic) case.  I'm trying to determine how to explore the action space. I have read through this text book by Sutton, and, in section 13.7, he gives one way to explore a continuous action space.  In essence, you train the policy model to give a mean and standard deviation as an output, so you can sample a value from that Gaussian distribution to pick an action.  This just seems like the continuous action-space equivalent of an $\epsilon$-greedy policy.
Are there other continuous action space exploration strategies I should consider?
I've been doing some research online and found some articles related to RL in robotics and found that the PoWER and PI^2 algorithms do something similar to what is in the textbook.
Are these, or other, algorithms "better" (obviously depends on the problem being solved) alternatives to what is listed in the textbook for continuous action-space problems?
I know that this question could have many answers, but I'm just looking for a reasonably short list of options that people have used in real applications that work.

I have not personally worked enough with continuous action spaces to be capable of confidently giving advise based on my own experience, but I can point you to likely relevant research (more recent than the research you already pointed to yourself):

The most common / "popular" area of research in recent years that involves RL and continuous action spaces uses robot / physics simulators such as MuJoCo. Some examples:

Asynchronous Methods for Deep Reinforcement Learning mentions using an entropy cost term in the loss function used for training to encourage exploration (see Supplementary Material after References).
Parameter Space Noise for Exploration describes the idea of adding noise directly to the learned parameters on the Neural Network. That way it basically always predicts a different action to be "optimal" (due to the noise), and therefore a "greedy" policy based on the noisy parameters will in fact have exploration and not be fully greedy.
DeepMind Control Suite is a somewhat recent paper that proposes a suite of benchmarks for continuous control problems. It is filled with references to relevant papers describing all kinds of algorithms, and hopefully every single one of those papers would also describe how they perform exploration.

Recently, at the ICML 2018 conference, there was a complete workshop dedicated to Exploration in Reinforcement Learning. Here is the list of accepted papers at this workshop. Note that it's about Exploration in RL, not only about continuous action spaces, so there might be papers in there that are only applicable to discrete action spaces. Nevertheless, I would be highly surprised if there's nothing in there that's relevant.


Are there any easy ways to create annotated training images for object detection? For the purposes of object detection, are there any easy ways to create annotated training images? For example, if we have $10,000$ images and want to draw bounding boxes on 2 objects for each image, do we have to physically draw those boxes? Is that what most people do these days to create training data?

From what I read in the papers and my experience, the trend lately is to generate synthetic data using a photorealistic 3D graphics engine for automating the image labelling. The process would be:

Get a visually realistic 3D engine such as Unreal, Unity... any videogame engine would work
Build 3D models of the objects you want to detect (or ask for help to a 3D artist)
Build a 3D environment for placing the objects 3D models
Generate random instances of the objects in the environments with different environmental conditions: lighting, brightness, position, rotation...

Since you have the 3D model of the object and the whole control in the 3D engine you can annotate images, bounding boxes, pixels... or whatever stuff you want in an automated way.

This is an approach for automatic image labeling with synthetic data. Automating the labeling of real data is a challenge yet to be tackled.