How to make input variable as trainable parameter in a neural network? I am working on an optimization problem. First, I have done forward training to work the network as a surrogate model, then I freeze the output and I want to find an optimal value of input for a given output.

It's just a typical optimization problem. You want to optimize function
\begin{equation}
f(\theta, x)
\end{equation}
with fixed parameters $\theta$ (network weights) and optimization variable $x$. Typically the approach is consisted of choosing a search direction and direction step. For deciding search direction you can use the gradient of objective with respect to decision variable and have an update of form
\begin{equation}
x = x - \alpha \nabla_x f(\theta, x)
\end{equation}
where $\nabla_x f(\theta, x)$ is the gradient. You can also use Newton or Quasi-Newton methods of form
\begin{equation}
x = x - \alpha B^{-1}\nabla_x f(\theta, x)
\end{equation}
where $B$ is a Hessian or approximate Hessian of $f(\theta, x)$ with respect to $x$. Step length parameter is usually decided with line search or you can also use trust region approaches which decide step direction with constraints to it's length. For more details about optimization you can consult Numerical Optimization by Nocedal and Wright


Can ELMO embeddings be used to find the n most similar sentences? Assume I have a list of sentences, which is just a list of strings. I need a way of comparing some input string against those sentences to find the most similar. Can ELMO embeddings be used to train a model that can give you the $n$ most similar sentences to an input string?

For reference, gensim provides a doc2vec model that can be trained on a list of strings, then you can use the trained model to infer a vector from some input string. That inferred vector can then be used to find the $n$ most similar vectors.

Could something similar be done, but using ELMO embedding instead?

Any guidance would be greatly appreciated.

I ended up finding this article which does what I'm looking for.
Below is the portion of code I adapted for my needs

from sklearn.metrics.pairwise import cosine_similarity

import tensorflow_hub as hub
import tensorflow as tf

elmo = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True)

def elmo_vectors(x):
  embeddings=elmo(x, signature="default", as_dict=True)["elmo"]

  with tf.device('/device:GPU:0'):
    with tf.Session() as sess:
      sess.run(tf.global_variables_initializer())
      sess.run(tf.tables_initializer())
      # return average of ELMo features
      return sess.run(tf.reduce_mean(embeddings,1))

corpus=["I'd like an apple juice",
        "An apple a day keeps the doctor away",
         "Eat apple every day",
         "We buy apples every week",
         "We use machine learning for text classification",
         "Text classification is subfield of machine learning"]

elmo_embeddings=[]
print (len(corpus))
for i in range(len(corpus)):
    print (corpus[i])
    elmo_embeddings.append(elmo_vectors([corpus[i]])[0])

print ( elmo_embeddings, len(elmo_embeddings))
print(elmo_embeddings[0].shape)
sims = cosine_similarity(elmo_embeddings, elmo_embeddings)
print(sims)
print(sims.shape)


What are the most compact Real Time-Strategy Games? There was a recent informal question on chat about RTS games suitable for AI benchmarks, and I thought it would be useful to ask a question about them in relation to AI research.

Compact is defined as the fewest mechanics, elements, and smallest gameboard that produces a balanced, intractable, strategic game. (This is important because greater compactness facilitates mathematical analysis.)

This is an interesting question. One good place to start would be the gargantuan catalogue of rather simple RTS flash games and the like. There are many sites that have 100s of such titles, and it would not be too difficult to build the framework for these types of games(there are open source tools that can help as well).

As far as "real" mainstream games, the pickings are rather slim for simple benchmark RTS titles. One that comes to mind is Rome: Total War. At least in exhibition, there is not much to strategize over besides who to attack and from where. Also, as I assume this relates to starcraft, one good simplified predecessor to starcraft would be the Dune series, and in particular Dune 2.


What is the difference between abstract, autonomous and virtual intelligent agents? On Wikipedia, we can read about different type of intelligent agents:

abstract intelligent agents (AIA),
autonomous intelligent agents,
virtual intelligent agent (IVA), which I've found on other websites, e.g. this one.

What are the differences between these three to avoid confusion?

For example I've used term virtual artificial agent here as:

  Basically a robot is a mechanical or virtual artificial agent which exhibit intelligent behavior (AI).

so basically I'd like to know where other terms like autonomous or abstract agents can be used and in what context. Can they be all defined under 'virtual' robot definition? How to distinguish these terms?

I don't think there are many contexts where there is any really meaningful distinction between these terms.  Even in the WP article you refer to, it is shown that "abstract intelligent agent" and "autonomous intelligent agent" are generally just synonyms for "intelligent agent" but used to highlight certain aspects of intelligent agents in some contexts.   Net-net, I'd say there just isn't any difference there that's going to matter in practice.

"Virtual intelligent agent" OTOH, used in the context you used it, suggests the distinction between an IA that's implemented in software only, versus one that has a physical manifestation.   I don't know how useful that distinction is and I haven't seen anybody else make it.

All in all, I expect that in almost every possible context, if you just say "Intelligent Agent" with no qualifiers, that's going to be sufficient.   But if there were going to be an exception, I'd say it would be around the term "autonomous" since an agent which is truly autonomous, versus one that needs to operate in a specific, constrained environment, is a distinction that - at least in principle - could be useful.


How could facts be distinguished from opinions? As a software engineer, I am searching for an existing solution or, if none exists, willing to create one that will be able to process texts (e.g. news from online media) to extract/paraphrase dry facts from them, leaving all opinions, analysis, speculations, humor, etc., behind.
If no such solution exists, what would be a good way to start creating it (considering that I have zero experience in AI/machine learning)?
It would be no problem to manually create a set of examples (pairs of original news + dry facts extracted), but is that basically what it takes? I doubt so.
(This knowledge domain is already huge, so which parts of it need to be learned first and foremost to figure out how to achieve the goal?)

I will be starting my PhD in natural language processing in a few days and this is very similar to my proposed topic. It's an open problem that ties NLP and AI into philosophy of science and epistemology and is, I think, extremely interesting. I say all this to drive home the point that this is not a simple problem.
Two major theoretical concerns come to my mind:

What is a "fact"? Is it a universal truth, if there is such a thing? Or is it a generally accepted theory, and if so how do you measure acceptance? That is, accepted by whom, where, when?

Are there any linguistic markers of opinions vs. facts? Only in rare cases, such as when the speaker prefaces their statement with something like "I believe". In most cases, I think, opinions will be stated linguistically similarly to facts. For example, compare "Cats are felines." (a "fact") with "Cats are aliens." (an opinion some may hold). They have the exact same syntactic structure. The difference here is deeply semantic, and probably relates to the speaker's intention. I'd venture that often people state their opinions with the intention of communicating a "fact".

Some more practical concerns are:

Information extraction (also called relationship extraction, text mining, etc.), which for the most part assumes that the "facts" given in the labeled datasets are correct, is far from a solved problem. E.g. the state of the art model developed for a task released in 2010 has an F1 of only 76! What you propose adds significant uncertainty to these types of tasks.

I suspect that even if you were able to compile a dataset of facts and opinions with corresponding labels you would encounter a number of modeling problems. Given the linguistic similarity between the statements of facts and opinions, I'd guess that your model will simply memorize the dataset, making it generalize poorly to your test set. Either that or it would would pick up on random, hidden correlations in the data to solve the problem (neural nets are really good at this), perhaps generalizing to the test set, but failing to apply to any other data.

Fact vs. opinion is something that is embedded in a cultural milieu, so a model would, I think, need access to some proxy for what is culturally accepted in order to make this distinction, perhaps a via knowledge base. This may be feasible for limited, highly curated domains (e.g. biomedicine), but there is currently nothing suitable for a general-purpose fact finder.

tldr: No, it is not enough to simply create a dataset of facts vs. opinions. This problem poses major theoretical concerns related to epistemology, linguistics, and cognitive science. Additionally, there are more mundane (but non-trivial!) modeling issues to consider. @Sceptre is right that it will be impossible to start this without knowledge of AI/ML/NLP, especially a rather deep knowledge of what current AI systems are really capable of.


How do we know that the neurons of an artificial neural network start by learning small features? I'd like to ask you how do we know that neural networks start by learning small, basic features or "parts" of the data and then use them to build up more complex features as we go through the layers. I've heard this a lot and seen it on videos like this one of 3Blue1Brown on neural networks for digit recognition. It says that in the first layer the neurons learn and detect small edges and then the neurons of the second layer get to know more complex patterns like circles... But I can't figure out based on pure maths how it's possible.

We do it experimentally; you're able to look at what each layer is learning by tweaking various values throughout the network and doing gradient ascent. For more detail, watch this lecture: https://www.youtube.com/watch?v=6wcs6szJWMY&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&index=12 it provides many methods used for understanding exactly what your model is doing at a certain layer, and what features it has learnt.


Why does an LSTM cycle on initialisation? I initialised an LSTM with Xavier initialisation, although I've found this occurs for all initialisations I have tested. When initialised, if the LSTM is tested with a random input, it will get stuck in a cycle, either over a few characters or just one. Example output:

nhhbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

f(b(bf(bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

kk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,m

I've also noticed the LSTM is particularly bad in this way, that even when trained it has a tendency to get stuck in loops like this. It seems it has difficulty truly retaining context strong enough to over power the input, activation and output gates with only the forget gate. Is there an explanation for this?

This doesn't necessarily answer the question, but it does give some possible solutions to mitigate the problem.

Apparently, the emphasised looping behaviour above is a result of improper initialisation, in which I was initialising with only positive and 0 weights. The looping behavior is largely diminished by proper initialisation, however it is not totally removed. See below for examples:

djbgnuywjkfrrkrkrpueeeeeffrrdv
kmkkkkkkkkkkkrkkkkkkkkkkkkkkkk
clfwtpjbsqeeeeeeesssjjeeeeeeee
ldwgmbvcmmbvcmmmgmmmgmmmgmmmmg
ywzrfotntntnttntntunuwqzffotnt
lvvurrrrrrxafllvvurrrrrrxcfhla

(These are samples from 6 different uniquely initialised LSTM's on the exact same architecture).

I have also found that this common behaviour is typically combated by adding a certain degree of randomness in feeding in the LSTM's output. This is normally done by interpreting the output at time step $t$ of the LSTM as a probability distribution, used to pick the next value fed into the LSTM at $t+1$. This helps break non-confident looping behaviour which is where looping behaviour most commonly occurs through some quick experimentation, while still mostly retaining confident predictions.

I also tested this by randomising the input a little bit, using a probability distribution where 90% of the time the correct input is selected and 10% a random one is, and back-propagating as normal. This didn't seem to have much of an effect on the looping behaviour, although perhaps might work as a good form of regularisation. I am yet to test it.

You can also use the temperature method in LSTMs, which explained really well here: https://cs.stackexchange.com/q/79241/20691.


Is there a formal name for this philosophical AI problem? Expressed in my own words:

  Suppose we create something that passes all of our tests and is
  indistinguishable from another human. How can you know if this is
  truly a conscious being as a human is, or simply a simulation of
  conscience?

What is the name for this? Where was it first written about?

If something is indistinguishable from a human it is as intelligent as a human. There is no such thing as simulated intelligence.

Consciousness of course is a different matter and I suspect that's what you really have in mind (no pun intended).

The question how to distinguish somebody who is truly conscious from somebody who just acts outwardly like a conscious being is often discussed under the term philosophical zombie.


How to implement RL policies learned on a finite horizon? I am modelling a ride-hailing system where passenger requests continuously arrive into the system. An RL model is developed to learn how to match those requests with drivers efficiently.
Basically, the system can run infinitely as long as there are requests arriving (infinite horizon reality). However, in order for the RL training to conduct, the episode length should be restricted to some finite duration, say $[0,T]$ (finite horizon training).

My question is how to implement the learned policy based on finite horizon $[0,T]$ to the real system with infinite horizon $[0,\infty]$?

I expect there would be a conflict of objectives. The value function near $T$ is partially cut off in a finite horizon and would become an underestimate and affect policy performance in an infinite horizon implementation. To this end, I doubt the applicability of the learned policy.

A normal way to deal with training on infinite horizon (aka "continuing" or "non-episodic") problems is to use TD learning or other bootstrapping methods (of which Q-learning in DQN is one example), and to treat the cutoff at $T$ for pseudo-episodes as a training artefact.
If the state at time $T$ was really a terminal state, the TD target would be just $r$ because $q(s^T,\cdot) = 0$ by definition, but that doesn't apply in your case.
So always use the bootstrapped TD target - e.g. $r + \gamma \text{max}_{a'} \hat{q}(s',a',\theta)$ for single step TD target with a Q estimate having $\theta$ as learned params - and don't treat the horizon data any differently.
If you do this, then your concerns about under-estimates should not be an issue. Your pseudo-episodes do need to be long enough to observe the impact of multiple requests in the long term is the main issue (setting $T$ too low so that the system does not reach any kind of equilibrium would be a problem).
You could also use an average reward setting and differential value functions. It is slightly better from a theoretical standpoint, but Q-learning and DQN is fine if you don't want to be bothered with that. The same basic rule applies - ignore "episode end" for constructing TD targets because it is just a training artefact. Also ensure you set $T$ high enough that the long term impacts of a policy are observable.
If your starting state is special (e.g. cars all in fixed places, and no requests in progress) this could also be an issue, because the real world system will rarely be in that state but you will have many episode starts with it. If you cannot start the system in a reasonable random initial state for your problem, you may also want to discard data from episode starts and allow a run-in time before using data for training from the pseudo-episodes.


Is there a way to make my neural network discard inputs with bad results from learning? What I want to achieve is this: If my desired outputs are [1, 2, 3, 4] I would rather have my network produce this output:
[0.99, 2.01, 999, 4.01]
than say this:
[0.94, 1.88, 3.12, 4.1]
So I'd rather have a few very accurate outputs and the rest completely off, than have them all be decent but no more than that. My question is, is there a known way to do this? If not, would it make sense to remove the inputs that produce poor outputs, and redo the learning phase?

I assume [1, 2, 3, 4] are the desired outputs for different examples for a regression task. Sound like you need a different loss function. From your description it seems you don't care how big the error is if it's bigger than some value. Try the Huber loss(in Pytorch and TensorFlow). Examples that are far from the expected value won't produce big gradients (:


Has anybody tried unsupervised deep learning from youtube videos? YouTube has a huge amount of videos, many of which also containing various spoken languages. This would presumably provide something like the data that a "challenged" baby would experience - "challenged" meaning a baby without arms or legs (unfortunately many people are born that way).

Would this not allow unsupervised learning in a deep learning system that has both vision and audio capabilities? The neural network would presumably learn correlations between words and images, and could perhaps even learn rudimentary language skills, all without human supervision. I believe that the individual components to do this already exist.

Has this been tried, and if not, why?

Yes, it is possible, and yes, it probably has been done before. Odds are, however, the person(s) who tried were disappointed with the results and forgot to tell others.

The reasons they might be disappointed could be any of the following:

Took to long to train
Even when fully trained, (or appeared to be), it did not give the expected output
Too sensitive to variations in the input


Why is the average reward plot for my reinforcement learning agent different than the usual plots? I'm building an RL agent using SARSA and Q-Learning for testing its capabilities.

The environment is a 10x10 grid, where it gets a reward of 1 if he reaches the goal while he takes -1 every time he takes a step out of the grid. So, it can freely move out and every time it takes a step outside of the grid it gets -1.

After tuning the main parameters

alpha_val: 0.25
discount: 0.99
episode_length: 50
eps_val: 0.5

I get the following plot for 10000 episodes (The plot is sampled every 100 episodes):

But when I look at the plots online I see usually plots like this one:

Since I'm new at RL, I'm asking some comments about my outcome or any type of suggestion if anyone of you think that I'm doing something wrong.

Well, the way to know that the agent is actually learning is by looking at its behavior while it performs the task, and by comparing against a known optimal performance.

So, does your agent reaches the goal quickly? Does it step out of the grid frequently? What is the maximum possible sum of rewards / minimum number of steps attainable? Is the agent close to that limit? From your graphic, and if I understood correctly your RL problem, the maximum average reward per step should be close to 1 (depending on the specific environment you are using), so I guess you are not so far from the optimal solution.

Also, probably if you keep training for a longer period, your agent will reach a stable solution that might or might not be optimal. If you keep training after that, your curves surely will look like the ones you found online.


How does DDPG algorithm know about my action mapping function? I am using DDPG to solve a RL problem. The action space is given by the Cartesian product $[0,20]^4\times[0,6]^4$. The actor is implemented as a deep neural network with an output dimension equals to $8$ with tanh activation.
So, given a state s, an action is given by a = actor(s) where a contains real numbers in [-1,1]. Next, I map this action a into a valid action valid_a that belongs to the action space $[0,20]^4\times[0,6]^4$. Than, I use valid_a to calculate the reward.
My question is: how does the DDPG algorithm know about this mapping that I am doing? In what part of the DDPG algorithm should I specify this mapping? Should I provide a bijective mapping to guarantee that the DDPG algorithm learns bad from good actions?

I would recommend doing is allowing your network to output any real number and then clipping the output. For instance, I was working with an agent that had to learn an angle between $[0, 2\pi]$ and $[0, 1]$. If the network outputted e.g. 10 in the first dimension then this would just be clipped to $2\pi$.
This way the agent only learns about actions within the action space and the weights of the network would eventually be adjusted to only output actions within this action space, provided that the boundaries aren't the optimal actions.


Is back-propagation applied for each data point or for a batch of data points? I am new to deep learning and trying to understand the concept of back-propagation. I have a doubt about when the back-propagation is applied.  Assume that I have a training data set of 1000 images for handwritten letters,

Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch?

Is back-propagation applied $n$ times till the neural network gives a satisfactory result for a single data point before going to work on the next data point?

Short answers

Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch?

You can perform back-propagation using (or after) only one training input (also known as data point, example, sample or observation) or multiple ones (a batch). However, the loss function to train the neural network is slightly different in both cases.

Is back-propagation applied $n$ times till the neural network gives a satisfactory result for a single data point before going to work on the next data point?

If we use only one example, we usually do not wait until the neural network gives satisfactory results for a single input-label pair $(x_i, y_i)$, but we keep feeding it with several input-label pairs, one after the other (and each time updating the parameters of the neural network using back-propagation), without usually caring whether the neural network already produces a satisfactory output for an input-label pair before feeding it with the next (although you could also do that).
Long answer (to the 1st question)
In case you want to know more about the first question, keep reading!
What is back-propagation?
Back-propagation is the (automatic) process of differentiating the loss function, which we use to train the neural network, $\mathcal{L}$, with respect to all of the parameters (or weights) of the same neural network. If you collect the $N$ parameters of the neural network in a vector
$$\boldsymbol{\theta} =
\begin{bmatrix}
\theta_1\\
\vdots \\
\theta_N
\end{bmatrix},
$$
then the derivative of the loss function $\mathcal{L}$ with respect to $\boldsymbol{\theta}$ is called the gradient, which is a vector that contains the partial derivatives of the loss function with respect to each single scalar parameter of the network, $\theta_i$, for $i=1, \dots, N$, that is, the gradient looks something like this
$$
\nabla \mathcal{L} =
\begin{bmatrix}
\frac{\partial \mathcal{L}}{ \partial \theta_1}\\
\vdots \\
\frac{\partial \mathcal{L}}{ \partial \theta_N}
\end{bmatrix},
$$
where the symbol $\nabla$ denotes the gradient of the function $\mathcal{L}$.
Loss functions
The specific loss function $\mathcal{L}$ that is used to train the neural network depends on the problem that we need to solve. For example, if we need to solve a regression problem (i.e. predict a real number), then the mean squared error (MSE) can be used. If we need to solve a classification problem (i.e. predict a class), the cross-entropy (CE) (aka negative log-likelihood) may be used instead.
Example
Let us assume that we need to solve a regression problem, so we choose the mean squared error (MSE) as the loss function. For simplicity, let's also assume that the neural network, denoted by $f_{\boldsymbol{\theta}}$, contains only one output neuron and contains no biases.
Stochastic gradient descent
Given an input-label pair $(x_i, y_i) \in D$ (where $D$ is a dataset of input-label pairs), the squared error function (not the mean squared error yet!) is defined as
$$\mathcal{L}_i(\boldsymbol{\theta}) = \frac{1}{2} (f_{\boldsymbol{\theta}}(x_i) - y_i)^2,$$
where $f_{\boldsymbol{\theta}}(x_i) = \hat{y}_i$ is the output of the neural network for the the data point $x_i$ (which depends on the specific values of $\boldsymbol{\theta}$) and $y_i$ is the corresponding ground-truth label.
As the name suggests, $\mathcal{L}_i$ (where the subscript $_i$ is only used to refer to the specific input-label pair $(x_i, y_i)$) measures the squared error (i.e. some notion of distance) between the current prediction (or output) of the neural network, $\hat{y}_i$, and the expected output for the given input $x_i$, i.e. $y_i$.
We can differentiate $\mathcal{L}_i(\boldsymbol{\theta})$ with respect to the parameters of the neural network, $\boldsymbol{\theta}$. However, given that the details of back-propagation can easily become tedious, I will not describe them here. You can find more details here.
So, let me assume that we have a computer program that is able to compute $\nabla \mathcal{L}_i$. At that point, we can perform one step of the gradient descent algorithm
$$
\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma \nabla \mathcal{L}_i, \label{1}\tag{1}
$$
where $\gamma$ is the learning rate and $\leftarrow$ is the assignment operator. Note that $\boldsymbol{\theta}$ and $\nabla \mathcal{L}_i$ have the same dimensions, $N$.
So, I have just shown you that you can update the parameters of the neural network using only one input-label pair, $(x_i, y_i)$. This way of performing GD with only one input-label pair is known as stochastic gradient descent (SGD).
Batch (or mini-batch) gradient descent
In practice, for several reasons (including learning instability and inefficiency), it is rarely the case that you update the parameters using only one input-label pair $(x_i, y_i)$. Instead, you use multiple input-label pairs, which are collected in a so-called batch
$$B = \{(x_1, y_i), \dots, (x_M, y_M) \},$$
where $M$ is the size of batch $B$, which is also known as mini-batch when $M$ is smaller than the total number of input-label pairs in the training dataset, i.e. when $|B| = M < |D|$. If you use mini-batches, typical values for $M$ are $32$, $64$, $128$, and $256$ (yes, powers of 2: can you guess why?). See this question for other details.
In this case, the loss function $\mathcal{L}_M(\boldsymbol{\theta})$ is defined as as the mean (or average) of the squared errors for single input-label pairs, $\mathcal{L}_i(\boldsymbol{\theta})$, i.e.
\begin{align}
\mathcal{L}_M(\boldsymbol{\theta})
&= \frac{1}{M} \sum_{i=1}^M \mathcal{L}_i(\boldsymbol{\theta}) \\
&= \frac{1}{M} \sum_{i=1}^M \frac{1}{2} (f_{\boldsymbol{\theta}}(x_i)-y_i)^2 \\
&= \frac{1}{M} \frac{1}{2} \sum_{i=1}^M (f_{\boldsymbol{\theta}}(x_i)-y_i)^2.
\end{align}
The normalisation factor, $\frac{1}{M}$, can be thought of as averaging out the losses of all input-label pairs. Note also that we can take the $\frac{1}{2}$ out of the summation because it is a constant with respect to the variable of the summation, $i$.
In this case, let me also assume that we are able to compute (using back-propagation) the gradient of $\mathcal{L}_M$, so that we can perform a gradient descent (GD) update (using a batch of examples)
$$
\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma \nabla \mathcal{L}_M \label{2} \tag{2}
$$
The only thing that changes with respect to the GD update in equation \ref{1} is the loss function, which is now $\mathcal{L}_M$ rather than $\mathcal{L}_i$.
This is known as mini-batch (or batch) gradient descent. In the case $M = |D|$, this is simply known as gradient descent, which is a term that can also be used to refer to any of its variants (including SGD or mini-batch GD).
Further reading
You may also be interested in this answer, which provides more details about mini-batch gradient descent.


In practice, are perceptrons typically implemented as objects? I'm fairly new to ANNs. I know the general structure, the math, and the algorithms behind them. I figured the logical next step on my journey to fully understanding them would to be implement one myself from scratch, even if it's a fairly small one.
So I'm curious, coming from those who actually work with and deploy these things, are perceptrons/neurons typically implemented as objects with class variables, methods, etc. (kind of like nodes in a Linked List)? Or is there a more practical/memory-conservative way to build them?

Unless one performs an exhaustive search, it's difficult to answer your question.
However, in the widely used libraries, such as TensorFlow, PyTorch and sklearn, most abstractions (like neural networks and layers) are implemented as classes (see this, this and this examples, respectively), as the main programming language supported by these libraries is Python, which an object-oriented language (but note that Python also supports other programming paradigms, such as functional programming).
I don't know the statistics, but, from my experience, I would say that OOP (which tends to be intuitive to humans for obvious reasons) is the mostly widely used programming paradigm, as opposed to the (pure) functional paradigm.
However, in general, the programming paradigm used to implement a certain concept probably depends on the language that you want to use. For example, in Haskell, a purely functional programming language, you will probably implement a perceptron as a sequence of functions (see this example). Another example is NumPy, which, although the primary interface is written in Python, under the hood, is primarily implemented in C, a non-OOP language (see e.g. this example, where you see many functions, but no class).
This should also partially answer your other question. In some cases, you will implement a concept using the programming language and paradigm that improves the efficiency of your implementation (e.g. the case of NumPy).


Is known math really enough for AI As an Electronics & Communication Engineering student I've heard some stories and theories about "The math we have is not enough to complete a thinker-learner AI."

What is the truth? Is humankind waiting for another Newton to make new calculus or another Einstein-Hawking to complete the quantum mechanics?

If so, what exactly do we need? What will we call it?

Maybe.

AI has a long history of encountering mathematical impossibilities and then working around them already. While the individuals who solved these problems don't get as much press as Newton, Einstein, or Hawking, a case could be made that their contributions to human knowledge are on a similar scale. Unfortunately, their results don't relate to physical systems, so they can be harder to explain to the layperson.

Something to keep in mind is that your question assumes the correctness of the "Great Man Theory" of science history, which holds that science advances by the efforts of exceptional people, and that we need to wait around for more such people to appear for science to advance. This view of scientific history is overly simplistic, and probably wrong. For example, most (all?) of Newton's discoveries were likely to have been made by someone else, around the same time, if he didn't make them (see, e.g. Leibnitz, who actually did discover calculus at the same time), so a better view might be of a large community of researchers who gradually develop more advanced models based on each others' work.

To answer your question, I've listed out some past examples of problems that were overcome, and some outstanding problems that might require the development of new mathematical tools to solve properly. Keep in mind though, that we can't know for sure whether new tools are needed: maybe existing tools are sufficient, but no one has applied them in the right way yet!

Some past problems that plagued AI and required the development of better mathematical approaches:

The combinatorial explosion problem appeared to preclude having an AI system reason about probabilities and causation in a logically correct way. This problem was solved by the development of Bayesian networks and causal networks, with the work being led by Judea Pearl and his students. Pearl won a Turing award for this, but has received very little coverage in the popular press.
Many optimization problems that AI systems need to solve are NP-hard. This means that no general purpose algorithms exist that can give exact solutions to these problems in a polynomial number of steps. Because of this, the problems were initially viewed as intractable. The development of PTAS algorithms, and a deep understanding of phase-changes in NP-Hard problems (pioneered by Cheeseman et al.) led to an ability to identify exactly what makes these problems hard, to identify which subparts cause the hardness, and to practical algorithms that solve all sorts of problems in these domains.
Much effort in AI was spent designing new classification methods, and arguing about why one might or might not be better than another. The No Free Lunch Theorems, along with other work in computational learning theory provided a clear mathematical framework for understanding how systems can learn, and where their limits are.
As a more recent example, games involving chance, like Poker, have state spaces so vast that they cannot be searched through effectively, even with heuristics. Phrasing these games in the language of Game Theory, and proving the convergence of the counter-factual minimax regret algorithm provided solutions. Bowling et al. were at the center of this work. It got some press coverage, but most of the coverage focused on "computers can play poker", and not the exiting, more generalizable, lessons AI researchers had learned in the process.

Some problems that are still outstanding, and might require new mathematical techniques:

Are there efficient algorithms for solving problems in the complexity classes PPAD, NP, or #P? Many AI problems fall into these categories. We do not know the answer definitively, and although most researchers suspect that no such algorithms exist, understanding why not seems like it could also provide major research advantages. Existing proof techniques do not seem likely to crack this problem.
We have no good mathematical models of subjective experience. While some researchers (like Churchland) think this is not likely to play a major role in the development of intelligent systems, a concrete model could either support or refute that view, and might provide solid frameworks for current problems in AI like the study of motivation.
The original project of the social sciences was to provide mathematical models of individual humans, and then of human societies. This has mostly been abandoned (Psych is still at it, but Economics prefers to study rational agents instead of humans, and most of the others gave up mathematical modeling entirely to pursue the methodologies of the humanities instead of the sciences). Nonetheless, the lack of sound mathematical descriptions of human behaviours is becoming a major topic within AI, with work on norms, trust, emotion, and other topics. If a mathematical framework were developed to describe the actions of human societies or of individual humans, then AI would be advanced significantly (along with many other fields!).


Are we supposed to "decide" ethics for intelligent systems? Ethics is defined to be the set of moral principles that governs a person's or group's behavior.

Innately, shouldn't any system devise ethics for itself? Most of our articulations, in either direct or indirect manner, talk about intelligent systems in context of human presence. So, why, in first place, are we studying AGI ethics?

Given Roseau's reasoning for a society

  Human beings can improve only when they leave the state of nature and enter a civil society.

Doesn't that also automatically apply to any intelligent system? If it does, then can't we imply that any inorganic intelligent system should, perhaps, come as an offset of a network of intelligent computers (a society per se)?

Who are we to constitute ethics and rules, morale for another society which, perhaps, could be much more intelligent than us?

Nonhuman ethics can be totally arbitrary

Regarding your citation, perhaps it's useful to consider why should anything that philosophers have said about human ethics apply for arbitrary nonhuman systems? We all share a common biological background, our behavior, emotions and especially positive/negative feelings arise from the particular way how our brains are built. Man is by nature a social animal, and much of our behavior and feelings of empathy, compassion, fairness, greed, social status, etc are formed and expressed in society. Our minds are adjusted for this, as are the minds of other mammals. A human that forms an ethical system when joining a civil society does so in the basis of all this shared context, and within the limitations of it.

However, all this is orthogonal to intelligence. All the variety of mankind encompasses a tiny island in the whole space of possible intelligences and their goals. Any system of ethics is feasible, a vast majority of them having no relationship whatsoever to what we could consider ethical. A human can be expected to join a society and learn ethics because our brain is hardwired to respond to social stimuli - an AI would learn ethics from a society only if its previous ethics already happen to value that highly, and that isn't likely to happen without careful design.

There is a thought experiment that is commonly used as an example - the "paperclip maximizer", an artificial agent with an ethical system that defines 'good' proportionally to the number of paperclips that exist in the universe and only that - and that is a perfectly reasonable example, because a completely random ethical system is overwhelmingly likely to reduce to something stupid like that. The space of "reasonable" (to us) ethical systems is very tiny compared to all ethical systems possible, and expecting that a randomly selected ethical system will just happen to land there is like relying on winning a high-stakes lottery.

We want their ethics to contain certain things

So the main reason why we are studying AGI ethics is because we (or some of us) really, really, want the AGI ethical system to include certain things. For example, I'd like that ethical system to include my survival. I'd prefer that ethical system to be consistent with my happiness and wellbeing - and the very existence of a very powerful entity that simply happens to not care about me at all is not compatible with my survival and wellbeing, so I'd really prefer for it to care about me, or alternatively never become very powerful.

One could certainly argue that we have no right to constitute ethics, rules and morale for another society which could be much more intelligent and better than us. That another society might be "better" in some manner - but better for whom ? If it's better for "them" but worse for me (and humanity), then I have strong motivation to simply claim that right and try my hardest to ensure that this hypothetical future society includes me, my desires and my ethics.


What is the difference between deep learning and shallow learning? What is the difference between deep learning and shallow learning?

What I am interested in knowing is not the definition of deep learning & shallow learning, but understanding the actual difference.

Links to other resources are also appreciated.

That article only mentions "shallow learning" in the title and it mentions "shallow" to refer to the fact that deep learning models are not really learning any "deep" concepts, where "deep" here means "philosophically deep". So, I think the title is just (fairly?) provocative.

Currently, in machine learning, the expression "shallow learning" isn't really standardized, as opposed to deep learning, which refers to learning, with gradient descent and back-propagation, from (typically) huge amounts of data with neural networks. Nevertheless, "shallow learning" may occasionally refer to everything that isn't deep learning (e.g. traditional machine learning models, such as support vector machines), but most likely it refers to learning in neural networks with only a small number (0-2) of hidden layers (i.e. non-deep neural networks).

Note that the difference between deep and shallow neural networks isn't really clear. Some people may consider neural networks with only 1-2 hidden layers already deep, while others may consider only neural networks with e.g. 5-10 hidden layers deep. This also shows that deep learning isn't actually well-defined too.

The other linked article actually says

  CNN performance was compared to that of conventional (shallow) machine learning methods, including ridge regression (RR) on the images’ principal components and support vector regression.

So, in this article, they use "shallow learning" to refer to traditional (or conventional) machine learning models, which confirms what I said above.


Why do very deep non resnet architectures perform worse compared to shallower ones for the same iteration? Shouldn't they just train slower? My understanding of the vanishing gradient problem in deep networks is that as backprop progresses through the layers the gradients become small, and thus training progresses slower. I'm having a hard time reconciling this understanding with images such as below where the losses for a deeper network are higher than for a shallower one. Should it not just take longer to complete each iteration, but still reach the same level if not higher of accuracy?

In theory, deeper architectures can encode more information than shallower ones because they can perform more transformations of the input which results in better results at the output. The training is slower because back propagation is quite expensive, as you increase the depth, you increase the number of parameters and gradients that need to be computed.

Another issue that you need to take into account is the effect of the activation function. Saturating functions like sigmoid and hyperbolic tangent result in very small gradients in their edges, other activation functions are just flat, eg. ReLU is flat on the negatives therefore, there is no error to propagate because the gradient is either very small (as in saturating functions) or zero. Batch Norm greatly assists in this operation because it collapses values in better ranges where the gradients aren't close to zero.


Why isn't the implementation of my policy evaluation for a simple MDP converging? I am trying to code out a policy evaluation algorithm to find the $V^\pi(s)$ for all states. The following diagram below shows the MDP.

In this case i let p = q = 0.5.
the rewards for each states are independent of action. I.e $r(\sigma_0)$ = $r(\sigma_2)$ = 0,$r(\sigma_1)$ = 1, $r(\sigma_3)$ = 10. Terminal state is $r(\sigma_3)$

I have the following policy, {0:1, 1:0, 2:0}, where key is the state and value is the action. 0 for $a_0$ and 1 for $a_1$.

#Policy Iteration solver for FUN
class PolicyEvaluation:
    def __init__(self, policies):
        self.N = 3
        self.pi = policies
        self.actions = [0, 1] # a0 and a1
        self.discount = 0.7
        self.states = [i for i in range(self.N + 1)]

    def terminalState(self, state):
        return state == 3

    # assume p = q = 0.5
    def succProbReward(self, state):
        # (newState, probability, reward)
        spr_list = []
        if (state == 0 and self.pi[state] == 0):
            spr_list.append([1, 1.0, 1])
        elif (state == 0 and self.pi[state] == 1):
            spr_list.append([2, 1.0, 0])
        elif (state == 1 and self.pi[state] == 0):
            spr_list.append([2, 0.5, 0])
            spr_list.append([0, 0.5, 0])
        elif (state == 2 and self.pi[state] == 0):
            spr_list.append([1, 1.0, 0])
        elif (state == 2 and self.pi[state] == 1):
            spr_list.append([3, 0.5, 10])
            spr_list.append([2, 0.5, 0])
        return spr_list

def policyEvaluation(mdp):
    # initialize
    V = {}
    for state in mdp.states:
        V[state] = 0

    def V_pi(state):
        return sum(prob * (reward + mdp.discount*V[newState]) for prob, reward, newState in
        mdp.succProbReward(state))

    while True:
    # compute new values (newV) given old values (V)
        newV = {}
        for state in mdp.states:
            if mdp.terminalState(state):
                newV[state] = 0
            else:
                newV[state] = V_pi(state)

        if max(abs(V[state] - newV[state]) for state in mdp.states) < 1e-10:
            break
        V = newV
        print(V)
    print(V)

pE = PolicyEvaluation({0:1, 1:0, 2:0})
print(pE.states)
print(pE.succProbReward(0))
policyIteration(pE)

I've tried to run the code above to find the values for each state, however, I am not converging with my values.

Is there something wrong that I did?

The issue is that in your list comprehension in def V_pi(state) you have

return sum(prob * (reward + mdp.discount*V[newState]) for prob, reward, newState in
        mdp.succProbReward(state))

whereas with the way you have defined the succProbReward output, it should be

return sum(prob * (reward + mdp.discount*V[newState]) for newState, prob, reward in
        mdp.succProbReward(state))

When I run this it converges immediately with a reward of 0 for all states, which I believe is correct for the policy you specified. If I change the policy it also seems to give reasonable results.


NLP: Are hashtags tokenised? I am exploring a potential NLP project and was wondering what generally is done with the hashtags words (e.g. #hello). Are those words ignored? is the # removed and the word tokenised? is it tokenised with the #?
Thank you

Word including Hash tag need to be tokanised. It has a special meaning and given a context. The word by itself has a different meaning.


Autoencoder produces repeated artifacts after convergence As experiment, I have tried using an autoencoder to encode height data from the alps, however the decoded image is very pixellated after training for several hours as show in the image below. This repeating patter is larger than the final kernel size, so I would think it would possible to remove these repeating patterns from the image to some extent.

The image is (1, 512, 512) and is sampled down to (16, 32, 32). This is done with pytorch. Here is the relevant sample of the code in which the exact layers are shown.

        self.encoder = nn.Sequential(
                # Input is (N, 1, 512, 512)
                nn.Conv2d(1, 16, 3, padding=1), # Shape (N, 16, 512, 512)
                nn.Tanh(),
                nn.MaxPool2d(2, stride=2), # Shape (N, 16, 256, 256)
                nn.Conv2d(16, 32, 3, padding=1), # Shape (N, 32, 256, 256)
                nn.Tanh(),
                nn.MaxPool2d(2, stride=2), # Shape (N, 32, 128, 128)
                nn.Conv2d(32, 32, 3, padding=1), # Shape (N, 32, 128, 128)
                nn.Tanh(),
                nn.MaxPool2d(2, stride=2), # Shape (N, 32, 64, 64)
                nn.Conv2d(32, 16, 3, padding=1), # Shape (N, 16, 64, 64)
                nn.Tanh(),
                nn.MaxPool2d(2, stride=2) # Shape (N, 16, 32, 32)
            )
        self.decoder = nn.Sequential(
                # Transpose convolution operator
                nn.ConvTranspose2d(16, 32, 4, stride=2, padding=1), # Shape (N, 32, 64, 64)
                nn.Tanh(),
                nn.ConvTranspose2d(32, 32, 4, stride=2, padding=1), # Shape (N, 32, 128, 128)
                nn.Tanh(),
                nn.ConvTranspose2d(32, 16, 4, stride=2, padding=1), # Shape (N, 32, 256 256)
                nn.Tanh(),
                nn.ConvTranspose2d(16, 1, 4, stride=2, padding=1), # Shape (N, 32, 512, 512)
                nn.ReLU()
            )

Relevant image: left side original, right side result from autoencoder

So could these pixellated effects in the above image be resolved?

Perhaps you are getting checkerboard artifacts Explained here, solutions involve changing the kernel and stride size to prevent them from being not divisible. Besides that, a solution could be to apply Gaussian smoothing to minimize the artifacts.
For example, using Gaussian smoothing in OpenCV with your image results in
import cv2
img = cv2.imread('s.png') # I took a screen shot to see how it would look
blur = cv2.blur(img,(8,8))
plt.imshow(blur)

The artifact is gone using a kernel of size (8,8). I hope this can help


How can I combine the readings of multiple lidars into 1 point cloud? I have a car with 8 lidars, each with a field of view of 60 degrees. My car looks like this:

How can I merge all the lidar readings into 1 point cloud?

A high-level solution is to use the NVIDIA DriveWorks SDK.  NVIDIA's Developer Blog has a good post, "Point Cloud Processing with NVIDIA DriveWorks SDK".

The NVIDIA DriveWorks SDK contains a collection of CUDA-based low level point cloud processing modules optimized for NVIDIA DRIVE AGX platforms. The DriveWorks Point Cloud Processing modules include common algorithms that any AV developer working with point cloud representations would need, such as accumulation and registration.

I believe you don't need the AGX hardware for the problem you posted.


How does an svm work? How does it perform comparisons between malignant and benign tumor How do Support Vector Machines (SVMs) differentiate between a glass and a bottle or between a malignant and a benign tumor when it dealing with it for the first time?

What will be the analysis mechanism involved in this?

I will try to give you a simplified explanation of how SVMs work.

The data one works with can be of two types. Either it is very easily separable and there is a clear straight line boundary between data points of different classes. We call such data linearly separable(image) or the data points of the classes are mixed in a way that there is no clear straight line boundary that separates them. In other words the data is not linearly separable.(image).

One way to make data points linearly separable is to map them into a higher dimension where they would become linearly. If we have a set of data points which are not linearly separable in 2 dimensions, we could map them into 3 dimensions where the data becomes linearly separable(like so).

SVMs work by converting data from lower to higher dimensions where it is linearly separable and then tries to find the boundary that separates the data.

The same concept could be applied to images, where the images are converted into a very high dimension in which images of different types can be separated linearly.

This could mean that the SVM model trained is able to easily distinguish between a bottle and a glass or different tumour types but that would depend entirely on the data used to train the model. The problem arises when the distribution of the training data and testing data is completely different. So for example, you trained the SVM to distinguish between glasses and cups of a certain shape and colour and while testing you show it glasses of completely different shapes and colours, the SVM will not be able to distinguish between them.


Is CNN capable of extracting the descriptive statistics features I was trying to build a CNN model. I used time series data of daily temperature to predict if there is risk of an event, say bacteria growth. I calculated the descriptive statistics of the time series, ie. mean, variance, skewness, kurtosis etc for each observation and added them to input data.

My question:

Is CNN capable of extracting the effect of the descriptive statistics the label, meaning that adding these descriptive statistics features manually does not make a difference?
(I will still try this later, but like to hear what you think about it). Thanks

CNNs learn convolutional filters that get trained on finding local, recurring patterns in some kind of image/volume data. 1D convolution is actually a thing, but I think what would be more suitable for your case is using Recurrent Neural Nets. They are specifically designed for working on time series-es of heterogeneous data.
Update:
I would like to reconsider the answer I gave earlier.
First of all, in case of dealing with time series data where it is uncertain over which time span a given event to be detected lasts, I'd generally consider using Recurrent Neural Networks (RNNs) rather than CNNs, since RNNs maintain so-called hidden states that can carry potentially useful information over time, i.e. past inputs may influence present outputs. So, either if you know that the event to be detected spans many time steps (which might not even be consecutive time steps) or if you are uncertain about how long exactly the time span of an event to be detected is, then I'd suggest going for RNNs. If you decide to use RNNs, adding extra statistics to your data might make the learning task easier for the network, but is not strictly required since you would expect an RNN to learn relevant time-series statistics on its own.
However, there is a case where using CNNs might actually suffice and yield good results as well. Consider the case of the figure shown below, where the white tiles are time series inputs and the blueish area is the area that a CNN filter/kernel (or multiple ones) is working on right now.

As you can see, there is a time series of data points and some statistics derived from that time series that have been appended to the input data stream. So, in the figure, the augmented time series input consists of the measurements themselves as well as mean, variance, and skewness values derived from the time series data (over consecutive time steps). Under the assumption that the type of event you are interested in is bounded, in terms of time steps, to the width of the CNN filter(s)/kernel(s) that you slide over your input data, then a simple CNN might suffice to detect the events you are interested in. In other words, if the width of your filter spans over at least as many time steps as an event of interest lasts, then your CNN will be able to learn to detect the event (given proper training). In that case, adding further statistics to your input data might indeed help or even be necessary to detect certain events, since it implicitly widens the time window that your CNN filter can observe at any given time step, since a mean (or other statistics) may also considers past data that the filter itself (or multiple filters...) might not span over at a given time step where you want to detect an event.


Why do I get a straight line as an output from a neural network? I am using feedforward neural network for regression and what I get as a result of prediction is a constant value visible on the graph below:

Data I use are typical standardised tabular numbers. The architecture is as follows:

model.add(Dropout(0.2))
model.add(Dense(units=512, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(units=256, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=1))

adam = optimizers.Adam(lr=0.1)

model.compile(loss='mean_squared_error', optimizer=adam)

reduce_lr = ReduceLROnPlateau(
    monitor='val_loss',
    factor=0.9,
    patience=10,
    min_lr=0.0001,
    verbose=1)

tensorboard = TensorBoard(log_dir="logs\{}".format(NAME))

history = model.fit(
    x_train,
    y_train,
    epochs=500,
    verbose=10,
    batch_size=128,
    callbacks=[reduce_lr, tensorboard],
    validation_split=0.1)

It seems to me that all weights are zeroed and only constant bias is present here, since for different data samples from a test set I get the same value, but I am not sure.

I understand that the algorithm has found smallest MSE for such a constant value, but is there a way of avoiding such situation, since straight line is not really good solution for my project?

You should try experimenting with a lower learning rate as a starting point. You're starting with 0.1 which is quite high for most of the cases, and only reduce by 0.9 which is not much.


Computation of initial adjoint for NODE I'm reading the paper Neural Ordinary Differential Equations and I have a simple question about adjoint method. When we train NODE, it uses a blackbox ODESolver to compute gradients through model parameters, hidden states, and time. It uses another quantity $\mathbf{a}(t) = \partial L / \partial \mathbf{z}(t)$ called adjoint, which also satisfies another ODE. As I understand, the authors build a single ODE that computes all the gradients $\partial L / \partial \mathbf{z}(t_{0})$ and $\partial L / \partial \theta$ by solving that single ODE. However, I can't understand how do we know the value $\partial L / \partial \mathbf{z}(t_1)$ which corresponds to the initial condition for the ODE corresponds to the adjoint. I'm using this tutorial as a reference, and it defines custom forward and backward methods for solving ODE. However, for the backward computation (especially ODEAdjoint class in the tutorial) we need to pass $\partial L / \partial \mathbf{z}$ for backpropagation, and this enables us to compute $\partial L / \partial \mathbf{z}(t_i)$ from $\partial L / \partial \mathbf{z}(t_{i+1})$, but we still need to know the adjoint value $\partial L / \partial \mathbf{z}(t_N)$. I do not understand well about how pytorch's autograd package works, and this seems to be a barrier to understand this. Could anyone explain how it operates, and where $\partial L / \partial \mathbf{z}(t_1)$ (or $\partial L / \partial \mathbf{z}(t_N)$ if this is more comfortable) comes from? Thanks in advance.

Here's my guess for the initial adjoint from simple example. Let $d\mathbf{z}/dt = Az$ be a 2-dim linear ODE with given $A \in \mathbb{R}^{2\times 2}$. If we use Euler's method as a ODE solver, then the estimate for $z(t_1)$ is explicitly given as $$\hat{\mathbf{z}}(t_1) = \mathrm{ODESolve}(\mathbf{z}(t_0), f, t_0, t_1, \theta))= \left(I + \frac{t_1 - t_0}{N}A\right)^{N} \mathbf{z}(t_0) $$ where $N$ is the number of steps for Euler's method (so that $h = (t_1 - t_0) /N$ is the step size). If we use MSE loss for training, then the loss will be
$$
L(\mathbf{z}(t_1)) = \Bigl|\Bigl| \mathbf{z}_1 - \left(I + \frac{t_1 - t_0}{N}A\right)^N\mathbf{z}(t_0)\Bigr|\Bigr|_2^2
$$
where $\mathbf{z}_1$ is the true value at time $t_1$, which is $\mathbf{z}_1 = e^{A(t_1 - t_0)}\mathbf{z}(t_0)$. Since adjoint $\mathbf{a}(t) = \partial L / \partial \mathbf{z}(t)$ satisfies $$\frac{d\mathbf{a}(t)}{dt} = -\mathbf{a}(t)^{T} \frac{\partial f(\mathbf{z}(t), t, \theta)}{\partial \mathbf{z}} = \mathbf{0},$$
$\mathbf{a}(t)$ is constant and we get $\mathbf{a}(t_0) = \mathbf{a}(t_1)$. So we do not need to use augmented ODE for computing $\mathbf{a}(t)$. However, I still don't know what $\mathbf{a}(t_1) = \partial L / \partial \mathbf{z}(t_1)$ should be.  If my understanding is correct, since $L = ||\mathbf{z}_1 - \mathbf{z}(t_1)||^{2}_{2}$, it seems that the answer might be
$$
\frac{\partial L}{\partial \mathbf{z}(t_1)} = 2(\mathbf{z}(t_1) - \mathbf{z}_1).
$$
However, this doesn't seem to be true: if it is, and if we have multiple datapoints at $t_1, t_2, \dots, t_N$, then the loss is
$$
L = \frac{1}{N} \sum_{i=1}^{N}||\mathbf{z}_i  -\mathbf{z}(t_i)||_{2}^{2}
$$
and we may have
$$
\frac{\partial L}{\partial \mathbf{z}(t_i)} = \frac{2}{N} (\mathbf{z}(t_i) - \mathbf{z}_i),
$$
which means that we don't need to solve ODE associated to $\mathbf{a}(t)$.

Maybe later I can work through your math though I can say this.
First, a forward pass is done to obtain predictions of $z$, at every $t$. Then the adjoint state is run backward in time for every $t$. Which gives the learning impulse. So an initial run is done to obtain values of the dynamical system at all time points and the last of these is the initial point for the backward pass.
This is probably not enough to satisfy your question, hopefully, I can get more specific with respect to the math later.


Training network with 4 GPUs performance is not exactly 4 times over one GPU why? Training neural network with 4 GPUs using pyTorch, performance is not even 2 times (btw 1 & 2 times) compare to using one GPU. From Nvidia-smi we see GPU usage is for few milliseconds and next 5-10 seconds looks like data is off-loaded and loaded for new executions (mostly GPU usage is 0%). Is there any way in pyTorch to improve the data upload and offload for the GPU execution.

Your dataset class probably have a lot of preprocessing code. You should use a dataloader. It will prefetch data from the dataset when the GPU is processing. Also, you can process all the data beforehand and save to a file. Multiple GPU cannot scale as the GPU have to get all data to one GPU to calculate the loss. The performance of 4 GPU is around 3.5x. A large batch size also would help as each GPU will have 1/4 the batch size. a batch size of 64-128 is good for 4 GPU. See the following example code fro CIFAR-10 for multi gpu code. It have dataloaders and dataparallel.

import os
import time
import datetime

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
import torch.backends.cudnn as cudnn

import torchvision
import torchvision.transforms as transforms
from torchvision.datasets import CIFAR10
from torch.utils.data import DataLoader

from model import pyramidnet
import argparse
from tensorboardX import SummaryWriter

parser = argparse.ArgumentParser(description='cifar10 classification models')
parser.add_argument('--lr', default=0.1, help='')
parser.add_argument('--resume', default=None, help='')
parser.add_argument('--batch_size', type=int, default=768, help='')
parser.add_argument('--num_worker', type=int, default=4, help='')
parser.add_argument("--gpu_devices", type=int, nargs='+', default=None, help="")
args = parser.parse_args()

gpu_devices = ','.join([str(id) for id in args.gpu_devices])
os.environ["CUDA_VISIBLE_DEVICES"] = gpu_devices

def main():
    best_acc = 0

    device = 'cuda' if torch.cuda.is_available() else 'cpu'

    print('==> Preparing data..')
    transforms_train = transforms.Compose([
        transforms.RandomCrop(32, padding=4),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])

    dataset_train = CIFAR10(root='../data', train=True, download=True,
                            transform=transforms_train)

    train_loader = DataLoader(dataset_train, batch_size=args.batch_size,
                              shuffle=True, num_workers=args.num_worker)

    # there are 10 classes so the dataset name is cifar-10
    classes = ('plane', 'car', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck')

    print('==> Making model..')

    net = pyramidnet()
    net = nn.DataParallel(net)
    net = net.to(device)
    num_params = sum(p.numel() for p in net.parameters() if p.requires_grad)
    print('The number of parameters of model is', num_params)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(net.parameters(), lr=args.lr)
    # optimizer = optim.SGD(net.parameters(), lr=args.lr,
    #                       momentum=0.9, weight_decay=1e-4)

    train(net, criterion, optimizer, train_loader, device)

def train(net, criterion, optimizer, train_loader, device):
    net.train()

    train_loss = 0
    correct = 0
    total = 0

    epoch_start = time.time()
    for batch_idx, (inputs, targets) in enumerate(train_loader):
        start = time.time()

        inputs = inputs.to(device)
        targets = targets.to(device)
        outputs = net(inputs)
        loss = criterion(outputs, targets)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        train_loss += loss.item()
        _, predicted = outputs.max(1)
        total += targets.size(0)
        correct += predicted.eq(targets).sum().item()

        acc = 100 * correct / total

        batch_time = time.time() - start

        if batch_idx % 20 == 0:
            print('Epoch: [{}/{}]| loss: {:.3f} | acc: {:.3f} | batch time: {:.3f}s '.format(
                batch_idx, len(train_loader), train_loss/(batch_idx+1), acc, batch_time))

    elapse_time = time.time() - epoch_start
    elapse_time = datetime.timedelta(seconds=elapse_time)
    print("Training time {}".format(elapse_time))

if __name__=='__main__':
    main()

Source: https://github.com/dnddnjs/pytorch-multigpu/blob/master/data_parallel/train.py

Hope I can help you and have a nice day!


How to separate image recognition from logic? For example I would like to implement transparent AI in the RTS game which doesn't offer any AI API (like old games), and I'd like to use image recognition algorithm for detecting the objects which can talks to another algorithm which is responsible for the logic.

Given I'd like to use two neural networks, what are the approaches to setup the communication between them? Is it just by exporting result findings of the first algorithm (e.g. using CNN) with list of features which were found on the screen, then use it as input for another network? Or it's more complex than that, or I need to have more than two networks?

The underlying abstraction (which is essentially what you'd be using the first network for) is that of reducing the state-space of the raw input via feature extraction/synthesis and/or dimensionality reduction.

At present, there are few definite rules for doing this: practice is more a question of 'informed trial and error'.

If you add some information to your question regarding what has been previously attempted in this area (e.g. on the
ALE platform), this it might be possible to offer some more specific advice.


How to transform a PDDL to search? I have a question about search and planning:
I still haven't understood the difference from the two, but they seem very similar to me; here is a question I am struggling with:

  "Having formulated a PDDL problem, transform it into research,
  emphasizing what the differences are."

Someone can do an example?

I attached an example of simple PDDL from my book (I'm using  Russell & Norvig)

Not all search is planning (is A connected to B), but all planning is search (how do I get from this to that).

Here's an example in Prolog with a domain described in terms of actions, when they are possible, and what the result of the actions are. The description is of an uncomputed graph of un-calculated size where each node is a situation and each edge is an action. Then we have A* search algorithm that searches this graph, calculating it as it goes, to find a plan to reach the goal state. Running the final query will produce a plan via search.


How to organize artificial intelligence efforts at work? Some context: Recently all kind of salesmen have been knocking on our company's door to provide their "artificial intelligence" expertise and projects suggestions. Some don't know the difference between words estimation and validation (really), some have extraordinary powerpoints and paint themselves as gurus of the field. Our management has gone with the hype and definitely we're starting some kind of project on "artificial intelligence" (meaning rpa with some machine learning possibly).

What is the best way to start when we don't yet know to what problem we want to apply all this and I'm worried it will lead to long expensive projects with meager results? What are the things to watch out for? Any good practical books or war stories out there?

I know what you mean. It can be difficult to parse between the hype and the application of AI. Although AI (specifically deep learning) can do a lot of things, this doesn't negate previous methods that work just as well or better in certain domains. Some times managers will hear 'AI' and think of giant networks when they actually just needed a simple linear regression for their problem.

That being said, here are some suggestions to help decide if AI is right for your project and some things to consider if so :

Get some advice from a researcher. There are many AI consulting firms popping up in large cities that give companies hours of consulting time. If there isn't one in your city, go to AI meetups and meet the community, find educated people and ask them about your general questions. For more in depth advice, perhaps a local university can lend a hand.
Understand your problem. Taking the time to know exactly what problems you are trying to solve is invaluable. Not only will it help new employees but if you do end up using AI and get consultation hours from an AI expert, it will save time and money explaining what you want to do.
Know your data. First this is a check to see if you have the right data for AI and if you have enough of it. For example many problems are approached using supervised learning which requires having a lot of labeled data (Eg, think of 1000s of pictures labeled Cat or Dog depending on the image content). If you don't have or can't easily collect a large amount of labeled data, perhaps you can use an existing data set with similar data to help get you started. If that's not an option, then you probably won't like the alternative where you hand label the data or hire Amazon MechanicalTurks.
Be prepared to fail at first. AI is not easy and is even harder when you can't just Google your questions because you are doing something no one has done before. It takes some time to understand your problem, your data, and what kind of models you should try.
Do you have the infrastructure? If everything is going well and you have a working model on your local GPU, if your model is part of your product, look at how you might deploy it. Do you have your own GPU servers? Can you afford the cloud GPS that some companies offer? Do you have to learn a model for each user?


how to recognise handwriting and convert into text? I'm trying to develop a project that recognises handwriting and converts into text. What are the algorithms and tools to be used?

What you described sounds to me like Optical Character Recognition(OCR).

If you want to implement your own, I would say read through how an open source OCR like Tesseract was implemented.

Otherwise just google for OCR and you will find a list of OCR engines, both comercial and open-source. To list but a few:

Tesseract: https://github.com/tesseract-ocr/tesseract
Matlab Computer Vision Toolbox's OCR: http://www.mathworks.de/de/help/vision/ref/ocr.html

Most OCRs makes use of a combination of a segmantation and image classification(usually done with Convolutional Neural Networks).

See here for more information.


What method to identify markers in data series via machine learning I have data that is collected from several different instruments simultaneously that is generally analyzed on a location-by-location basis.  A skilled interpreter can identify "markers" in the data that represent a certain change in conditions with depth - each marker only occurs once in each series of data.  However, it is possible that a maker is absent either due to missing data or that physical condition not existing.

Often, there are dozens of these markers per location and thousands, if not 10's of thousands of measurements that need to be interpreted.  The task is not that difficult and there are many strong priors that can be used to guide interpretation.  E.g., if marker A is in location #1, and location #2 is very close to location #1, it is likely that marker A will be present in a very similar relative position.  Also, if you have markers A, B, and C they will always be in that order.  Although it could be that you have A/B/C or A/C or B/C, etc.

I am including a hand-sketched example below with 4 example locations and one data stream (I normally have 4-5 data streams per location.

I am looking for guidance on the type of algorithm to apply to this problem.  I have explored Dynamic Time warping, but the issue is that with 10-20k data samples per location, and thousands of locations, the problem becomes computationally challenging.

Also, in general you may have 10000 locations, with maybe 100 that have been hand interpreted by an expert.

Thresholding based on variance is one method for anomaly detection in time-series data.


Can gamma be greater than 1 in a DQN? If I have a DQN, and I care A LOT about future rewards (moreso than current rewards), can I set gamma to a number greater than 1? Like 1.1 perhaps?

$ \gamma $ goes up to 1, but cannot be greater than or equal to 1 (this would make the discounted reward infinite).

The discount factor $ \gamma $  determines the importance of future rewards. A factor of 0 will make the agent "myopic" (or short-sighted) by only considering current rewards, while a factor approaching 1 will make it strive for a long-term high reward. If the discount factor meets or exceeds 1, the action values may diverge. For $ \gamma =1$, without a terminal state, or if the agent never reaches one, all environment histories become infinitely long, and utilities with additive, undiscounted rewards generally become infinite.

Source:https://en.wikipedia.org/wiki/Q-learning
https://cs.stanford.edu/people/karpathy/reinforcejs/puckworld.html


Maximize loss on non-target variable I have a neural network that should be able to classify documents to target label A. The problem is that the network is actually classifying label B, which is an easier task.

To make the problem more clear: I need to classify documents from different sources. In the training data each source occurs repeatedly, but the network should be able to work on unknown sources. All documents from a single source have the same class. In this case, it is easier to identify sources than the target label so in practice the network is not really identifying the target label, but the source.

The solution to this problem is making sure that the model is bad at identifying the sources in the training data, while still attaching the right target labels.

I think the first step is to get two output layers, one for the target label and one for identifying which source it is from. My approach fails however at the training procedure: I want to minimize the loss on the target output, but maximize the loss on the non-target output. But if I maximize the loss on that non-target output, that does not mean that the network 'unlearns' the non-target labels. So the main question for the non-target output is:

TLDR; How do I define a training procedure that minimizes the loss on a non-target output layer, and then maximizes that loss on all layers before it. My goal is to have a network that is good at classifying label A, but bad at a related label B. If anyone wants to give a code example, my prefered framework is PyTorch.

You can maximize the loss by doing 1-loss (if the range of the loss is between 1 and 0. However, I am not sure if that would help. The network can just propagate the weights of the source classifier to always output the wrong answer. a better approach is to hand craft features of the document to make it less identifiable as the source. If the document have stated the source, remove it. This may help. Also, another way is to use documents from more sources so the network cannot use the source to classify the label. Hope I can help you.


What is the definition of artificial intelligence? The concept is intrinsically related with building some sort of media for the AI to exists. We may think of a digital computer, programmed to use language and act in a way that we cannot be distinguished from a human. But, does the media really mater (unconventional computation paradigms)? Does having a certain control over the limits of what the AI can do matter? Synthetic biology has the ultimate goal of building biological systems from scratch , would a synthetic brain, potentially introduced in a synthetic human, constitute AI?

I am just looking for a clear definition of what most people have in mind when they refer to AI.

AI is a broad term referring to more than one concept, each with its own definition.
Different Types of AI

The lowest levels are extremely simple and common, such are an artificial chess opponent. They work well because the programs internal model of reality is a 8x8 grid with only a few rules. The program chooses a preferred action by running simulations in it's internal model of reality.

What is often meant by AI is a "General Intelligence" that can come up with a creative solution to any directive, based on it's internal comprehension of the world. There is no existing example of this as of yet. The problem is that its internal model of reality needs to provide for every possible action. And it's possible actions and reactions may not be limited to a finite number of discretely distinct moves as in a game of chess. (At least if it works on the basis of conventional programming) Even ignoring the computational power needed to run such a broad and inefficient program, it would also require some stupendous amount of labour to program this internal model of reality in the first place.

I think in Sci-Fi, when people say AI, they mean a computer program that has a kind of awareness of itself and the world around it and can come up with creative and unexpected courses of actions in order to achieve its objective. Often the AI is NOT sentient, which is why it does not understand that its actions are morally wrong, or that its solution defeats the underlying intention of its assigned directive. The Horror lay in the concept that an amoral entity has more processing power than human kind.


When labelled data is not available, what are some common unsupervised learning algorithms for pattern recognition that can be used? In pattern recognition systems, when no labeled data is available, what are some common unsupervised learning algorithms for pattern recognition, that can be used?

There are some unsupervised learning algorithms that can be used for pattern recognition (i.e. the discovery of patterns in data). The most notable one is probably k-means, which is a clustering algorithm. In k-means, you cluster your unlabeled data into groups (or clusters) based on the distance (or similarity) between them. When a new data point arrives, you'll associate it with the most similar cluster. In this sense, you are performing pattern recognition in an unsupervised way.

Here's an excerpt from the famous book Pattern Recognition and Machine Learning (2006) by C. Bishop

  In other pattern recognition problems, the training data consists of a set of input vectors $x$ without any corresponding target values. The goal in such unsupervised learning problems may be to discover groups of similar examples within the data, where it is called clustering, or to determine the distribution of data within the input space, known as density estimation, or to project the data from a high-dimensional space down to two or three dimensions for the purpose of visualization.

So, there are other problems, apart from the problem of clustering, that you may want to solve with unsupervised learning algorithms for the purpose of pattern recognition, such as density estimation (see e.g. mixture models) or dimensionality reduction (see e.g. PCA).


What research has been done in the domain of “general game playing”? I have come across this domain via this Wikipedia article: General game playing
So, where are we when it comes to general game playing AI? (The wiki article doesn't mention the recent advances and the achievements of this domain of research, except the annual games results.)
PS: I understand that this is a General project of the Stanford Logic Group of Stanford University, California. But since then, it has become an area of research in the domain of AI.

If you haven't already come across DeepMind's advances in developing general game playing AI, you can take a look at it's DQN research. The paper describes how their deep reinforcement learning system is able to beat human levels in nearly Atari 2600 games with raw pixels and scores as input.

Also here's an interesting website - General Video Game AI Competition 2017 that also hosts links to the latest advances and research papers in this field.


Does AI rely on determinism? I don’t believe in free will, but most people do. Although I’m not sure how an act of free will could even be described (let alone replicated), is libertarian freewill something that is considered for AI? Or is AI understood to be deterministic?

I'm going to assume that by free will, you mean something like the philosophical concept of libertarian free will, which is defended by philosophers like Robert Kane. In Libertarian Free Will, individuals have some capability to make choices about their actions. The classic way to argue this is by assuming some kind of spirit-stuff (e.g. a soul) that exists outside the material world, and that this spirit-stuff constitutes the consciousness of a person. Kane tries some mental gymnastics to avoid this, but then concedes something like it in a footnote. I'm not aware of any serious work that doesn't make some kind of non-physical assumption to justify this view. If someone can point at one, I'll update the answer.

By determinism, I'm going to assume you mean the usual notion of philosophical determinism: since people's decisions depend on what happened in the past, and where they are in the present, they don't really have a choice in any meaningful sense. Philosopher's like Dennett adopt a slightly softer view (compatibilism, essentially: you don't get to make big choices, but you do get to make small ones). Appeals to Quantum Mechanics are common to justify that view. In this context, free action means something more like "did something we couldn't predict exactly". An example might be: you are pre-destined to put a can of campbell's brand tomato soup in your shopping cart, but "make a choice" about exactly which of the dozens of cans you will put in. Since small choices can have large impacts (maybe that can will give you food poisoning, and the others wouldn't), this can make all sorts of things impossible to predict exactly.

I think most AI researchers don't worry too much about these issues, but Turing actually addresses them in his paper right at the start of the field, Computing Machinary and Intelligence. The deterministic/compatibilist view point is introduced as Lady Lovelace's objection: Computers only know how to do what we tell them to, so they can't be called intelligence.

Turing's counterargument uses two prongs. First, Turing notes that computers can probably be made to learn (he was right!). If they can learn, they can do things that we don't expect, just like children do. Second, Turing notes that computers already do all sorts of things we don't expect: they have bugs. Anytime a computer exhibits a bug, it did something that was unexpected. Since we cannot generally rule out bugs in programs, computers will always do surprising things. Therefore, computers satisfy the deterministic notion of free will.

Turing also addresses the libertarian notion of free will, which is part of what he calls the "Theological Objection". The objection is that intelligence requires some kind of divine spark (like free will). Turing argues that we can't detect sparks like this right now (he actually thought we would be able to one day, and spent a lot of time looking at supernatural phenomena too). However, there's no reason to suppose that computers with the right programs won't be endowed with them. A divine creator could decide that anytime you build something brain-like, it gets a spark. If we build a program that's brain-like, maybe it gets a spark too. In the absence of some way to detect souls, it seems like we ought to just agree to treat things that seem intelligent as though they had these souls, since otherwise we don't have a really clear way to decide who is and isn't intelligent. The only remaining way is to say "only things made of human meat have souls and are intelligent". While a lot of people do actually say things like this (e.g. animals have no souls), this is a pretty arbitrary view, and I think there's no hope arguing against it. Turing suggests the "Anthropic Principal": we shouldn't assume that we're special, because the earth isn't in a special place in the galaxy, or in the universe, and we have pretty compelling evidence that we're an evolved version of other animals around us, but some groups (e.g. biblical literalists) find this unconvincing.


Dataset containing images of varying dimensions and orientations I am new to deep learning.

I have a dataset of images of varying dimensions of a certain object. A few images of the object are also in varying orientations. The objective is to learn the features of the object (using Autoencoders).

Is it possible to create a network with layers that account for varying dimensions and orientations of the input image, or should I strictly consider a dataset containing images of uniform dimensions? What is the necessary criteria of an eligible dataset to be used for training a Deep Network in general.

The idea is, I want to avoid pre-processing my dataset by normalizing it via scaling, re-orienting operations etc. I would like my network to account for the variability in dimensions and orientations. Please point me to resources for the same.

Almost always people will resize all their images to the same size before sending them to the CNN.  Unless you're up for a real challenge this is probably what you should do.

That said, it is possible to build a single CNN that takes input of images as varying dimensions.  There are a number of ways you might try to do this, and I'm not aware of any published science analyzing these different choices.  The key is that the set of learned parameters needs to be shared between the different inputs sizes.  While convolutions can be applied at different images sizes, ultimately they always get converted to a single vector to make predictions with, and the size of that vector will depend on the geometries of the inputs, convolutions and pooling layers.  You'd probably want to dynamically change the pooling layers based on the input geometry and leave the convolutions the same, since the convolutional layers have parameters and pooling usually doesn't.  So on bigger images you pool more aggressively.

Practically you'd want to group together similarly (identically) sized images together into minibatches for efficient processing.  This is common for LSTM type models.  This technique is commonly called "bucketing".  See for example http://mxnet.io/how_to/bucketing.html for a description of how to do this efficiently.


How can I measure an object using Computer Vision and neural network techniques? I am studying about Computer Vision around 2-3 months long. Recently, I have an idea to conduct a research about computer vision and Artificial neural networks. The idea is I want to make an automatic object measurement using ANN. The purpose is to measure the distance between two opposite sides of the objects.

The main thing is this research is going to work as the fractional caliper tools works. Fractional caliper is a tool that use to measure the object. However, it will be time consuming to measure huge amounts of objects if there are a lot of small objects in the industry. Through this tool, I found an idea in how to make a system that can automatically give an accurate results similar as the fractional calipers.

For example, giving an image of the object and the system can define the depth, weight or high, etc., Thus, the idea has been explained above, please suggest me what kinds of Computer Vision model used to apply for this study? And how to combine it with ANN technique?

Father Ted explains why this is a hard problem.

Seriously -- if you have stereo images it should be possible, since that's what we use for depth perception. If you know how far away points x1 and x2 are, then you can measure distance using trigonometry. No neural networks needed, I guess. https://en.wikipedia.org/wiki/Triangulation_(computer_vision)


Real time ticket similarity I'm dealing with a "ticket similarity task".

Every time new tickets arrive at the help desk (customer service), I need to compare them and find out about similar ones.

In this way, once the operator responds to a ticket, at the same time he can solve the others similar to the one solved.

I expect an input ticket and all the other tickets with their similarity in output.

I thought about using DOC2VEC, but it requires training every time a new ticket enters ...

what do you recommend?

You need to create an active learning loop over the process of the learning. Try to start from a history of tickets and using doc2vec to get the similarity. When you find a bad result in the result of your classifier, then report it and then try to retrain the classifier. Also, you can wait to retrain the model, up to finding the predefined batch0size of the new data which are not in the training set.

Also, to get a better result in the active learning loop, you can testify incoming data by the measuring of the classifier uncertainty over it. If the entropy of the classifier over the data is not in a good situation, you can label the data by the operator (as an oracle) and then up reach to the predefined batch-size, retrain the classifier.

Morevoer, to know better about the active learning process and query strategies follow this link (and other articles in that link like this article).


How to stop evaluation phase in reinforcement learning with epsilon-greedy Monte Carlo agent? I have implemented an epsilon-greedy Monte Carlo reinforcement learning agent like suggested in Sutton and Barto's RL book (page 101). As far as I understood epsilon-greedy agents so far, the evaluation has to stop at some point to exploit the gained knowledge.

I do not understand, how to stop the evaluation here, because the policy update is linked to epsilon. So just setting epsilon equal to zero at some point does not seem to make sense to me.

I do not understand, how to stop the evaluation here, because the policy update is linked to epsilon. So just setting epsilon equal to zero at some point does not seem to make sense to me.

If by evaluation you mean the act of exploring new paths, it does not need to stop by changing epsilon to 0 instantly. Instead, in order to facilitate the convergence of the algorithm, the epsilon can be progressively decreased until it reaches 0.

I do not think your question is meant to be related with monte carlo specifically, but if this was not what you wanted to know, please comment.

  But would not a epsilon value of 0 lead to an unintended policy update?

No, a value of epsilon equal to 0 will make you always choose your action according to the policy. I think what is confusing is that the "update" they make to the policy on the last for, is not an update. It is a statement saying the probability of taking such action. What it means is the following:

For action $a$, the probability of taking $a$ in state $S_t$, when $\epsilon$ is 0 is:

1 if action $a$ is the action with the best value
0 if action  $a$ is not the action with the best value

What this means is that when epsilon is 0, the action taken will always be the greedy action.


How can I suppress a CNN’s translation invariant or translation equivariant? I am trying to understand this post, but I get confused by the definitions and the differences. What's definition of equivariant?

If I remove all the pooling layers from a CNN, will it make the network to detect features in pixel resolution? For example, detecting the local maximum of a pixel. For example, can a CNN be designed to return True for the following case?

And False for the shifted window:

In the second case it returns false because the 3x3 submatrix isn't centered (yellow dash line) around the local maximum.

Will an architecture that is

from keras.layers import Dense, Conv2D, Flatten
model = Sequential()
model.add(Conv2D(128, kernel_size=2, activation=’relu’, padding='same', input_shape=(3,3,1)))
model.add(Conv2D(64, kernel_size=2, activation=’relu’, padding='same'))
model.add(Flatten())
model.add(Dense(10, activation=’softmax’))

be able to differentiate between the tiling of the larger grayscale image?

Question 1

To make it simple

You have a transformation $T$ and an operator $C$ acting on a given input $x$
Let's say you do this experiment

compute $y_{1} = C(T(x))$
compute $y_{2} = C(x)$

You can get three different results:

$y_{1} = y_{2}$ then you can say the operator is invariant with respect to the given transformation
$y_{1} = T(y_{2})$ then you can say the operator is equivariant to the given transformation as applying it to the input basically reflects its effect completely on the output
none of the 2

Questions 2

Q2.1

Spatial Pooling is not the only way to perform dimensionality reduction. You can achieve it even simply applying the Conv Kernel with no padding.

For example, let's say your input is a WxHxC tensor and you are applying a kernel which is KxKxC the result will have spatial domain size (W-(K/2))x(H-(K/2)) with K/2 the integer truncation so if K=5 then K/2=2.

Alternatively you can reduce the spatial domain with stride.

Q2.2

It seems to me you are talking about a sort of Non Max Suppression Operator rather an emergent behaviour, but certainly with the proper supervision signal you can train a CNN to do this work (even if practically it does not make sense as you can explicitly define it).


Can deep reinforcement learning algorithms be deterministic in their reproducibility in results? I ran a deep q learning algorithm (DQN) for $x$ number of epochs and got policy $\pi_1$.
I reran the same script for the same $x$ number of epochs and got policy $\pi_2$. I expected $\pi_1 $ and $\pi_2$ to be similar because i ran the same script. However, when computing the actions on the same test set, i realised the actions were very different.

Is this supposed to be normal when training deep q networks or is there something that I am missing ?

I am using prioritised experience replay when training the model.

Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?

Yes, but only if you control all places in the code where stochastic methods are used (typically by seeding the affected RNGs):

Neural network weight initialisation
Action choice for $\epsilon$-greedy or other behaviour policy (does not apply in your case, because you work exclusively from experience replay)
Minibatch sampling from experience replay
Stochastic choices in the environment (does not apply in your case)
Other stochastic parts of training that may be in use, such as dropout regularisation

Controlling all these should make your training process deterministic and repeatable. It won't necessarily make it correct.

  I reran the same script for the same $x$ number of epochs and got policy $\pi_2$. I expected $\pi_1 $ and $\pi_2$ to be similar because i ran the same script.

This is subtly different. It seems you hoped that convergence of the algorithm would mean you got to the same approximately optimal policy. In principle this is possible, because Q-learning should find a deterministic policy. However, there are some details to bear in mind:

Many environments support multiple equivalent optimal policies. A simple grid world can have multiple equivalent paths from start to goal states. A Q-learning with approximation function will slightly prefer one or other path, resulting in very different, but still optimal, policies.
Q-learning with approximation can go wrong and learn incorrectly. The usual checks and balances against this are running large numbers of simulations and testing.

You don't have great options here, from your comments you are training purely offline from historic data. Your one sanity check - do I get the same policy if I re-try - has shown inconsistency. However, it doesn't necessarily mean you have a problem, perhaps the two policies are equivalent.

Here are a couple of additional tests that may help:

Instead of looking at the maximising action choice in the test data, look at how each Q function scores the behaviour policy action choice. If the scores are close (by some measure such as MSE), then the two Q-learners are basically agreeing and are more likely to have equivalent but different policies, as opposed to radically different end results.
Have each Q network score the other's Q function action choice over an arbitrary (but realistic) set of states. If the values are similar to each other, then again this points to successful convergence given the training data, but with different outcomes due to small details.

If either of these checks shows the networks are radically different, then you have a problem. Which run, if any, has found a viable policy, and which has failed?

Even if the checks agree, it is circumstantial evidence that the Q learning process is stable, not proof that you have an agent that is better than the prevailing behaviour policy in your real world system.

You won't know if the agent is truly better, unless you can find a more independent way to assess the agent.