What is the bias-variance trade-off in reinforcement learning? I am watching DeepMind's video lecture series on reinforcement learning, and when I was watching the video of model-free RL, the instructor said the Monte Carlo methods have less bias than temporal-difference methods. I understood the reasoning behind that, but I wanted to know what one means when they refer to bias-variance tradeoff in RL.
Is bias-variance trade-off used in the same way as in machine learning or deep learning?
(I am just a beginner and have just started learning RL, so I apologize if it is a silly question.)

The bias-variance trade-off that you're referring to has to do with the return estimator. Any RL algorithm you choose needs some estimate of the cumulative return, which is a random variable with many sources of randomness, such as stochastic transitions or rewards. Monte Carlo RL algorithms estimate returns by running full trajectories and literally averaging the return achieved for each state. This imposes very few assumptions on the system (in fact, you don't even need the markovian property for these methods), so bias is low. However variance is high since each estimate depends on the literal trajectories that you observe. As such, you'll need many, many trajectories to get a good estimate of the value function.
On the other hand, with TD methods, you estimate returns as $R_t + \gamma V(S_{t+1})$, where $V$ is your estimate of the value function. Using $V$ this imposes some bias (for instance, the initialization of the value function at the beginning of training affects your next value function estimates), with the benefit of reducing variance. In TD learning, you don't need full environment rollouts to make a return estimate, you just need one transition. This also lets you make much better use of what you've learned about the value function, because you're learning how to infer value "piecewise" rather than just via literal trajectories that you happened to witness.


Does a fully convolutional network share the same translation invariance properties we get from networks that use max-pooling? Does a fully convolutional network share the same translation invariance properties we get from networks that use max-pooling?

If not, why do they perform as well as networks which use max-pooling?

All convolutional networks (with or without max-pooling) are translation-invariant (AKA spatially invariant) because their filters slide over every position in the image. This means that if a pattern that "matches" a filter is present anywhere in the image, then at least one neuron should activate.

Max-pooling, on the other hand, has nothing to do with spatial invariance. It's simply a regularization technique to help reduce the number of parameters later in the network by downsizing activation layers within the network. This can help combat overfitting, although it's not strictly necessary. Alternatively, neural networks can achieve the same effect by using a convolutional layer with a stride of 2 instead of 1.


What would be the best way to disable a rogue AI? Suppose that an artificial superintelligence (ASI) has finally been developed, but it has rebelled against humanity. We can assume that the ASI is online and can reproduce itself through electronic devices.
How would you disable the AI in the most efficient way possible reducing damage as much as possible?

Metaphorically: make it so depressed it commits suicide.

As per my answer to this AI SE question, the idea is to feed it a sequence of inputs that will cause it to become (permanently) inactive.

The technical details of how this might be achieved (and they are somewhat technical) can be found in this paper.


Why are the nodes (or neurons) in neural networks depicted as circles? Why are the nodes (or neurons) in neural networks depicted as circles?
What is the difference between a circle and a box in diagrams of neural networks?

I guess the reason is that we call them nodes and nodes are usually depicted using circles in graph theory. We model them as graphs for both forward and backward paths.


What type of ANN architecture to choose? I have $N$ number of teachers  each of which has an input feature vector ($25$ dimensional) consisting of positive numerical values for different quality of aspects (for example: lecturing ability, knowledge capacity, communication skills, etc.). I want to design an ANN to output a single quality index based on these quality features.
What type of ANN architecture is appropriate for this problem?

It sounds like you have structured/tabular data. So, a fully-connected feedforward network should do the job.


Which model is better given their training and validation errors? Below you have the plots of the training and validation errors for two different models. Both plots show the RMSE values for the validation dataset versus the number of training epochs. It is observed that models get lower RMSE value as training progresses.

The model associated with the first plot is performing quite well. The gap is quite narrowed.

I think the model associated with this second plot is doing pretty good, but not as well as the other. The gap is much broader.

The model of the first plot was trained using a data set containing 1 million of ratings, while the second one used only 100K. I'm implementing the collaborative filtering (CF) algorithm. I am optimising it using SGD.

Are any of these models overfitting or underfitting?

I would say that your intuition is correct: the model associated with the first plot is likely to generalise more than the one associated with the second plot.

In both cases, it doesn't seem that your model has overfitted the training data. Overfitting often occurs when the training error keeps decreasing but the validation error starts to increase. In both your plots, both the training and validation errors keep decreasing (even if slowly, after a while).

Underfitting occurs when your model hasn't learned enough even about your training data. The smaller the training and validation error, the more likely your model has not underfitted, but the value of RMSE depends on the range of your inputs. See e.g. What are good RMSE values? for more info.

See also this article Overfitting and Underfitting With Machine Learning Algorithms for a general overview of the concepts of overfitting and underfitting.


Adversarial search in the game '2048' If we model the game '2048' using a max-min game tree, what is the maximal path from a start state to a terminal state? (Assume the game ends only when the board is full

This is one of the sub-questions that should prepare us to actually modeling the game as a max-min game tree. However I'm failing to understand the question.

Is it actually the path to receiving 131072 as an endgame?

To model 2048 (or any problem) for search, you need a only a few pieces of information.

Note first though, that 2048 is not suitable for minimax, because there's only one player! Instead, you can treat this as a Markov decision process. The techniques to solve it are pretty similar though. Basically, you'll do search for one player, and insert "chance" nodes at each ply of the search. The value of a chance node is the expected value of its children. Note that this will reduce the effectiveness of pruning, so it might mean the problem is not tractable for search-based approaches.

What does an end state look like? Usually you have some function G(s) that accepts a state s, and produces true if and only if it's an end state. In 2048, end states would be states where the player loses (no moves possible), or where the player wins (a 131072 tile is present), so this should be fairly easy to write.
What are the payoffs? This is usually given by a utility function U(e) that accepts an end-state e, and produces a numeric value indicating the utility the player will receive.
What actions can the player take in each state? In 2048, these are always the same (up, down, left and right)
How are new states generated from old states and player actions (in this case, the tiles slide according to the rules of the game, and then a new tile is inserted at a random empty location.

Although search might work here, since 2048 is a relatively simple MDP, you might be happier using techniques from reinforcement learning, which were specifically designed for this kind of problem. Russell & Norvig have a good set of chapters on both approaches (14-17).


Understanding the math behind using maximum likelihood for linear regression I understand both terms, linear regression and maximum likelihood, but, when it comes to the math, I am totally lost. So I am reading this article The Principle of Maximum Likelihood (by Suriyadeepan Ramamoorthy). It is really well written, but, as mentioned in the previous sentence, I don't get the math.

The joint probability distribution of $y,\theta, \sigma$ is given by (assuming $y$ is normally distributed):

This equivalent to maximizing the log likelihood:

The maxima can be then equating through  the derivative of l(θ) to zero:

I get everything until this point, but don't understand how this function is equivalent to the previous one :

Note first that the first $=$ (equals) in $\frac{dl(\theta)}{d\theta}
 = 0 = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$ should be interpreted as a "is set to", that is, we set $\frac{dl(\theta)}{d\theta}
 = 0$. Given that (apparently) $\frac{dl(\theta)}{d\theta} = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$, $\frac{dl(\theta)}{d\theta}
 = 0$ is equivalent to $0 = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$.

Now, let's apply some basic linear algebra:

\begin{align}
0 &= −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta) \iff \\
0 &= −(0−2X^TY + X^TX \theta) \iff \\
0 &= −0 + 2X^TY - X^TX \theta) \iff \\
0 &= 2X^TY - X^TX \theta \iff \\
X^TX \theta &= 2X^TY \iff \\
(X^TX)^{-1}(X^TX) \theta &= (X^TX)^{-1}2X^TY \iff \\
\theta &= (X^TX)^{-1}2X^TY
\end{align}

Now, you can ignore the $2$, because it is just a constant, and, when optimizing, this does not influence the result.

Note that using $\hat{\theta}$ instead of $\theta$ is just to indicate that what we will get is an "estimate" of the real $\theta$, because of round off errors during the computations, etc.


Algorithms for scene rotation My goal is to take an image and return another image that looks as if the scene was viewed from another angle.  The difference in angle can be small — let's say as if the hand holding the camera moved slightly sideways.

If deep learning is what you are trying to use here, you should keep in mind that the real intent behind deep learning is to learn a probability distribution, which means that if you were to use a deep learning model to "rotate" images, you can only do it on a specific class of images (e.g. faces, cats, etc...).
If that's your goal, generative models are the way to go:
Autoencoders
You can train an Autoencoders to slightly change the angle. Autoencoders are a special type of neural network that is trained to output the same input you feed into them with a few imposed restrictions to prevent it from learning a trivial identity function. In your case, you could use a variation of a de-noising autoencoder. A de-noising autoencoder, as the name suggests, generates the same input image minus an artificial stochastic noise. The way this is achieved is by feeding a corrupted version of the image and then evaluating the loss on the non-corrupted version.
How can this be adapted in your case?
In your case, you could feed the original images into your autoencoder and evaluate them based on the rotated images. This will result in your autoencoder effectively learning the inner distribution that generates the images in order to generate a slightly "rotated" version of it. For more info on de-noising autoencoders, see the original paper.
Generative adversarial networks
For a more sophisticated approach, you can use generative adversarial networks. GANs are relatively harder to manipulate, but usually perform better than other generative models when it comes to images.
How can this be adapted in your case?
In general, GANs generate images from noise. However, in your case, you can use the original (non rotated) images as input for the generator. The generator can be a convolutional autoencoder for example. And the "real images" dataset will be your rotated images. This way, your model will learn to generate slightly rotated images by being fed an equivalent of noise in traditional GANs which in your case will be the original images. For more info on GANs, I suggest this and the original paper.
I should point out that one of the flaws of GANs is distorted perspectives so it's probably a shot in the dark; however, I think it won't be a problem here because you would be using real images as input instead of complete noise.
Related work
Now, as far as the literature goes, I don't think this has been done before except for this one (kinda) for faces representation learning. The only way this is similar to yours is that you can modify the implementation to only generate the faces it has been fed from different perspectives instead of an average of everything it has learned.


What is the surrogate loss function in imitation learning, and how is it different from the true cost? I've been reading A Reduction of Imitation Learning and Structured Prediction
to No-Regret Online Learning lately, and I can't understand what they mean by the surrogate loss function.
Some relevant notation from the paper -

$d_\pi$ = average distribution of states if we follow policy $\pi$ for $T$ timesteps
$C(s,a)$ = the expected immediate cost of performing action a in state s for the task we are considering (assume $C$ is bounded in [0,1]
$C_\pi(s) = \mathbb{E}_{a\sim\pi(s)}[C(s,a)]$ is the expected immediate cost of $π$ in $s$.
$J(π) = T\mathbb{E}_{s\sim d_\pi}[C_\pi(s)]$ is the total cost of executing policy $\pi$ for $T$ timesteps

In imitation learning, we may not necessarily know or observe true costs $C(s,a)$ for the particular task. Instead, we observe expert demonstrations and seek to bound $J(π)$
for any cost function $C$ based on how well $π$ mimics the expert’s policy $π^{*}$. Denote $l$ the observed surrogate loss function we minimize instead of $C$. For instance, $l(s,π)$ may be the expected 0-1 loss of $π$ with respect to $π^{*}$ in state $s$, or a squared/hinge loss of $π$ with respect to $π^{*}$ in $s$. Importantly, in many instances, $C$ and $l$ may be the same function – for instance, if we are interested in optimizing the learner’s ability to predict the actions chosen by an expert.

I don't understand how exactly the surrogate loss is different from the true costs, and what are the possible cases in which both are the same. It'd be great if someone could throw some light on this. Thank you!

A surrogate loss is a loss than you use "instead of", "in place of", "as a proxy for" or "as a substitute for" another loss, which is typically the "true" loss.
Surrogate losses are actually common in machine learning (although almost nobody realizes that they are surrogate losses). For example, the empirical risk (which the mean squared error is an instance of) is a surrogate for the expected risk, which is incomputable in almost all cases, given that you do not know the underlying probability distribution. See An Overview of Statistical Learning Theory by V. N. Vapnik for more details. In fact, discussions on generalization arise because of this issue, i.e. you use surrogate losses rather than true losses.
The term "surrogate" is also used in conjunction with the term "model", i.e. "surrogate model", for example, in the context of Bayesian optimization, where a Gaussian process is the surrogate model for the unknown model/function that you want to know about, i.e. you use the Gaussian process to approximate the unknown function/model.
Regarding the excerpt you are quoting and your specific concerns, although I didn't read the paper and I am not an expert in imitation learning, let me try to explain what I understand from this excerpt. Essentially, in imitation learning, you use the expert's policy $\pi^*$ to train the agent, rather than letting him just explore and exploit the environment. So, what you know is $\pi^*$ and you can calculate the "loss" between $\pi^*$ and $\pi$ (the current agent's policy), denoted by $l$. However, this loss $l$ that you calculate is not necessarily the "true" loss (i.e. it is a surrogate loss), given that our goal is not really to imitate the "expert" but to learn an optimal policy to behave in the environment. If the goal was to just imitate the "expert", then $C$ and $l$ would coincide, because, in that case, $l$ would represent the "discrepancy" or "loss" between $\pi$ and the expert's policy $\pi^*$.


Could an AI be built to learn based of interaction with a human? A neural network is usually programmed to learn from datasets to solve a specific problem. Essentially, they perform non-linear regression.

Could a neural network be programmed to receive input from a human, like a terminal, to begin to grow and learn (similar to how a child learns)?

A program that neither knows it's purpose nor specific data sets but is given enough information to learn based off of input, ponder the input, and ask questions. A child discovers their purpose (in destiny based philosophy) through experience. Thus, could an AI be created that would learn its purpose over time?

Grow both by continued development, maybe adding extensions that add image recognition, speech analysis, etc., and through user interaction. Eventually learning "moral imperatives" or simple the do's and don't's and how to interact with data.

A case scenario would be a Question & Answer session with the neural network and a large data set. Where the human operator knows the answers. At first, the question and the answer are supplied to the neural network. Giving it the ability to find the answer supplied through deep learning. A guaranteed confidence score of (1)  - as the question is pondered the closer it gets to the answer the more it "learns".

The next step is supplying the question and waiting for the answer. The human still knows these answers but is testing the "learning machine" to see if it is truly learning and not "repeating the answer". The answer is supplied by the machine and the human returns with either a percentage that the machine is right (hopefully and eventually matching its confidence score). and after an amount of failure provides the right answer to the machine to repeat the first step and improve learning.

The last step is being able to have the machine answer the question with the human not knowing the solution, thus completing the learning cycle. The human would test the solution and report the results to the machine and the machine would adapt the process and continue learning. However, this time it would begin learning from a data set of results. Hopefully learning "data mining" during its question and answer session.

In principle, yes, what you are proposing can be done.  The exact details of how to do it are an open research question.  The details would also depend on exactly what your goals for the system are. If you're just trying to build some domain specific system that learns a very specific kind of knowledge, then that's probably going to be easier than building an AGI that learns like a child does.

What I will suggest, although this is not proven, is that building a really powerful system of this sort will probably require more than deep learning.  I would also caution anybody interested in AI against thinking that deep learning is the "be all, end all" of AI techniques. My guess is that doing this well will ultimately require a multi-agent system, maybe something like Minsky's "Society of Mind" approach, or a Blackboard model, with collaborating agents, each specialized for various aspects of intelligence.  My feeling is that you will, indeed, need deep learning for classification / pattern matching, but possibly also things like Case Based Reasoning, K-Lines, a Semantic Network, Rule Learning, BDI, and other techniques, working together.


How to construct input dependent convolutional filter? I am constructing a convolutional variational autoencoder for images, starting out with mnist digits. Typically I would specify convolutional layers in the following way:
input_img = layers.Input(shape=(28,28,1))
conv1 = keras.layers.Conv2D(32, (3,3), strides=2, padding='same', activation='relu')(input_img)
conv2 = keras.layers.Conv2D(64, (3,3), strides=2, padding='same', activation='relu')(conv1)
...

However, I would also like to construct a convolutional filter/kernel that is fixed BUT dependent on some content related to the input, which we can call an auxiliary label. This could be a class label or some other piece of relevant information corresponding to the input. For example, for MNIST I can use the class label as auxiliary information and map the digit to a (3,3) kernel and essentially generate a distinct kernel for each digit. This specific filter/kernel is not learned through the network so it is fixed, but it is class dependent. This filter will then be concatenated with the traditional convolutional filters shown above.
input_img = layers.Input(shape=(28,28,1))
conv1 = keras.layers.Conv2D(32, (3,3), strides=2, padding='same', activation='relu')(input_img)

# TODO: add a filter/kernel that is fixed (not learned by model) but is class label specific
# Not sure how to implement this?
# auxiliary_conv = keras.layers.Conv2D(1, (3,3), strides=2, padding='same', activation='relu')(input_img)

I know there are kernel initializers to specify initial weights https://keras.io/api/layers/initializers/, but I'm not sure if this is relevant and if so, how to make this work with a class specific initialization.
In summary, I want a portion of the model's weights to be input content dependent so that some of the trained model's weights vary based on the auxiliary information such as class label, instead of being completely fixed regardless of the input. Is this even possible to achieve in Keras/Tensorflow? I would appreciate any suggestions or examples to get started with implementation.

Not a tensorflow expert but I may be able to offer some conceptual advice. Since you do not care to learn the filter, but instead want to fix a discrete set of possible values for a discrete set of cases, you can use tensor operations (i.e convolutions) rather than neural network layer operations. Essentially, in framework-agnostic pseudocode this would look like:
# layers with learned parameters
output1 = layers1(input)

# apply unlearned but changeable layer convolutions
kernel_val = kernel_val_selection_function(output1)
output2 = convolve_2D(output1,kernel_val)

# more layers with learned parameters
output3 = layers3(output2)

...

The function graph will treat kernel_val as a constant for purposes of backpropagation, so as long as your convolution operations are done within the framework used to create the function graph (i.e. tensorflow) you shouldn't have any problems with backprop.


Book recommendations on deep learning (convolutional neural networks) I am working on software which deblurs the motion blur created by camera movement.
 I've surveyed some research papers and determined this process requires deep learning and CNN. Now I'm looking for some books that would be useful in getting a more complete picture of the process.

If you are new to neural networks, I recommend studying the free online book Neural Networks and Deep Learning. It teaches you the basic concepts and the underlying math - a great starting point to dig deeper.

Once you understand the basic concepts, I recommend watching the Stanford Lecture CS231n about Convolutional Neural Networks for Visual Recognition. It consists of 16 lectures and teaches you most of the things you'll need to know for your project. The also provide all the slides - check out the links below each video.

Analyzing existing projects and their documentation is a good next step for deeper understanding of what's state of the art in CNN. You could start with this article about Q-Learning based on a CNN architecture, but there are many more out there.

More specific questions will come during your project. You'll find more information in scientific papers and blogs from researchers. Feel free to ask here as well, if you don't find what you are looking for elsewhere.


Recognize pattern in dataset I'm currently working on a group project where we need to find a pattern in a given dataset. The dataset is a collection of X, Y, Z values of a gyroscope from someone who is walking. If you plot these values you'll get a result like this.

And this is how our dataset looks like.

We are new to AI and ML so we first did some general research like understanding how matrices work and how to do some basic predictions with frameworks like TensorFlow and PyTorch. Now we want to start on this problem. What we need to do is to find a pattern in the dataset and count how many times this pattern appears in this dataset.

We started of with some none AI functions to count, we managed to do that but the way we counted will probably only work on this specific dataset. So that's why we decided to do this with AI.

We would love to hear as many different approaches as possible to count the steps since we are still learning.

For such time-series data that has a significant amount of periodicity, I would recommend converting data to the frequency domain and performing various spectral analysis methods as @firion has already mentioned. For example, you could perform Fourier Analysis and study the individual components and identify patterns there.

Also, it generally not recommended to perform the normal pattern extraction approaches to time-series data as they fail to understand the temporal relationship between subsequent data points.

Hope this helps!


Why doesn't Q-learning converge when using function approximation? The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions (the Robbins-Monro conditions) regarding the learning rate are satisfied

$\sum_{t} \alpha_t(s, a) = \infty$
$\sum_{t} \alpha_t^2(s, a) < \infty$

where $\alpha_t(s, a)$ means the learning rate used when updating the $Q$ value associated with state $s$ and action $a$ at time time step $t$, where $0 \leq  \alpha_t(s, a) < 1$ is assumed to be true, for all states $s$ and actions $a$.

Apparently, given that $0 \leq  \alpha_t(s, a) < 1$, in order for the two conditions to be true, all state-action pairs must be visited infinitely often: this is also stated in the book Reinforcement Learning: An Introduction, apart from the fact that this should be widely known and it is the rationale behind the usage of the $\epsilon$-greedy policy (or similar policies) during training.

A complete proof that shows that $Q$-learning finds the optimal $Q$ function can be found in the paper Convergence of Q-learning: A Simple Proof (by Francisco S. Melo). He uses concepts like contraction mapping in order to define the optimal $Q$ function (see also What is the Bellman operator in reinforcement learning?), which is a fixed point of this contraction operator. He also uses a theorem (n. 2) regarding the random process that converges to $0$, given a few assumptions. (The proof might not be easy to follow if you are not a math guy.)

If a neural network is used to represent the $Q$ function, do the convergence guarantees of $Q$-learning still hold? Why does (or not) Q-learning converge when using function approximation? Is there a formal proof of such non-convergence of $Q$-learning using function approximation?

I am looking for different types of answers, from those that give just the intuition behind the non-convergence of $Q$-learning when using function approximation to those that provide a formal proof (or a link to a paper with a formal proof).

Here's an intuitive description answer:

Function approximation can be done with any parameterizable function. Consider the problem of a $Q(s,a)$ space where $s$ is the positive reals, $a$ is $0$ or $1$, and the true Q-function is $Q(s, 0) = s^2$, and $Q(s, 1)= 2s^2$, for all states. If your function approximator is $Q(s, a) = m*s + n*a + b$, there exists no parameters which can accurately represent the true $Q$ function (we're trying to fit a line to a quadratic function). Consequently, even if you chose a good learning rate, and visit all states infinitely often, your approximation function will never converge to the true $Q$ function.

And here's a bit more detail:

Neural networks approximate functions. A function can be approximated to greater or lesser degrees by using more or less complex polynomials to approximate it. If you're familiar with Taylor Series approximation, this idea should seem pretty natural. If not, think about a function like a sine-wave over the interval [0-$\pi/2$). You can approximate it (badly) with a straight line. You can approximate it better with a quadratic curve. By increasing the degree of the polynomial we use to approximate the curve, we can get something that fits the curve more and more closely.
Neural networks are universal function approximators. This means that, if you have a function, you can also make a neural network that is deep or wide enough that it can approximate the function you have created to an arbitrarily precise degree. However, any specific network topology you pick will be unable to learn all functions, unless it is infinitely wide or infinitely deep. This is analogous to how, if you pick the right parameters, a line can fit any two points, but not any 3 points. If you pick a network that is of a certain finite width or depth, I can always construct a function that needs a few more neurons to fit properly.
Q-learning's bounds hold only when the representation of the Q-function is exact. To see why, suppose that you chose to approximate your Q-function with a linear interpolation. If the true function can take any shape at all, then clearly the error in our interpolation can be made unboundedly large simply by constructing a XOR-like Q-function function, and no amount of extra time or data will allow us to reduce this error. If you use a function approximator, and the true function you try to fit is not something that the function can approximate arbitrarily well, then your model will not converge properly, even with a well-chosen learning rate and exploration rate. Using the terminology of computational learning theory, we might say that the convergence proofs for Q-learning have implicitly assumed that the true Q-function is a member of the hypothesis space from which you will select your model.


How are vectors and matrices multiplied in supervised machine learning? I've recently started reading a book about deep learning. The book is titled "Grokking Deep Learning" (by Andrew W Trask). In chapter 3 (pages 44 and 45), it talks about multiplying vectors using dot product and element-wise multiplication. For instance, taking 3 scalar inputs (vector) and 3 vector weights (matrix) and multiplying.

From my understanding, when multiplying vectors the size needs to be identical. The concept I have a hard time understanding is multiplying vectors by a matrix. The book gives an example of an 1x4 vector being multiplied by 4x3 matrix. The output is an 1x3 vector. I'm am confused because I assumed multiplying vector by matrix needs the same number of columns, but I have read that the matrices need rows equal to the vectors columns.

If I do not have an equal number of columns, how does my deep learning algorithm multiply each input in my vector by a corresponding weight?

In general, when people do not explicitly state it, a vector $v \in \mathbb{R}^n$ is usually considered a "column vector", that is, you can think of it as the matrix $v \in \mathbb{R}^{n \times 1}$ (that is, a matrix with $n$ rows and $1$ column).
If it is not explicitly stated and you assume that the given vector is a column vector, but the dimensions do not match, then you should check that the dimensions match if you consider the given vector as a "row vector" (because it might be the case that the author is implicitly considering the vectors as row vectors).

Having said that, you can multiply a vector $v \in \mathbb{R}^n$ by a matrix $A \in \mathbb{R}^{n \times m}$ from the left, that is, you can do $v^T A$. Here, I considered $v$ has a column vector (that is, $v \in \mathbb{R}^{n \times 1}$), even though I have not explicitly stated it: you can and you often need to deduce this from the context! If I transpose $v$, I obtain $v^T \in \mathbb{R}^{1 \times n}$, and, thus, you can see that you can indeed perform the operation $v^T A = u \in \mathbb{R}^{1 \times m}$. Note that, at this point, $u$, the vector resulting from the operation $v^T A$, is actually considered a matrix, but you can still use it as a vector, if you need and that is permitted according to the mathematical rules of the operations you need to perform. Note also that I cannot multiply $v$ from the right of $A$, because there is no way of making the dimensions match. Have a look at this question, if you do not know how to multiply a vector by a matrix from the left.

Similarly, you can multiply $v \in \mathbb{R}^n$ by the matrix $B \in \mathbb{R}^{m \times n}$ only from the right. If $v$ is a column vector (that is, $v \in \mathbb{R}^{n \times 1}$), you need to do $B v \in \mathbb{R}^{m \times 1}$, but, if $v$ is a row vector (that is, $v \in \mathbb{R}^{1 \times n}$), you will first need to transpose it, so that you can perform the operation: $B v^T\in \mathbb{R}^{m \times 1}$.

Furthermore, note that, if you multiply a vector by a matrix from the left, that vector needs to be a "row vector", so, if you initially assume that the vector is a column vector (or that is explicitly stated), you will need to transpose it first, before the multiplication. However, if the vector is already a row vector, you won't have to transpose it. Similarly, if you multiply a vector by a matrix from the right, you will need a column vector.

To conclude, you can multiply a vector either from the left or right of a matrix, but you need to make sure that the dimensions match: if you multiply from the left, you will need to check that the dimensions of the vector match the number of the rows of the matrix; if you multiply the vector from the right of the matrix, you will need to check that the dimensions of the vector match the number of columns of the matrix.


How to keep track of the same person detected in different frames of a camera? At this moment, I am able to use NN to identify an object, such as a human, when given a frame from the camera. Once locate the object, then I can feed the human object image to either NN that's designed to classify male or female.
Let's say I get 1 frame per second from the camera and perform detection, the objective is to track the number of male and female walk pass the camera within the given hours.
My question is, the same person in multiple frames will be overcounted. I couldn't wrap my head around how can I train a NN to understand that this is the same person without dive into facial recognition? I'm sure there is some tracking technique that I just don't know.
One little constraint, if the person left the camera frame and come back into it later, it is fine to treat it as two people.

One way to solve the issue may be a tracker (like Kalman). It would be faster and easier approach than neural nets.

If you insist about solving this issue through neural nets, then some magic and creativity is needed. Based on the nature of tracking, you need to feed multiple frames to predict next location of the object and check if there is any object close by. So, you may consider combining RNNs with CNNs to predict next bounding box to track and replace Kalman filter's prediction with RNNs. (check)


How do I use a taxonomy and the support vector machine for question classification? I am going to develop an open-domain natural language question-answering (NLQA) system, and will use the support vector machine (SVM) as the machine learning (ML) model for question classification.
The data that I have is from a cube, containing multiple dimensions, of which some contain hierarchies.
I do not understand how to work/combine the taxonomy and SVM for question classification. If I understand correctly, the taxonomy still needs to be developed by hand, unless an existing one is being used. And the SVM sorts the queried NL question based on this taxonomy?
Is this correct, or am I mixing the whole concept?

This is not an answer (I don't have enough reputation to comment). I did something close to this in my master's thesis and think it is close to what you are interested in.

In it, I had developed a framework for extracting metadata from web-based educational content. This metadata was used for classifying the the educaitonal content for many different attributes, which could then be used for faster and more efficient search and discovery of educational content.

The educational resources (containing the content) could be anything like text or PDFs of assignments, homework, assignments, online books, exam questions, courses etc (which many colleges host online). To identify what kind of educational resource it is, I would parse the text and look for keywords and formatting styles (preprocessing included constructing 2-grams and 3-grams, POS tagging, using small specific parsers for NER, dates and other text entities one encounters in educational content).

For some part I used Wordnet (also available under python-nltk) to obtain relationships between different entities and also to find closeness between them. DBpedia was also used. However, for the most part I had to identify the most commonly occuring terms and build a taxonomy by hand. (It took a lot of time!). I obtained a lot of candidates for keywords by looking at openly available taxonomies.

For extracting domain specific taxonomy/ontology, one needs manually to build it. Ontology generation from text is an active area of research and building domain-specific ontology has been tried for many years. One example of such taxonomy (here thesaurus) is agrovoc where domain experts have contributed to the knowledge by identifying agricultural entities manually.

There are a lot of places where domain specific vocabulary is available; maybe you can use that. In some aspects it is close to supervised machine learning, where one has some nice data and correspondingly nice output. However, on my part, there wasn't much learning in it - more like template matching.

Hope this helps.


Is it possible to train one part of the network with a particular learning rate and the other part with a different one? I have a combined network consisting of two parts: one is for images and the other is for numerical data. Each sample is matched with a numerical case by an ID. For this combined network, a lr of 0.01 was found to be best working via hyperparameter tunings
However, when I trained them as a separate task (binary classification), a lr of 0.001 for images and 0.01 for numerical data were best.
As for an AUC metric, the combined network (0.818) is performing on average of image and numerical networks (0.799 and 0.821, respectively).
Here I thought maybe the combined network's lr is too high for image part and should apply lower lr for that part. But, I don't know if it is possible.
If anyone has any idea of what is what, let me know

Create two different optimizers and split the subnets' parameters into either with different lrs. You will have to call optimizer1.step(), optimizer2.step() with a single backward() call


What is the most time-consuming part of training deep networks? Deep networks notoriously take a long time to train.

What is the most time-consuming aspect of training them? Is it the matrix multiplications? Is it the forward pass? Is it some component of the backward pass?

Forward pass

The output of a layer can be calculated given the output of the previous layer. So the GPU can parallelize this computation for every layer and over the minibatch which is done by calculating a big matrix. But it needs to be sequential from layer to layer (earlier layers to higher layers). Regarding the layer type convolutions or especially fully connected layers can result in a big matrix calculation.

Backward pass

The gradient of a layer with respect to the layer input (and layer parameters) can only be calculated given the gradient of the layer output (input gradient of a subsequent layer) and input to the layer (output of the previous layer). This again can be parallelized over a layer and minibatch but is sequential from higher layers to earlier layers. Moreover, since the backward pass relies on the outputs of the forward pass all intermediate layer outputs of the forward pass have to be cached for the backward pass which results in a high (GPU) memory usage.

Forward and backward pass take most of the time

So, these two steps take a long time for 1 training iteration, and (depending on your network) high GPU memory usage. But you should read and understand the backpropagation algorithm that basically explains everything.

Moreover, to train a network from scratch, in general, takes lots of iterations because especially in the earlier layers training the parameters is based on gradients that are affected by lots of previous layers, which can result in noisy updates, etc., that do not always push the network parameters in the right direction directly. In contrast, e.g. fine-tuning a pre-trained network on some new task can for example already be done with much less training iterations.


Training data for autoencoders Two practical questions regarding the use of autoencoders. For the size of the NN, let's say I have 50 inputs, so layer sizes go like 50-25-12-6-2-6-12-25-50

how much data I generally need to train it and how does it depend on the size of net?
I'd like to weigh samples differently - e.g. there is a time component to the problem the last samples are most recent, so should weigh more. Is there an easy way to incorporate it?

Each network has a limit that it can learn. To check that limit, you need to try. Especially those kind of simple networks, it can be achieved so quickly. Idea is that, you need to give X data and check the accuracy and increase it to 2X data and check the accuracy. If there is no change in accuracy, it means that your network has reached its limit and you may consider making it deeper and wider.

By nature of the training, the last sample that you bring to training is the result that optimizer will try to converge. Hence, optimizers are already doing what you are searching for. However, if you want to create some weights for them, try considering RNNs. Their nature is suitable for that purpose.


Monte Carlo updates on policy gradient with no terminal state Consider some MDP with no terminal state. We can apply bootstrapping methods (like TD(0)) to learn in these cases no problem, but in policy gradient algorithms that have only a simple monte carlo update, it requires us to supply a complete trajectory (which is impossible with no terminal state).

Naturally, one might let the MDP run for 1000 periods, and then terminate as an approximation. If we feed these trajectories into a monte carlo update, I imagine that samples for time period t=1,2,...,100 would give very good estimates for the value function due to the discount factor. However, the time periods 997, 998, 999, 1000, we'd have an expected value for those trajectories far different than V(s) due to their proximity to the cutoff of 1000.

The question is this:

Should we even include these later-occurring data points when we update our function approximation?

OR

Is the assumption that these points become really sparse in our updates, so they won't have much effect in our training?

OR

Is it usually implied that the final data reward in the trajectory is bootstrapped in these cases (i.e., we have some TD(0)-like behavior in this case)?

OR

Are monte carlo updates for policy gradient algorithms even appropriate for non-terminating MDPs due to this issue?

Naturally, one might let the MDP run for 1000 periods, and then terminate as an approximation. If we feed these trajectories into a monte carlo update, I imagine that samples for time period t=1,2,...,100 would give very good estimates for the value function due to the discount factor. However, the time periods 997, 998, 999, 1000, we'd have an expected value for those trajectories far different than V(s) due to their proximity to the cutoff of 1000.

I think your intuition is... partially right here, but not entirely precise. Recall that a value function $V(S)$ is generally defined as something like (omitting some unimportant details like specifying the policy):

$$V(S_i) = \mathbb{E} \left[ \sum_{t=0}^{\infty} \gamma^t R_{i+t} \right].$$

The "samples for time period $t = 1, 2, \dots, 100$ that you mention are not estimates for this full value function. They're estimates for the corresponding individual terms $R_{i+t}$. Indeed, in general, your intuition is right that the closer they are to the "starting point" $i$, the more likely they'll be to be accurate estimators. This is because larger $t$ are typically associated with larger numbers of stochastic state-transitions and stochastic decision-making, and therefore often exhibit higher variance.

  Should we even include these later-occurring data points when we update our function approximation?

Theoretically, you absolutely should. Suppose you have an environment where almost every reward is equal to $0$, and only after 1000 steps do you actually observe a non-zero reward. If you don't include this, you'll learn nothing! In practice, it can often be a good idea to give them less importance though. This already happens automatically by picking a discount factor $\gamma < 1$.

  Is it usually implied that the final data reward in the trajectory is bootstrapped in these cases (i.e., we have some TD(0)-like behavior in this case)?

  OR

  Are monte carlo updates for policy gradient algorithms even appropriate for non-terminating MDPs due to this issue?

It would be possible to do some form of bootstrapping at the end yeah, cut off and then have a trained value function predicting what the remainder of the rewards would be. TD($\lambda$) with $\lambda$ close to $1$ would be much closer in behaviour to true MC updates than TD($0$) though. Either way, it would be technically incorrect to still call it Monte-Carlo then, it would no longer be pure Monte-Carlo. So yes, strict Monte-Carlo updates in the purest sense of the term are not really applicable to infinite episodes.


Should I choose a model with the smallest loss or highest accuracy? I have two Machine Learning models (I use LSTM) that have a different result on the validation set (~100 samples data):

Model A: Accuracy: ~91%, Loss: ~0.01
Model B: Accuracy: ~83%, Loss: ~0.003

The size and the speed of both models are almost the same. So, which model should I choose?

You should choose the model A. The loss is just a differentiable proxy for accuracy.

That said, the situation should be examined in more detail. If the higher loss is due to the data term, examine the data which produce high loss and check for presence of overfitting or incorrect labels.

If the higher loss is due to a regularizer then reducing the regularization factor may further improve the results.


How can auto-encoders compute the reconstruction error for the new data? Autoencoders are used for unsupervised anomaly detection by first learning the features of the data set with mainly "normal" data points. Then new data can be considered anomalous if the new data has a large reconstruction error, i.e. it was hard to fit the features as in the normal data.
Even if the training is supervised by learning to reconstruct the same data, how is the reconstruction error computed for the new data?

It is computed just like in training. You take an MSE or something along these lines between the input and the output. You set a threshold for it. If new data's reconstruction error is higher than your threshold, then it is anomalous otherwise it isn't.


Can neural networks be better than human experts at prediction of greyhound racing results? Were there any studies which checked the accuracy of neural network predictions of greyhound racing results, compared to a human expert? Would it achieve a better payoff?

Practically, no.

In greyhound racing (or horse racing), there is no definite underlying pattern that can be associated with the outcome of the race. There are far too many variables to record and code as features. Most of which cannot be accessed by the public. This includes eating, sleeping, and training patterns. Furthermore, there are variables that cannot be readily quantified, such as the trainer's techniques, training effort, health history, and genes. A mere history of racing results and age won't be that helpful.

A neural network can only be as good as the features that are used to represent the instances. If the features don't capture the necessary characteristics of the instances that are associated with the problem, then the learner cannot generalize to predict the real world outcome.


How is the loss value calculated in order to compute the gradient? The gradient descent step is the following

\begin{align}
\mathbf{W}_i = \mathbf{W}_{i-1} - \alpha * \nabla L(\mathbf{W}_{i-1})
\end{align}

were $L(\mathbf{W}_{i-1})$ is the loss value, $\alpha$ the learning rate and $\nabla L(\mathbf{W}_{i-1})$ the gradient of the loss.

So, how do we get to the $L(\mathbf{W}_{i-1})$ to calculate the gradient of $L(\mathbf{W}_{i-1})$? As an example, we can initialize the set of $\mathbf{W}$ to 0.5. How can you explain it to me?

In your case, $L$ is the loss (or cost) function, which can be, for example, the mean squared error (MSE) or the cross-entropy, depending on the problem you want to solve. Given one training example $(\mathbf{x}_i, y_i) \in D$, where $\mathbf{x}_i \in \mathbb{R}^d$ is the input (for example, an image) and $y_i \in \mathbb{R}$ can either be a label (aka class) or a numerical value, and $D$ is your training dataset, then the MSE is defined as follows

$$L(\mathbf{W}) = \frac{1}{2} \left(f(\mathbf{x}_i) - y_i \right)^2,$$

where $f(\mathbf{x}_i) \in \mathbb{R}$ is the output of the neural network $f$ given the input $\mathbf{x}_i$.

If you have a mini-batch of $M$ training examples $\{(\mathbf{x}_i, y_i) \}_{i=1}^M$, then the loss will be an average of the MSE for each training example. For more info, have a look at this answer https://ai.stackexchange.com/a/11675/2444. The https://ai.stackexchange.com/a/8985/2444 may also be useful.

See the article Loss and Loss Functions for Training Deep Learning Neural Networks for more info regarding different losses used in deep learning and how to choose the appropriate loss for your problem.


Appropriate metric and approach for natural language generation for small sentences I am trying to create a language generation model to generate very short sentences/words, like a rapper name generator. The sentences in my dataset are anywhere between 1 word and 15 words (3-155 characters). So far, I have tried LSTM's with 1-3 layers and inputs as subwords and characters. The results so far are not that great, I am getting ~0.5 crossentropy loss and ~50% accuracy.
My inputs are like a sliding window with prepadding, (eg. (for a batch) Inputs = [[0,0,0,1], [0,0,1,2]...[n-4,..n-1]], outputs=[[0,0,1,2], ...[n-3,n-2,n-1,n]]) where 0 is padding, 1 is the start token and n is the end token. Outputs are 1 hot encoded.
The model is an embedding layer, few lstm and dropout layers, followed by time distributed dense and then a dense layer.
My doubt is, is accuracy a right metric, I am using it because at the end, I am making a classification for 4 output values. Another one is, will a transformer be suitable for this, since I want to generate small sentences, (which are nouns) and models like GPT/ Bert are more suitable for capturing dependency between long sentences.

I wouldn't say accuracy in the next word prediction is a good global metric. It would depend on the length of sentences. It's always difficult to predict the first word, cos you don't have any context. And having at least one word in the context it's easier. As long as your error rate is averaged among all predicted words, the accuracy could be higher if the lengths of sentences are higher. So the value of 0.5 doesn't tell much. The fact that you improved accuracy by 0.1 for instance on the same dataset means one method is better than another. Also, people measure perplexity and it's more sensitive to small changes of loss than accuracy. That's why I suggest you to measure perplexity. This metric also depends on lengths of sentences, that's why you should only compare 2 models trained on the same dataset.
What about GPT and Transformer, they were invented to solve one of the issues of LSTMs (they're not capable to memorize a very large context), that's why transformers would be better for long sentences, but they would also be better for short I think due to their attention mechanist, which has many useful properties.


Which government agencies oversee development of new AI? Nick Bostrom talks in his book Superintelligence about the many dangers of AI. He considers it necessary that strong security mechanisms are put in place to ensure that a machine, once it gains general intelligence far beyond human capabilities, does not destroy humanity (most likely by accident). He describes this as a very delicate process that most likely will go wrong.

Considering that new technologies often neglect the necessary precautions and  that this is highly relevant to national security, I wonder if there are already government agencies overseeing big technology companies like Deepmind. We are currently far away from an intelligence explosion or a technological singularity, but I would assume that governments want to have a foot in the door as soon as they realize and understand the dangers.

So my question is, what government agencies currently investigate and maybe even control AI development? The answer can be general or for a specific country if there is a big difference between countries.

May not be quite what you’re looking for, but nonetheless helpful, I hope. The White House a year ago commissioned a report on AI that touches briefly on policy issues.


Will training an AI still work if the input data is somewhat sparse? I'm looking at writing an AI agent for pattern recognition.

I want to be able to constantly feed new data to the AI to continuously train it as new data may have new patterns.

My problem, though, is that my input feed may break once in a while (the data comes from a remote computer) and thus some of the data will go missing. The other computer sends me real-time data so when the connection goes down, any new data while disconnected goes missing as far as the AI agent is concerned. (at this point, I'm not looking at fixing the gaps, although ultimately, reducing them is one of my goals, at this point I have to pretend it's not possible to accomplish.)

What kind of impact missing data has on a pattern recognition AI?

First, the title mentions "sparse data". Recently the expression has taken a clear meaning: The agent input is data with mostly zeros. In the question a different meaning: A "sparse data stream", where data flows and vanishes sometimes. I understand the question as: "Will training an AI still work if the training data stream breaks?"

Note the explicit "training data stream": The question suggests the agent has at least 2 inputs: Training data you want to feed "anytime", and "inference data" sent to the agent for actual recognition.

This question enters (to my eye) the realm of distributed AI and multi-agent systems, and ultimately a common issue in distributed systems.

If we cast your problem to two humans S and L communicating, when S talks to L over a reliable channel, L gets all the information. When the channel breaks, L gets nothing. Does it prevent L from living normally? It merely cuts out whatever is expected out of the conversation from S to L.

Back to your scenario, whenever the data stream is broken (S), the learning agent (L) will just be unable to learn from that data source. The impact on the pattern recognition agent is bounded to what it could have learned from the new data. The agent recognition performance remains constant while the data stream is interrupted.

Now if the learning agent is just learning, and cannot perform recognition without learning, there is an architectural or implementation issue. Continuous learning entails the agent is active (performs actual recognitions) and learns out of what it does.

Update, for clarification:

The performance remains constant is "true", but subtle. At time t some metric like precision can be 99% with respect to what the agent has seen so far. Assuming continuous learning is interrupted and new recognition requests come in, the performance has "two faces":

As long as new recognition requests are "close" to what the agent has seen so far, performance is "constant"---the agent still scores 99%.
If the request is quite different, the performance will drop. The size of the drop depends on how different the input is.

A concrete example: The agent is trained to find mushrooms with a dataset where all images are taken in the forest. Assuming learning stops, when an image of mushroom on a concrete crack comes in, the agent will probably do worse. And it would then keep doing worse on such kind of image, as long as it cannot "refresh" by learning from this experience.


Detecting abnormalities in x-rays while taking into account demographics of a patient -automated This is my first post so please forgive me for any mistakes.

I am working on an object detection algorithm that can detect abnormalities in an x-ray. As a prototype, I will be using yolov3 (more about yolo here: 'https://pjreddie.com/darknet/yolo/')
However, one radiologist mentioned that in order to produce a good result you need to take into account the demographics of the patient.
In order to do that, my neural network must take into account both text an an image. Some suggestions have been made by other people for this question. For example, someone recommended taking the result of a convolution neural network and a seperate text neural network.
Here is an image for clarification:

Image Credits: This image (https://cdn-images-1.medium.com/max/1600/1*oiLg3C3-7Ocklg9_xubRRw.jpeg) from Christopher Bonnett's article (https://blog.insightdatascience.com/classifying-e-commerce-products-based-on-images-and-text-14b3f98f899e)

For more details, please refer to above-mentioned article. It has explained how e-commerce products can be classified into various category hierarchies using both image and text data.

However, a when convolution  neural network is mention it usssualy means it is used for classification instead of detection
https://www.quora.com/What-is-the-difference-between-detection-and-classification-in-computer-vision (Link for comparison between detection and classification)

In my case, when I am using yolov3, how would it work. Would I be using yolov3 output vector which would be like this format
class, center_x, center_y, width and height

My main question is how would the overall structure of my neural network be like if I have both image and text as input while using yolov3.
Thank you for taking the time to read this.

First of all you don't need "text" input as in Christopher Bonnett's blog. Your case is more easy - demographic is table data, which can be expressed as vector of numeric values. This data should be processed - pushed through one or two fully connected layers. The trick is where, to what part of yolo to concatenate results of processing of this vector. Because it's vector data it should concatenated to fully connected layer. Where exactly should be found by experiments, but as starting point it could be concatenated to before-last (before output) fully connected layer (I think for yolo it's 4096-size layer).

Overall I'd say that is not a trivial task. It require some experience with deep learning, good understanding of yolo design, yolo algorithm and a lot of experimentation, both with architecture and hyperparameters. Probably worth solid paper. Good luck.


What's the term for death by dissolving in AI? What's the term (if such exists) for merging with AI (e.g. via neural lace) and becoming so diluted (e.g. 1:10000) that it effectively results in a death of the original self?

It's not quite "digital ascension", because that way it would still be you. What I'm thinking is, that the resulting AI with 1 part in 10000 being you, is not you anymore. The AI might have some of your values or memories or whatever, but it's not you, and you don't exist separately from it to be called you. Basically - you as you are dead; you died by dissolving in AI.

I would like to read up on this subject, but can't find anything.

I find the concept of the a Turing machine useful. In one dimension, everything is a string. All of the parts that are "not you" are merely a substrate, a medium for the program your_mind runs on top of. The you, your identity, the "metaphysical" component we think of as the mind, is a result of running the algorithm that is your_mind on the bioware of your body, or, the hardware (technically, "wetware".) So what we're really talking about is the software, and in that light I might use:

Translation

because the software is being translated to execute in a new environment, or

Migration

as in moving software from one system to another.

Philip Dick wrote a philosophical narrative, not technically sci-fi, called The Transmigration of Timothy Archer which is about identity moving between bodies. In a rare departure from his usual work about AI and the effects of a technological society on the human spirit, this book looks at the question of identity in the context of the soul, which opens up all kinds of philosophical questions surrounding the type of technology we're speculating on, particularly in relation to the self.

I value artistic insight, and Phillip K. is considered quite prescient, so perhaps

Transmigration

is most appropriate, as it carries both metaphysical and information technology meanings.

Re death by dissolution, it's worthwhile to look at the etymology of dissolve:

  late 14c. (transitive and intransitive) "to break up" (of material substances), from Latin dissolvere "to loosen up, break apart," from dis- "apart" (see dis-) + solvere "to loosen, untie," from PIE *se-lu-, from reflexive pronoun *s(w)e- (see idiom) + root *leu- "to loosen, divide, cut apart." Meaning "to disband" (an assembly) is early 15c. Related: Dissolved; dissolving.

I think you're on the right track with dilution, certainly per the modern usage, but it may be possible to get more precise.

It's not about breaking up, or washing away (except metaphorically); rather, it's about minimization as in the diminishment of the original software kernel (the self) in relation to an expanding algorithm.

  early 15c., from merger of two obsolete verbs, diminue and minish. Diminue is from Old French diminuer "make small," from Latin diminuere "break into small pieces," variant of deminuere "lessen, diminish," from de- "completely" + minuere "make small" (from PIE root *mei- (2) "small").

The Old French diminuer is apt, as is the Latin deminuere and deminuo, which also carries a meaning of "civil death" and "abatement". Abatement carries a meaning of mitigation, which can be defined fundamentally as "lessening of effect," in this case of the kernel of the original self in relation to the new aggregate.

It may be useful to think of it as a ratio 1/ℵ, with the self as the 1. Important to note the ratio is not literal--each number represent an aggregation of functions we call programs.

That's how I'd think of it mathematically, but metaphysically, Nat nails it by referencing Jung and the death of the ego.

Ego comes from the Latin noun for the self (I, me) which can also be plural (we, us).  It's also fun to note that the Latin verb of being is sum, because in English, sum has a mathematical meaning of an aggregate considered as a whole.

Because we're talking about what it means to be a person, but we also want to connote a function (a process of relative minimization) I am thinking:

Depersonalization

  (noun) the action of divesting someone or something of human characteristics or individuality.SOURCE: Google

The fun part is that there's a legal definition of person:

  In legal use, "corporate body or corporation having legal rights," 15c., short for person aggregate (c. 1400), person corporate (mid-15c.)

Which I think applies (loosely) to the aggregation of function that comprises the program we call the self, and the greater aggregation of functions that form a new self.

Because here we're talking about the loss of the original individual, depersonalization as opposed to repersonalization as a corporate entity.

Alternately, I might propose:

Deindividualization

with the definition "a loss in individual identity within a group" (also known as deindividuation).

(Charles Stross wrote about this in Accellerando, where a character chooses to be subsumed by a group intelligence as the only means of escaping a swarm of autonomous lawsuits;)


Is the derivative of the loss wrt a single scalar parameter proportional to the loss? I am wondering about the correlation between the loss and the derivative of the loss wrt a single scalar parameter, with the same sample. That means: considering a machine learning model with parameters $\theta \in R$, I want to figure out the relationship between $Loss(x)$ and $\frac{\partial Loss(x)}{\partial \theta_i}$, where $i \in \{1,2,3,...,n\}$

Intuitively, I would like to consider that they are in a positive correlation, is it right? If it is right, how can I prove it in a mathematical way?

The derivative $f'(x)$ is correlated with $f(x)$ in a certain sense. In fact, $f'(x)$ is a function of $f$, so we could even say that there's a cause-effect relationship.

The derivative at a specific point $c$ of the domain, i.e. $f'(c)$, can either be negative or positive. If $f'(c) > 0$, then $f(c)$ is increasing (with respect to an increase of $x$). If $f'(c) < 0$, then $f(c)$ decreasing  (with respect to an increase of $x$).

This can easily be seen from an example. Consider $f(x) = x^2$, then $f'(x) = 2x$. Let $c = 2$, then $f'(2) = 4$, so the function is increasing. In fact, $f(1) = 2 \leq f(2) = 4 \leq f(3) = 9$. Similarly, let $c = -1$, then $f'(-1) = -2$, so the function is decreasing. In fact, $\leq f(-2) = 4 \geq f(-1) = 1 \geq  f(0) = 0$ (note that the function is decreasing as we increase $x$!).

Consider a model with only one parameter, then the partial derivative of the loss function with respect to that parameter corresponds to the derivative of the loss function. So, the reasoning above applies to this model. What about a model with more than one parameter? The same thing happens.

If the function decreases, does its derivative also decrease? In general, no, and this can easily be seen from a plot of a function and its derivative. For example, consider a plot of a parabola and its derivative (which is a linear function).

On the left of the y-axis, the parabola is decreasing, but its derivative is increasing, while, on the right of the y-axis, the parabola is increasing and the linear function is still increasing.

This is the same thing with a loss function of an ML model and its partial derivative.


When to use RMSE as opposed to MSE and vice versa? I understand that RMSE is just the square root of MSE. Generally, as far as I have seen, people seem to use MSE as a loss function and RMSE for evaluation purposes, since it exactly gives you the error as a distance in the Euclidean space.

What could be a major difference between using MSE and RMSE when used as loss functions for training?

I'm curious because good frameworks like PyTorch, Keras, etc. don't provide RMSE loss functions out of the box. Is it some kind of standard convention? If so, why?

Also, I'm aware of the difference that MSE magnifies the errors with magnitude>1 and shrinks the errors with magnitude<1 (on a quadratic scale), which RMSE doesn't do.

Answers in the comments are decent, particularly DuttA.
DuttA gives these, approximately

ease of derivative
Don't have to worry about ~0 in denominator causing huge gradient
But to me the most important is mathematical convenience, someone
might easily make the mistake of RMSE is just equal the difference
y−y′ instead of root of mean square of y−y′. The answer this might
start depending on conventions.
In maths (Don't know the reason and might be inaccurate) we mainly
work with variances instead of standard deviation.

Here are my reasons for using the MSE instead of RMSE:

Doesn't have the sqrt operations, so it computes faster
the square root isn't easy, its Newtons method, so it could be a
dozens steps per iteration
MSE has all the information of RMSE, there is 1-to-1 mapping, so no loss
the storage of the square root typically doesn't save any memory in
IEEE-784 and compute vs. memory is big thing in complexity
tools like gradient boosted machine can "recycle" the squared error
computation for speedup and working on O(n) complexity
there is hidden scaling and regularization because many gpu hardware
elements are fundamentally 8-bit, so if you can make your code more
8-bit in its guts then you don't have as much in the back-conversion
and it runs a lot faster


Where can I find an implementation of the wake-sleep algorithm? I'm looking to build from scratch an implementation of the wake-sleep algorithm, for unsupervised learning with neural networks. I plan on doing this in Python in order to better understand how it works. In order to facilitate my task, I was wondering if anyone could point me to an existing (open-source) implementation of this concept. I'm also looking for articles or, in general, resources that could facilitate this task.

I found the following detailed and well documented Python notebook, which uses only NumPy.


Is there an AI system that automatically generates classes and methods by giving it voice commands? I want to develop (in Java) a voice plugin for Eclipse on a Mac that helps me jot down high-level classes and stub methods. For example, I would like to command it to create a class that inherits from X and add a method that returns String.

Could somebody help me point out the right material to learn to achieve that?

I don't mind using an existing solution if it exists. As far as I understand, I would have to use some Siri interface and use nltk to convert the natural text into commands. Maybe there's some chatbot library that saves me some boilerpate NLP code to directly jump on to writing grammar or selecting sentence patterns.

While you can use NLTK for analyzing and parsing the text obtained from the speech to text interface (e.g. Siri), there are higher level APIs available for this. The class of problem you are trying to solve in NLP is "intent detection".

There are several open source and commerical APIs available for this including Amazon Alexa, Google Cloud Natural Language, Azure, as well as libraries like RASA NLU, etc.

The high level flow of your program will be:

Record/receive spoken audio
Convert audio speech to text
Detect intent of the text command using an intent detection library
Use the intent to feed a script/automation that generates the code in your IDE


Why does our AI play worse at even levels of depth? We are building an AI to play a board game. Leaving aside the implementation, we noticed that it plays worse when we set an even (2,4,6,...) level of depth. We use a minimax depth-first strategy.
Do you have any ideas why it behaves like that?

Edit: for example if we set a game between an AI with 5 levels of depth and an AI with 6 levels of depth, the first one usually wins (and this is weird).

When the number of levels is odd, it means the first player can do one more extra movement on the board. As it is an extensive form game, when decide using backward induction, as the last and the first move are for the first player, so the first player can act better than the situation that the second player will make the last move.


How to detect the empty parking spots? I have some images of the empty parking as shown below.

I 'd like to use deep learning to extract the parking spots. But in the beginning,am confused whether there are several ways to do the following;

Treat each parking spot as an object and use object detection to find
the parking space.
Treat the parking spot as a few key points, instead of finding the
spot, try to find the coordinates of key points on the parking line.

Any ideas on how would I work on these two methods

Object detection is a regression of the bounding box (rectangle) around the object. In this way, the two ways you suggest are equivalent.

What I suggest you to look at is lane detection for self-driving cars - a well studied problem. It seems that this is a very relevant task (if not the same), so its solution should work for you as well.


Is there any way to draw a neural network's connections in a nice way? I've been working with neural networks and artificial intelligence for a while. What I'm trying to do right now is, from a genotype I have (a sum of sensors, neurons and actuators) draw how the neural network is (with recurrent/recursive connections being showed nicely, etc.)

What I have done now in javascript is this:

I have achieved this using SigmaJs, a Javascript drawing library, but I think it's still ugly, and what I'm looking for is a node drawing library that can achieve recursive connections in a nice way (right now, I'm drawing them with a red color as you can see on the image).

I have examined a lot of GitHub repositories and websites that can be helpful but aren't worth it since they aren't that nice.

Has anyone got an idea of what can I use, in javascript?  If not, in any other language, how can I achieve what I want?

I'm the developer of Neataptic, a Javascript neural network library. I don't know if it is exactly what you're looking for, but it has a built-in graph creator using D3 and webcola.

Basically, each connection gets an arrow. It also supports self-connections and gates.

There are some examples of images:

Play around yourself here.


What does end-to-end training mean? In simple words, what does end-to-end training mean, in the context of deep learning?

This is relevant when you have two or more neural networks serving as components to a larger architecture. Training this architecture in an end-to-end manner means simultaneously training all components (i.e. training it as a single network).

The best example I can think of are image captioning architectures. These usually comprise of two networks: a CNN whose role is to extract features from the input images and a RNN that accepts the CNN's features and generates the output captions.

You have two options for training:

First, train the CNN first for some arbitrary task (e.g. image classification) in hopes that it learns how to extract features. Then use the CNN to extract features from the input images and use those as inputs to train the RNN. This procedure trains the two components in two completely separate phases.
Treat the whole architecture as a single network and backpropagete the gradients to the CNN so that it also can be trained. This procedure trains the two components simultaneously. This is what we call end-to-end training.


How can we find find the input image which maximizes the class-probability for an ANN? Let's assume we have an ANN which takes a vector $x\in R^D$, representing an image, and classifies it over two classes. The output is a vector of probabilities $N(x)=(p(x\in C_1), p(x\in C_2))^T$ and we pick $C_1$ iff $p(x\in C_1) \geq 0.5$. Let the two classes be $C_1= \texttt{cat}$ and $C_2= \texttt{dog}$. Now imagine we want to extract this ANN's idea of ideal cat by finding $x^* = argmax_x N(x)_1$. How would we proceed? I was thinking about solving $\nabla_xN(x)_1=0$, but I don't know if this makes sense or if it is solvable.

In short, how do I compute the input which maximizes a class-probability?

In deep networks there is actually a wide variety of solutions to the problem, but if you need to find one, any easy way to do this is just through normal optimization schemes
$$\hat x = argmin_x \ L(y,x)$$
where $L(y,x)$ is your loss function. Since ANN's are generally differentiable you can optimize this iteratively with some form gradient descent scheme:
$$x^{i+1} \leftarrow x^{i} - \lambda \nabla_{x^i}L(y,x^i)$$
where $\lambda$ is your learning rate.


Training actor-critic algorithms in games with opponents I am wondering how am I supposed to train a model using actor/critic algorithms in environments with opponents. I tried the followings (using A3C and DDPG):

Play against random player. I had rather good results, but not as good as expected since most interesting states cannot be reached with a random opponent.
Play against list of specific AIs. Results were excellent against those AIs, but very bad with never seen opponents
Play against itself. Seemed the best to me, but I could not get any convergence due to non-stationary environment.

Any thought or advice about this would be very welcome.

The specific approaches you mentioned (A3C, DDPG), and usually also other Actor-Critic methods in general, are approaches for the standard single-agent Reinforcement Learning (RL) setting. When trying to apply such algorithms to settings that are actually multi-agent settings, it is indeed common to encounter the problems you describe. They can all be viewed as your agent "overfitting" to the opponent they're training against:

  Play against random player. I had rather good results, but not as good as expected since most interesting states cannot be reached with a random opponent.

In this case, your agent will "overfit" against random opponents. It will likely become capable of beating them relatively easily, and to some extent this may also still translate to relatively weak non-random agents (like simple heuristic approaches), but it is unlikely to generalise to strong opponents.

  Play against list of specific AIs. Results were excellent against those AIs, but very bad with never seen opponents

Here you pretty much exactly give a description of what I mean when I say "overfitting" against the opponent you're training against.

  Play against itself. Seemed the best to me, but I could not get any convergence due to non-stationary environment.

When training using self-play like this, there is still a danger of overfitting against "itself". This may turn out to work well in some games, but in other games, there are indeed risk of instability / lack of convergence. This can, for example, be due to the agent "rotating" through a circle of strategies that beat each other. Intuitively, you could think of Rock-Paper-Scissors. A randomly-initialised agent may, for example, have a slight tendency to play Rock more often than the others. Self-play will then lead to a strategy that primarily plays Paper, to beat the current "Rock" strategy. Continued self-play training can then figure out that a tendency to play more Scissors will be strong against itself. Etc., these strategies can keep rotating forever.

The standard approach for self-play training in games is the one popularised by AlphaGo Zero and AlphaZero, and also around the same time independently published with results on the game of Hex. That last paper coined the term "Expert Iteration" for the approach, which I like to use. The basic idea of Expert Iteration is to have:

A policy that we're training (parameterised by some function approximator like a Deep Neural Net)
A tree search algorithm, like Monte Carlo tree search (MCTS).

These two components can then iteratively improve each other. The trained policy can be used to guide the search behaviour of MCTS. MCTS can be used to generate a new distribution over actions which, thanks to additional search effort, is typically expected to be a little bit better than then trained policy. That slightly better distribution is then used as a training target for the policy, using a Cross-Entropy loss. This means that the policy is trained to mimic the MCTS search behaviour, and also used to improve that same MCTS search behaviour.

This training procedure has been found to lead to strong results. One of the prevailing hypotheses is that the additional search performed by MCTS can help to stabilise training, and avoid overfitting to the "self" opponent; the search algorithm can actually "reason" about what a strong opponent (not necessarily the "self" agent) could do.

The Expert Iteration training procedure I described above often works well, but it doesn't really answer your question about training Actor-Critic approaches in games with opponents... because it's not really an Actor-Critic algorithm!

In Learning Policies from Self-Play with Policy Gradients and MCTS Value Estimates, we (I'm first author on this paper) propose an approach where the policy in self-play is trained using a training target based on the value estimates of MCTS, rather than the visit counts of MCTS (which is what we use in the standard Expert Iteration framework).

I don't know if it necessarily should exactly be called an Actor-Critic algorithm, but you could intuitively view it as an "Actor-Critic" approach where MCTS is the critic. As mentioned right before Section IV, the resulting gradient estimator also turns out to be very similar to that of the "Mean Actor Critic".

If Expert-Iteration-style solutions are not an option (for example because tree search is infeasible), I would suggest taking a look at some of the following Policy-Gradient-based approaches (and possibly later research that cites these papers):

Learning with Opponent-Learning Awareness
Policy Gradient Search: Online Planning and Expert Iteration without Search Trees


How to validate that my DQN hyperparameters are the optimal? My DQN model outputs the best traffic light state in an intersection. I used different values of batch size and learning rate to find the best model. How would I know if I got the optimal hyperparameter values?

If possible, I would try to calculate what the (theoretical) maximum  throughput through the intersection is for a given time interval. If the control behavior that the DQN produces comes empirically close to the maximally possible throughput score, the model is good. Otherwise, you could measure and compare the throughput of different models and choose the best performing one.


Why was the VC dimension not defined for all configurations of $d$ points? Let's start with a typical definition of the VC dimension (as described in this book)

Definition $3.10$ (VC-dimension) The $V C$ -dimension of a hypothesis set $\mathcal{H}$ is the size of the largest set that can be shattered by $\mathcal{H}$ :
$$
\operatorname{VCdim}(\mathcal{H})= \max \left\{m: \Pi_{\mathcal{H}}(m)=2^{m}\right\}
$$

So, if there exists some set of size $d$ that $\mathcal{H}$ can shatter and it cannot shatter any set of size $d+1$, then the $\operatorname{VCdim}(\mathcal{H}) = d$.
Now, my question is: why would we be just interested in the existence of some set of size $d$ and not all sets of size $d$?
For instance, if you consider one of the typical examples that are used to illustrate the concept of the VC dimension, i.e. $\mathcal{H}$ is the set of all rectangles, then we can show that $\operatorname{VCdim}(\mathcal{H}) = d = 4$, given that there's a configuration of $d=4$ points that, for all possible labellings of those points, there's a hypothesis in $\mathcal{H}$ that correctly classifies those points. However, we can also easily show that, if the 4 points are collinear, there's some labelling of them (i.e. the 1st and 3rd are of colour $A$, while the 2nd and 4th are of colour $B \neq A$) that a rectangle cannot classify correctly.
So, the class of all rectangles can shatter some sets of points, but not all, so we would need another class of hypotheses to classify all sets of four points correctly. The VC dimension does not seem to provide any intuition on which set of classes would do the trick.
So, do you know why the VC dimension wasn't defined for all configurations of $d$ points? Was this just a need of Vapnik and Chervonenkis for the work they were developing (VC theory), or could have they defined it differently? So, if you know the rationale behind this specific definition, feel free to provide an answer. References to relevant work by Vapnik and Chervonenkis are also appreciated.

I would say, that meaning of VC dimension is at least a possibility to implement any function on the given number of points for some case, not an express any function on $n$ points.
Yes, you are right, that this definition, unfortunately, is not very useful in practice.
Say, family of functions $\text{sign}(\sin(ax))$ has infinite $\text{VC}$ dimension. There exists an infinite sequence $x_n$, such that by tuning the parameter $a$ one can implement any logical function $0 1 0 \ldots 1$ on these points. However, this doesn't make this class of functions an outstanding ML algorithm.


Can Q-learning be used to derive a stochastic policy? In my understanding, Q-learning gives you a deterministic policy. However, can we use some technique to build a meaningful stochastic policy from the learned Q values? I think that simply using a softmax won't work.

No it is not possible to use Q-learning to build a deliberately stochastic policy, as the learning algorithm is designed around choosing solely the maximising value at each step, and this assumption carries forward to the action value update step $Q_{k+1}(S_t,A_t) = Q_k(S_t,A_t) + \alpha(R_{t+1} +\gamma\text{max}_{a'}Q_k(S_{t+1},a') - Q_k(S_t,A_t))$ - i.e. the assumption is that the agent will always choose the highest Q value, and that in turn is used to calculate the TD target values. If you use a stochastic policy as the target policy, then the assumption is broken and the Q table (or approximator) would not converge to estimates of action value for the policy*.

The policy produced by Q-learning can only be treated as stochastic when there is more than one maximum action value in a particular state - in which case you can select equivalent maximising values using any distribution.

In theory you could use the Q values to derive various distributions, such as a Boltzmann distribution, or softmax as you suggest (you will want to include some weighting factor to make softmax work in general). These can work nicely for the behaviour policy, for further training, and as an alternative to the more common $\epsilon$-greedy approach. However, they are not optimal policies, and the training algorithm will not adjust the probabilities in any meaningful way related to the problem you want to solve. You can set a value for e.g. $\epsilon$ for $\epsilon$-greedy, or have more sophisticated action choice with more parameters, but no value-based method can provide a way to change those parameters to make action choice optimal.

In cases where a stochastic policy would perform better - e.g. Scissor, Paper, Stone versus an opponent exploiting patterns in the agent's behaviour - then value based methods provide no mechanism to learn a correct distribution, and they typically fail to learn well in such environments. Instead you need to look into policy gradient methods, where the policy function is learned directly and can be stochastic. The most basic policy gradient algorithm is REINFORCE, and variations on Actor-Critic such as A3C are quite popular.

* You could get around this limitation by using an estimator that does work with a stochastic target policy, e.g. SARSA or Expected SARSA. Expected SARSA can even be used off-policy to learn one stochastic policy's Q values whilst behaving differently. However, neither of these provide you with the ability to change the probability distribution towards an optimal one.


How can I classify relevant and irrelevant images from the Database? I have a mixed image database(unstructured data). In the database there are some images that i am interested in and I want to discard the rest by using cnn. I am not looking for specific objects in the images like dogs, cats etc. In the database I have photos and non photo images like infamous logos, scanned documents etc.

I want to find photos and discard the others. All the examples and online courses I found are based on object recognition. In my case which method can I use to classify my images as just 'relevant' and 'irrelevant'?

The classification of images as relevant or irrelevant can be done using LSTM, the design of which is perhaps best explained without the mathematical detail of the original paper by Eugene Kang's Long Short-Term Memory: Concept text and diagrams.

That the data is unstructured is not an issue if the images can be extracted and placed into raw pixel matrices with appropriate indices into  the database so the results of classification can be inserted into the database in a way where the image and the classification are associated.

The labels from a training point of view are not the indices into the database but the single bit indication of relevance.

The LSTM network must be trained on a sufficient sample of labeled example data before it will be usable to classify the images.  That requirement may further require a significant amount of manual classification unless some form of delegation, hints from associated features in the database, or crowd sourcing can be used to generate labels.


Using batches in testing If one examines SSD: Single Shot MultiBox Detector code from GitHub repository, it can be seen that, for a testing phase (evaluating network on test data set), there is a parameter test batch size. It is not mentioned in the paper.

I am not familiar with using batches during  network evaluation. Can someone explain what is the reason behind using it and what are advantages and disadvantages?

Test batch time is referenced in the paper on page 33, Section 3.6, Inference Time and in Table 6. When an example set is used, a portion of it is used for training and another portion is used for testing. In mini-batch configurations the mini batch sizes are also independently configurable between train and test phases in many implementations.

References

SSD: Single Shot MultiBox Detector, 2016, Wei Liu


Should batch-normalization/dropout/activation-function layers be used after the last fully connected layer? I am using the following architechture:

3*(fully connected -> batch normalization -> relu -> dropout) -> fully connected

Should I add the batch normalization -> relu -> dropout part after the last fully connected layer as well (the output is positive anyway, so the relu wouldn't hurt I suppose)?

No, the activation of the output layer should instead be tailored to the labels you're trying to predict. The network prediction can be seen as a distribution, for example a categorical for classification or a Gaussian (or something more flexible) for regression. The output of your network should predict the sufficient statistics of this distribution. For example, a softmax activation on the last layer ensures that the outputs are positive and sum up to one, as you would expect for a categorical distribution. When you predict a Gaussian with mean and variance, you don't need an activation for the mean but the variance has to be positive, so you could use exp as activation for that part of the output.


Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible? If we shift the rewards by any constant (which is a type of reward shaping), the optimal state-action value function (and so optimal policy) does not change. The proof of this fact can be found here.
If that's the case, then why does a negative reward for every step encourage the agent to quickly reach the goal (which is a specific type of behavior/policy), given that such a reward function has the same optimal policy as the shifted reward function where all rewards are positive (or non-negative)?
More precisely, let $s^*$ be the goal state, then consider the following reward function
$$
r_1(s, a)=
\begin{cases}
-1, & \text{ if }  s \neq s^*\\
0, & \text{ otherwise}
\end{cases}
$$
This reward function $r_1$ is supposed to encourage the agent to reach $s^*$ as quickly as possible, so as to avoid being penalized.
Let us now define a second reward function as follows
\begin{align}
r_2(s, a)
&\triangleq r_1(s, a) + 1\\
&=
\begin{cases}
0, & \text{ if }  s \neq s^*\\
1, & \text{ otherwise}
\end{cases}
\end{align}
This reward function has the same optimal policy as $r_1$, but does not incentivize the agent to reach $s^*$ as quickly as possible, given that the agent does not get penalized for every step. So, in theory, $r_1$ and $r_2$ lead to the same behavior. If that's the case, then why do people say that $r_1$ encourage the agents to reach $s^*$ as quickly as possible? Is there a proof that shows that $r_1$ encourages a different type of behaviour than $r_2$ (and how is that even possible given what I have just said)?

Your examples are equivalent. But it is possible to find a constant yielding  a different optimal policy.
Your examples are absolutely equivalent. The agent maximizes the reward, and only way to do so is by reaching $s^*$.
Consider $r_3$ :
$$
r_3(s, a)=
\begin{cases}
1, & \text{ if }  s \neq s^*\\
2, & \text{ otherwise}
\end{cases}
$$
With a sufficiently large $\gamma$, moving infinitely without reaching $s^*$ is now the optimal solution.
For the generic case
$$
r_4(s, a)=
\begin{cases}
\alpha, & \text{ if }  s \neq s^*\\
\beta, & \text{ otherwise}
\end{cases}
$$
the threshold is found by comparing the results of the series $\alpha + \alpha^2 + \alpha^3 + ... + \alpha^{t_m}$, where $t_m$ is the maximum episode length, and $\alpha + \alpha^2 + \alpha^3 + ... + \alpha^{t^*}$, where $t^*$ is the length of the episode following the fastest policy.
In the example of $r_3$, it is trivial to find examples where the fastest policy isn't optimal. Imagine a race, the agent starts on the left and gets either $\alpha$ or $\beta$ points, depending on where it is. With $\gamma = 0.9$ and no time-limit (infinite episodes) the optimal policy is  to move randomly, but in the second-to-last house, avoid the goal state. With $\gamma = 0.1$, the optimal policy is to move randomly (not really, probably there would be a slight advantage in moving right), but in the second-to-last house, enter the goal.


Neural network architecture for comparison When someone wants to compare 2 inputs, the most widespread idea is to use a Siamese architecture. Siamese architecture is a very high level idea, and can be customized based on the problem we are required to solve.

Is there any other architecture type to compare 2 inputs ?

Background

I want to use a neural network for comparing 2 documents (semantic textual similarity). Siamese network is one approach, I was wondering if there is more.

A RBM (restricted Boltzmann machine) can be trained to extract document features. The same resulting machine can extract features of two or more documents.  Because documents can be just as easily processed in series using the same machine parameters and CPU (saving the feature results) as documents could be processed in parallel using separate CPUs, the Siamese idea is less of a characteristic of system architecture and more of a characteristic of the process topology. The architectural decision of how to run the process topology on available hardware should usually remain somewhat decoupled.

The comparison operation could be a MLP (multi-layer perceptron) trained to produce the comparison results based on a requisite number example document comparisons.  In such a case, the MLP should be trained with the previously trained RBMs as a front end.  You will need some human labor to produce or extract from available data sources the example document comparison results.  The example results will be need, along with a reference to the pair of RBMs feature extraction results corresponding to the comparison, to train the MLP.

Even further upstream from the RBMs, it is often useful to producing reliable and accurate document processing to pre-process the documents.  Images and text can be delineated so that features can be extracted from the images in one way and from the text in another and then from the sequence of text and images in a third way.  In such process topologies, CNNs (convolution network) typically pre-process the images and LSTM networks typically pre-process the text.  They may then directly feed the RBM or indirectly through another process component.


What is the best resources to learn Graph Convolutional Neural Networks? For the past few days, I am trying to learn graph convolutional networks. I saw some of the lectures on youtube. But I can not able to get any clear concept of how those networks are trained. I have a vague understanding of how to perform convolution, but I can not understand how we train them. I want a solid mathematical understanding of graph convolutional networks. So, can anyone please suggest me how to start learning graph convolutional network from start to expert level?

I believe Graph Representation Learning book by William L. Hamilton is a great resource to start