How to implement SVM algorithm from scratch in a programming language? I'm a computer scientist who's studying support vector machines (SVMs) in a machine learning course. I have some understanding of how SVMs are designed, thanks to 16. Learning: Support Vector Machines - MIT. However, what I'm not understanding is the transition from the optimization problem of the Lagrangian function to its implementation in any programming language. Basically, what I need to understand is how to build, from scratch, the decision function, given a training set. In particular, how do I find Lagrange multipliers in order to know which points are to be considered to define support vectors and the decision function?
Can anyone explain this to me?

Based on this repository:
https://github.com/arkm97/svm-from-scratch/blob/master/SVM_from_scratch.ipynb
I will try to reverse engineering that concept:
So firstly there is issue for DataCleaning(removing 0,values, serialize, normalize)
normalizing dataset (replaced by its difference from the mean)
credit_df_norm = (credit_df - credit_df.mean())/(credit_df.std())

Divide dataset into Train & Test:
train_df = credit_df_norm.drop(target, axis=1).loc[:training_points]
train_target = credit_df_norm[target].replace(0, -1).loc[:training_points]

And here is starting the clue of that algorithms (with maths etc.)

given data points $\vec x_j \in \mathbb{R}^{1 \times N}$ and targets
$y_j = \pm 1$, where $j = 1, \dots, M$, find the maximum-margin
hyperplane that separates the two classes ($y_j = 1$ and $y_j = -1$).
Let $\vec w$ be the vector normal to the hyperplane. We want to find
$\vec w$ that satisfies
$$ y_j (\vec w \cdot \vec x_j + b) \geq 1 $$ The dual formulation of
the above is equivalent to maximizing the following over the
multipliers $\vec \alpha$:
$$ L(\vec \alpha) = \vec y \cdot \vec \alpha  - \frac 1 2  \vec \alpha
> K \vec \alpha^T$$ subject to the constraints $\sum_{j=1}^M \alpha_j =
> 0$ and $y_j \alpha_j \geq 0$. The matrix $K$ defines the kernel of the
SVM; I've chosen $K_{jk} = k(\vec x_j, \vec x_k) = \vec x_j \cdot \vec
> x_k$. The parameters of the plane are recovered from $\vec w = \vec
> \alpha \cdot \vec x$ and $b = y_j = \vec w \cdot \alpha_j$ for $j$
such that $\alpha_j \neq 0$

Let's divide it to the single factors:
we need to find maximum-margin-hyperplane for separate into two classes:
Separation cause SVN is classification - so we need to classify elements into categories.
 = 1
 = −1
This pattern allow to find hyperplane(?)
(⃗ ⋅⃗ +)≥1
The next pattern:
(⃗ )=⃗ ⋅⃗ −12⃗ >⃗
The author for kernel SVN choose those pattern:
=(⃗ ,⃗ )=⃗ ⋅>⃗
And the parameters for plane are here:
⃗ =>⃗ ⋅⃗  and ==⃗ ⋅ for  such that ≠0
Reference:
https://arxiv.org/pdf/1307.0471.pdf

In this line are added the kernel
Popular kernels are:

Polynomial Kernel,

Gaussian Kernel,

Radial Basis Function (RBF),

Laplace RBF Kernel,

Sigmoid Kernel,

Anove RBF Kernel
k_value = np.array(train_df @ train_df.T + np.identity(len(train_target))*1e-12)

The structure of k_value (kernel matrix) is an array.

Next there is use of cholesky decomposition—cholesky - which is next complicated concept.
np.linalg.cholesky(k_value)

Next some bits of data converting:
alpha = cp.Variable(shape=train_target.shape)
beta = cp.multiply(alpha, train_target) # to simplify notation
K = cp.Parameter(shape=k_value.shape, PSD=True, value=k_value)

# objective function
obj = .5 * cp.quad_form(beta, K) - np.ones(alpha.shape).T @ alpha

# constraints
const = [np.array(train_target.T) @ alpha == 0,
        -alpha <= np.zeros(alpha.shape),
        alpha <= 10*np.ones(shape=alpha.shape)]
prob = cp.Problem(cp.Minimize(obj), const)

The next step is recreate the hyperplane:
w = np.multiply(train_target, alpha.value).T @ train_df
S = (alpha.value > 1e-4).flatten()
b = train_target[S] - train_df[S] @ w
b = b[0]
# b = np.mean(b)

And finally - the classification and evaluation:
def classify(x):
    result = w @ x + b
    return np.sign(result)

correct = 0
incorrect = 0
validation_set = credit_df_norm.drop(target, axis=1)
predictions = []
for i, x in validation_set.iterrows():
    my_svm = classify(x)
    if my_svm==credit_df_norm[target].replace(0, -1)[i]: correct +=1
    else: incorrect +=1
    predictions.append(my_svm)
predictions = np.array(predictions)

print(f"fraction correct: {correct/(correct + incorrect)}")

That was the more detailed implementation - for shorcut there is an interesting article:
https://towardsdatascience.com/svm-and-kernel-svm-fed02bef1200
There are existing function from another library - that can be reused in simple manner:
# Fitting SVM to the Training set
from sklearn.svm import SVC
classifier = SVC(kernel = 'rbf', C = 0.1, gamma = 0.1)
classifier.fit(X_train, y_train)

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

# use seaborn plotting defaults
import seaborn as sns; sns.set()

Visualized Dataset:

Dataset after apply hyperplanes:


What is meant by a multi-dimensional continuous action space? In the context of Reinforcement Learning, what does it mean to have a multi-dimensional continuous action space?
I came across the following in the COBRA Paper

A method for learning a distribution over a multi-dimensional continuous action space. This learned distribution can be sampled efficiently.

and

During the initial exploration phase it explores its environment, in which it can move objects freely with a continuous action space but is not rewarded for its actions.

So, what do the multi-dimensionality and the continuity of the action space refer to? It'd be great if someone could provide an explanation with examples!

Let me rephrase it a little - it's a multidimensional continuous space of actions. So, you assign each action some vector from $R^{n}$. For intuition -- imagine you have a robot arm with four joints. For every joint you could applied a rotation force from [-1, 1] and thus you get a 4-D vector with float numbers for each possible action.


What is the appropriate RNN structure to do Sentiment Analysis with multiple dependent ratings? Suppose we are doing sentiment analysis for a restaurant. Customers can rate the restaurant by #1: how expensive the restaurant is, #2:how good is the food and #3: how likely they will come again. The ratings are dependent,i.e. the more expensive the restaurant is (higher #1), the less likely they will come back (lower #3), but whey will if the food is good (higher #2).

My questions are: is there a good RNN structure(review as input, #1-#3 as output) that can capture and model the dependency among #1 - #3?

Inspired by (1) machine translation where source phrases are first encoded to a feature vector and decoded by another RNN and (2) automatic captioning where a picture is also encoded to a feature vector before it's decoded by the RNN followed. A proper structure for the problem above may be the inverse of these, where the review is encoded by RNN to a vector, and a DNN is followed to decode the vector into #1 - #3.


Terminology for the use of datasets as data points As computers are getting bigger better and faster, the concept of what constitutes a single datum is changing.

For example, in the world of pen-and-paper, we might take readings of temperature over time and obtain a time-series in which an individual datum is a time, temperature pair.  However, it is now common to desire classifications of entire time-series, in the context of which our entire temperature time-series would be but a single data point in a data set consisting of a great number of separate time-series.  In image processing, an (x,y,c) triple is not a datum, but a whole grid of such values is a single datum.  With lidar data and all manner of other fields things that were previously considered a dataset are now best thought of as a datum.

What is the term for datasets that are themselves composed of datasets?  The term "metadata" is occupied, I should think.  Are there any papers that talk about this transition from datasets of data to datasets of datasets and what the implications are for data scientists and researchers?

I don't think that this is anything new. Let's use your example of classifying an entire time series, say predicting word 1 vs word 2 for speech recognition. We can write out the data as a data frame like we would do with any other multivariate data: observations at time 1, time 2, etc as the predictors and the classification label as the response variable.

Each observation is a vector of the values at particular times for your subject, plus the label--no different than any other multivariate data. Sure, there might be special dependence structure because of the time series nature of your data, but you can still write it as a multivariate problem.

Okay, let's say that you hit the speech signal with a wavelet transform, resulting in an image-looking spectrogram of 2D data. Then just consider each "pixel" (time-frequency pair) to be a variable in your multivariate problem, along with the classification label. This is some kind of bijection between an $m\times n$ matrix and $\mathbb{R}^{n\times m}$.

You can extend this idea to 3D or 4D data (or higher), too. Just unwrap the high-dimension tensor in some kind of map $T^{m\times n \times \dots} \rightarrow \mathbb{R}^{m\times n \times \dots}$.


Structure discrepancy of an LSTM? I've found multiple depictions of how an LSTM cell operates. See 2 below:

and

Each of these images suggest the hidden state is utilised differently. On the top diagram, it is shown that the hidden state is added along with the previous output and current input to both the forget gate and the input gate. The bottom image suggests the input and forget gates are calculated only using the previous output and current input. Which is it?

Also, when the previous output is fed in for the current layer, is this before or after it has been reshaped to the final output size and been put through a softmax?

There are different variants of LSTM, in most ML packages now of days you'll probably see what's shown in the bottom picture. For more details, intuition and motivations please see this paper.
It is not reshaped and there is no softmax layer. That is all done outside the LSTM.


Is it possible to design an AI with two inputs and a Boolean output? I am having a difficult time explaining to my boss that what he is trying to achieve may not be possible or within reason. We have a database of 3 Million data points per computer across hundreds of machines and when any data point is updated, changed, or removed. Some of these data points are the number of times a computer has been logged in, the names of printers attach, folders on the root of the drive. Some of the data points we do care about, others like a printer being out of ink, we don't care about but the same method would return if the printer was offline which we do care about.
He wants to design an AI that would check these data points and return with true or false on whether the data point is significant when they are changed, removed or updated. We are storing the name of the method to retrieve the data, the current data, all previous data, and the time the change was made. I can not foresee a way to train the data efficiently as we currently don't know which methods retrieve significant data or which values are not significant within the method.
Is it possible to design such an AI without hours of supervisor learning?

Yes... But...  you have a lot of hours.. possibly days or weeks.. of work before you are at that point.
Your bigger problem is apparent from your concerns in the second paragraph.  It doesn't sound as though your org has a solid grasp of the problem at the moment.  For that reason, it seems that some first steps are in order.
Data Exploration
Begin by collecting all of the data points.  I'd recommend that you perhaps begin with some statistical analysis of the data as a whole, including some basic visualization and generating a covariance matrix.  From there, begin to look at using some clustering methods to identify possible patterns.  Along the way, you will almost certainly go down the path of some dimensionality reduction, either via something like PCA or possibly identifying useless features.
Feature Selection
Based on the exploration work above you should now have a much better understanding of your data and relationships within it.  Based on this, it's time to start thinking about how to generate a model that produces the desired output.  Frankly, you may discover that something as simple as a random forest classification or even a clustering method such as DBSCAN can be used to initially train and then continuously fit your data, producing either a binary classification with the random forest or a yes/no cluster with the clustering technique.
Is More Required?
Of course, something more sophisticated might be required, but if it is you would know have a better handle on the problem and likely be able to intuitively generate a large dataset that could be used for a neural network.
Oh... And as a concluding thought... It might turn out that after all of this analysis the problem cannot be solved with the data that you have.  At that point you have to go back and see if there are other data points that could be gathered.


What are acting as weights in a convolution neural network? Looking at some old notes I took on CNN's and I wrote down that the weights in a CNN are acting like filters in a CNN but to be honest I don't really know what the weights are acting as in a CNN and was wondering if someone could explain that clearly to me.

There are many resources that answer your question, but, given that you're apparently new to machine learning (ML), deep learning (DL), and neural networks (NN), let me provide a simple answer that should clarify your doubts.
The term weight in the context of ML, DL, and NN is a synonym for parameter (sometimes, in some contexts, such as linear regression, it is also known as coefficient), which can be constant (i.e. do not change during, for example, learning/training) or learnable (i.e. change during the training/learning process). In a feedforward neural network (FFNN), with and without recurrent connections, the weights are the numbers on the connections between neurons in different or same layers. They are called weights to emphasize that their role is to weigh the effect of one neuron on the other.
In a CNN, the weights are the kernels/filters of the CNN, i.e. the matrices that you use to perform the convolution (or cross-correlation) operation in a convolutional layer. So, given that CNNs perform an operation that seems to be different than the linear combination followed by the non-linear activation function in FFNNs, you could think that the weights in a CNN are not very similar to the weights in FFNNs. However, this is not fully true, as CNNs can be viewed as FFNNs (with some specific structure): maybe this answer will provide you more info about this topic. So, your notes are right!
In my opinion, the best way to understand the role of the kernels (i.e. weights) of a CNN and all the details behind the convolution (or cross-correlation) operation is to first study some traditional computer vision and image processing techniques, so maybe you should pick a book that covers them.
In this answer, I briefly try to describe what a CNN is and what it can be used for. There are other answers on this site, such as this, this, and this, which you may want to read, if not now later, to understand even more all the details of CNNs. A  decent reference book that covers CNNs (and other DL topics) is Deep Learning by Ian Goodfellow et al.


Is there a Python API for manipulating publicly available datasets? In my daily machine learning / deep learning workflow, I often want to interact with a dataset in my code. Specifically, I would like to be able to load some module/package which can

make sure that I have the dataset in a specific folder
represent the structure of the dataset on a higher, possible object-oriented level
is homogeneous across datasets of the same class, e.g. image classification (set of images with labels, possibly bounding boxes)
open-source, such that I could contribute a specification to a previously uncovered dataset
that lets me specify a set of commonly used licenses to which I can comply, to filter all compatible datasets

Currently, I mainly use Python, but other languages are also in the scope of this question (Matlab, Java).
Does such an API exist? If not, which ones come closest to the requirements stated above?

To give you an example of how I would expect it to feel like, see the following python code

import dataset_api as da

in_2012 = da.get('ImageNet', version='2012')
in_2017 = da.get('ImageNet', version='2017')
coco = da.get('COCO')

image_classification = da.merge([in_2012, in_2017, coco])

images = image_classification.images

image = images[0]
image.path  # the absolute file path to the image file on my disk
image.objects  # an array of object that are visible in this image

scikit-learn has a small data sets API http://scikit-learn.org/stable/datasets/index.html
I imagine one can add more data sets locally.
Some data sets are for classification, other for regressions.

This is the only one I know about.


What is the physics engine used by DeepMimic? I found a video for the paper DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
 on YouTube.

I looked in the related paper, but could not find details of how to the environment was created, such as the physics engine it used. I would like to use it, or something similar.

Bullet physics engine

Their paper says

Physics simulation is performed at 1.2kHz using the Bullet physics engine [Bullet 2015].


What beginner-friendly machine learning method should I use to make teams for my pickup ultimate frisbee club fairly balanced? A bunch of friends and I play ultimate every week. Recently I wrote a program to choose our teams for us, as well as keep track of certain data (like which players were on which team, which team won, what was the score, how long the game was, etc). I wanted to use a machine learning technique to make the teams for us in order to optimize how fairly balanced the teams are (possibly measured by how many total points are scored in a game or how long a game lasts).

I am currently taking a machine learning MOOC and being introduced to very basic machine learning techniques (linear regression with gradient decent or normal equations, basic classification stuff, stuff like that). Although I hope I will come across a technique that fits my needs by the end of this course, I wanted to ask here to see if I can get a head start.

I've tried searching around everywhere, but couldn't find anything relevant. So my question is, is there an obvious technique I should look into for such a problem? If it's something too advanced for a beginner, that's fine too, but I'd like to get started learning/practicing it asap instead of waiting for my course to hopefully hit upon it.

Thank you!

EDIT: to clarify further, I would prefer something that looks at relationships between individual players like "when Steph plays with Bill she is more likely to win" or "Steph plays worse when she is on a team with players who have a high win percentage". I'd also prefer to be able to code it in python, but am willing to learn any other language

What you have on your hands is an attribution problem (and that’s a good keyword to help in your Googling). Two common approaches are computing Shapley values or a Markov chain.

In your case, I think Shapley values would be a good approach. To over-simplify, this approach attempts to first determine the total “surplus” of value created by different combinations of players, and then estimate how much was due to each player in the combination. Also under consideration are constrained combinatorial effects: e.g. having two great goalies doesn’t add much over one great goalie since they can’t play at the same time.

Depending on how hands-on you want to be you could implement yourself or find existing libraries by Googling e.g. “Shapley values python”.


Can we use a neural network that is trained using Reinforcement Learning for dynamic game level difficulty designing in realtime? I am a newbie to Machine Learning and AI. As per my understanding, with the use of reinforcement learning
(reward/punishment environment), we can train a neural network to play a game. I would like to know, whether it possible to use this trained model for deciding the difficulty of the next game level dynamically in realtime according to a player's skill level? As an example, please consider a neural network is trained using Reinforcement Learning for playing a mobile game (chess/puzzle, etc.). The game is not consists of a previously designed static set of game levels. After the training, can this model use to detect a particular player's playing style(score, elapsed time) to dynamically decide the difficulty of the next game level and provide customized game levels for each player in realtime?

Thank you very much and any help will be greatly appreciated.

There have been many researches in dynamic difficulty adjustment (DDA). I see this one is quite explaining: AI for Dynamic Difficulty Adjustment in Games. However, there are many factors when we are trying to do dynamic difficulty adjustment. As explained in paper above, one major problem is it is sometimes hard to make sure the created model will still deliver the intended experience.

You can also read this paper about game design and DDA on ace05The Case for Dynamic Difficulty Adjustment in Games .

So for short, you can use neural network -- or other learning methods -- to do dynamic difficulty adjustment. But it's more about game design and experience impact of your DDA.


Are PAC learning and VC dimension relevant to machine learning in practice? Are PAC learning and VC dimension relevant to machine learning in practice? If yes, what is their practical value?

To my understanding, there are two hits against these theories. The first is that the results all are conditioned on knowing the appropriate models to use, for example, the degree of complexity. The second is that the bounds are very bad, where a deep learning network would take an astronomical amount of data to reach said bounds.

Yes, PAC learning can be relevant in practice. There's an area of research that combines PAC learning and Bayesian learning that is called PAC-Bayesian (or PAC-Bayes) learning, where the goal is to find PAC-like bounds for Bayesian estimators.

For example, Theorem 1 (McAllester’s bound) of the paper A primer on PAC-Bayesian learning (2019) by Benjamin Guedj, who provides a nice overview of the topic, shows a certain bound that can be used to design Bayesian estimators. An advantage of PAC-Bayes is that you get bounds on the generalization ability of the Bayesian estimator, so you do not necessarily need to test your estimator on a test dataset. Sections 5 and 6 of the paper go into the details of the real-world applications of PAC-Bayes.

See e.g. Risk Bounds for the Majority Vote: From a PAC-Bayesian
Analysis to a Learning Algorithm (2015) by P. Germain et al. for a specific application of PAC-Bayes. There's also the related Python implementation.

See also these related slides and this blog post (by the same author and John Shawe-Taylor) that will point you to their video tutorials about the topic.

The VC dimension can also be useful in practice. For example, in the paper Model Selection via the VC Dimension (2019) M. Mpoudeu et al. describe a method for model selection based on the VC dimension.


What is ergodicity in a Markov Decision Process (MDP)? I have read about the concept of ergodicity on the safe RL paper by Moldovan (section 3.2) and the RL book by Sutton (chapter 10.3, 2nd paragraph).
The first one says that "a belief over MDPs is ergodic if and only if any state is reachable from any other state via some policy or, equivalently, if and only if":
$$\forall s, s', \exists \pi_r \text{ such that } E_\beta E_{s, \pi_r}^P [B_{s'}] = 1$$
where:

$B_{s'}$ is an indicator random variable of the event that the system reaches state $s'$ at least once, i.e., $B_{s'} = 1 \{ \exists t < \infty \text{ such that } s_t = s'\}$
$E_\beta E_{s, \pi_r}^P[B_{s'}]$ is the expected value for $B_{s'}$, under the belief over the MDP dynamics $\beta$, policy $\pi$ and transition measure $P$.

The second one says "$\mu_\pi$ is the steady-state distribution, which is assumed to exist for any $\pi$ and to be independent of $s_0$. This assumption about the MDP is known as ergodicity.". They define $\mu_\pi$ as:
$$\mu_\pi(s) \doteq \lim_{t \to \infty} \Pr\{s_t=s \vert a_{0:t-1} \sim \pi\}$$

i.e., there is a chance of landing on state $s$ by executing actions according to policy $\pi$.

I noticed that the first definition requires that at least one policy should exist for each $(s, s')$ pair for the MDP to be ergodic The second definition, however, requires that all policies eventually visit all the states in an MDP, which seems to be a more strict definition.
Then, I came accross the ergodicity definition for Markov chains:

A state $i$ is said to be ergodic if it is aperiodic and positive recurrent. In other words, a state $i$ is ergodic if it is recurrent, has a period of $1$, and has finite mean recurrence time. If all states in an irreducible Markov chain are ergodic, then the chain is said to be ergodic.

This leads me to believe that the second definition (the stricter one) is the most appropriate one, considering the ergodicity definition in an MDP derives from the definition in a Markov chain. As an MDP is basically a Markov chain with choice (actions), ergodicity should mean that independently of the action taken, all states are visited, i.e., all policies ensure ergodicity.
Am I correct in assuming these are different definitions? Can both still be called "ergodicity"? If not, which one is the most correct?

In short, the relevant class of a MDPs that guarantees the existence of a unique stationary state distribution for every deterministic stationary policy are unichain MDPs (Puterman 1994, Sect. 8.3). However, the unichain assumption does not mean that every policy will eventually visit every state. I believe your confusion arises from the difference between unichain and more constrained ergodic MDPs.
Puterman defines that a MDP is (emphasis in the following mine):

recurrent or ergodic if the state transition matrix corresponding to every deterministic stationary policy consists of a single recurrent
class, and
unichain, if the state transition matrix corresponding to every deterministic stationary policy is unichain, that is, it consists of a
single recurrent class and a possibly empty set of transient states.

Before unpacking this further, let's first recap what recurrent and transient states in a Markov chain are. The following definitions can be found in Puterman, Appendix A.2. For a state $s$, associate two random variables $\nu_s$ and $\tau_s$ that represent the number of visits and the time of the first visit (or first return if the chain starts in $s$) to state $s$. A state $s$ is

recurrent (sometimes also called ergodic), if $P_s(\tau_s < \infty) = 1$, that is, the state is eventually visited (or returned to), and
transient, if $\mathbb{E}[\tau_s] = \infty$ and $P_s(\tau_s < \infty) < 1$.

It is also true that $s$ is recurrent if and only if $\mathbb{E}[\nu_s] = \infty$, i.e., it is visited infinitely often, and $s$ is transient if and only if $\mathbb{E}[\nu_s] < \infty$, i.e., it is visited only finitely often (and thereby never again after some finite time).
So let's now return to the two types of MDPs above. Consider an arbitrary deterministic stationary policy $\pi$ which maps any state $s$ to an action $a = \pi(s)$. If the MDP is ergodic, then the stationary distribution $\mu_\pi(s)$ exists and is unique, because the Markov chain over states induced by any policy has a single recurrent class (it does not matter in which state $s_0$ the chain starts, the same stationary distribution is reached). There is a single class of recurrent states, i.e., all states are recurrent, therefore any $s$ is visited infinitely often and $\mu_\pi(s) > 0$.
Now, if the MDP is unichain, then once again the stationary distribution $\mu_\pi(s)$ exists and is unique, because the Markov chain over states induced by any policy has a single recurrent class. But, importantly, there may exist a policy $\pi$ for which the set of transient states in the induced Markov chain is not empty. Because any transient state $s$ will only be visited a finite number of times, in the (infinite horizon) stationary distribution $\mu_\pi(s)=0$!
So indeed, if the MDP is of the stricter ergodic type, every policy will eventually visit every state. This is not true for unichain MDPs however.
A final remark: some authors define a policy as ergodic (e.g., Kearns & Singh, 2002), if the resulting Markov chain over states is ergodic (i.e., has a unique stationary distribution). The unichain MDP is a type of MDP where every policy is ergodic.
References:
Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming.
Kearns & Singh. Near-Optimal Reinforcement Learning in Polynomial Time. Machine Learning, 49, 209–232, 2002


What roles knowledge bases play now and will play in the future? Nowadays, artificial intelligence seems almost equal to machine learning, especially deep learning. Some have said that deep learning will replace human experts, traditionally very important for feature engineering, in this field. It is said that two breakthroughs underpinned the rise of deep learning: on one hand, neuroscience, and neuroplasticity in particular, tells us that like the human brain, which is highly plastic, artificial networks can be utilized to model almost all functions; on the other hand, the increase in computational power, in particular the introduction of GPU and FPGA, has boosted algorithmic intelligence in a magnificent way, and has been making the models created decades ago immensely powerful and versatile. I'll add that the big data (mostly labeled data) accumulated over the past years is also relevant.

Such developments bring computer vision (and voice recognition) into a new era, but in natural language processing and expert systems, the situation hasn't seemed to have changed very much.

Achieving common sense for the neural networks seems a tall order, but most sentences, conversations and short texts contain inferences that should be drawn from the background world knowledge. Thus knowledge graphing is of great importance to artificial intelligence. Neural networks can be harnessed in building knowledge bases but it seems that neural network models have difficulty utilizing these constructed knowledge bases.

My questions are:

Is a knowledge base (for instance, a "knowledge graph", as coined by Google) a promising branch in AI? If so, in what ways KB can empower machine learning? How can we incorporate discrete latent variables into NLU and NLG?
For survival in an age dominated by DL, where is the direction for the knowledge base (or the umbrella term symbolic approach)? Is Wolfram-like z dynamic knowledge base the new direction? Or any new directions?

Am I missing something fundamental, or some ideas that address these issues?

First of all, I would like to point out the main differences between knowledge base and (Deep) machine learning, specially when the main focus is on "AI" not "Data Science":

NNs are like a black box; Even if they learn a dataset and gain the power of generalization over the problem domain, you'd never know how they are working. if you scrutinize the details of the developed model, all you see are digits, weights, poor and strong connections and transform functions. the "feature extraction" step before the training phase literally tells you: "hey human, enough with your complicated world, let's start zeros and ones". In the case of DL, it is worse! we do not even see what the selected and effective features are. I'm not a DL expert but as much as I know, DL's black box is darker! But knowledge bases are written in a human-friendly language. after a knowledge accumulation phase, you could see all the connections between the entities, and more important, you could interpret those connections. if you cut a wire in a knowledge base, your model will lose just a bit of its power, and you know what exactly it will lose; for example disconnecting the "Pluto" node from the "solar system" node, will tell your model what deGrasse Tyson told us. but in a ML model, this might turn it into a pure useless one: what happens if you manipulate the connection between the neuron number 14 and 47 in a NN model used to predict which planets belong to the solar system?!
ML models are merely an inscription of the data. They do not have the power of inference, and they don't give you one. knowledge base is on the other hand capable of inference from the prior knowledge as you indicated in your question. It is shown that DL models that have been trained with say image classification data, could also be applied to voice detection problem. But this doesn't mean DL models could apply its prior knowledge in the domain of images to the domain of voices.
You need kilos of data for traditional ML algorithms and tons of data for DL ones. but a single instance of a dataset will create a meaningful knowledge base for you.

There are two main research topics in NLP: machine translation and question answering. Practically it has been shown that DL works significantly with machine translation problems but acts kind of stupid in question answering challenge, specially when the domain of topics covered in the human-machine conversation is broad. Knowledge bases are no good choice for machine translation but are probably the key to a noble question answering machine. Since what matters in machine translation is only the translated version of a text (and I don't care how on earth has the machine done that as far as it is true) but in question answering problem, I don't need a parrot who repeats the same information I gave it to him, but an intelligent creature who gives me "apple is eatable" after I tell him "apple is a fruit" and "all fruits are eatable". ML models are used to elicit underneath patterns from the dataset (translation) while knowledge bases are used to extend the domain of knowns. ML models nitpick, KBs explore!


How does weight normalization work? I was reading the paper Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks about improving the learning of an ANN using weight normalization.

They consider standard artificial neural networks where the computation of each neuron consists in taking a weighted sum of input features, followed by an elementwise nonlinearity

$$y = \phi(\mathbf{x} \cdot \mathbf{w} + b)$$

where $\mathbf{w}$ is a $k$-dimensional weight vector, $b$ is a scalar bias term, $\mathbf{x}$ is a $k$-dimensional vector of input features, $\phi(\cdot)$ denotes an elementwise nonlinearity and $y$ denotes the
the scalar output of the neuron.

They then propose to reparameterize each weight vector $\mathbf{w}$ in terms of a parameter vector $\mathbf{v}$ and a scalar parameter $g$ and to perform stochastic gradient descent with respect to those parameters instead.

$$
\mathbf{w} = \frac{g}{\|\mathbf{v}\|} \mathbf{v}
$$

where $\mathbf{v}$ is a $k$-dimensional vector, $g$ is a scalar, and $\|\mathbf{v}\|$ denotes the Euclidean norm of $\mathbf{v}$. They call this  reparameterizaton weight normalization.

What is this scalar $g$ used for, and where does it come from? Is $\mathbf{w}$ is the normalized weight? In general, how does weight normalization work? What is the intuition behind it?

Your interpretation is quite correct. I could not understand how it would speed up the convergence though. What they are doing is basically re-assigning the magnitude of the weight vector (also called norm of the weight vector).

To put things in the perspective, the conventional approach to any Machine Learning cost function is to not only check the variations of the error with respect to a weight variable (gradient) but also add a normalization term which is $\lambda (w_0^2 + w_1^2 + \dots)$. This has got a few advantages:

The weights will not get exponentially high, even though if you make some mistake (generally bouncing off to exponential costs due to the wrong choice of learning rate).
Also, the convergence is quicker somehow (maybe because you have now 2 ways to control how much weight should be given to a feature. Unimportant features weights are not only getting reduced by normal gradient, but also the gradient of the normalization term $\lambda (w_0^2 + w_1^2 + \dots)$).

In this paper, they have proposed to fix the magnitude of the weight vector. This is a good way, although I am not sure if it is better than feature normalization. By limiting the magnitude of weight to $g$, they are fixing the resource available. The intuition is that, if you have 24 hours, you have to distribute this time among subjects. You'll distribute it in a way such that your grade/knowledge is maximized. So this might be helping in the faster convergence.

Also, another intuition would be, when you are subtracting the gradient from a weight vector you use a learning rate $\alpha$. This decides by how much weight-age of error you want to give which will be subsequently be subtracted from the weights. In this approach, you are not only subtracting the weights but also using another learning rate $g$ to scale the weight. I call this $g$ a learning rate because you can customize it which in turn customizes the value of weights which in turn affects the future reductions of weight gradient descent.

I am sure someone will post a better mathematical explanation of this stuff but this is all the intuition I could think of. I would be grateful if other intuitions and mathematical subtleties are pointed out. Hope this helps!


For forecasting and trading control, given limited data, what AI approaches are well matched? I'm working on stock price prediction and automatic or semi-automatic control of trading.  The price trends of these stocks exhibit recurring patterns that may be exploited.  My dataset is currently small, only in the thousands of points.  There are no images or very high dimensional inputs at all.  The system must select from among the usual trading actions.

Buy $n$ shares at the current bid price
Hold at the current position
Sell $n$ shares at what the current market will bear

I'm not sure if reinforcement learning is the best choice, deep learning is the best choice, something else, or some combination of AI components.

It doesn't seem to me to be a classification problem, with hard to discern features.  It seems to be an action-space problem, where the current state is a main input.  Because of the recurring patterns, the history that demonstrates the observable patterns is definitely pertinent.

I've tried some code examples, most of which employ some form of artificial nets, but I've been wondering if I even need deep learning for this, having seen the question, When is deep-learning overkill? on this site.

Since I have very limited training data, I'm not sure what AI design makes most sense to develop and test first.

This is an interesting problem, the answer of which is highly coveted for obvious reasons.  The production of an answer in this public space is appropriate, provided one believes in a more level distribution of assets across those alive and is comfortable with rewarding technical people for the abilities they developed while others might have been engaging in less technical pursuits.  I only ask that those who may successfully employ it consider using at least some of any windfall generated in a philanthropic manner, to benefit the less fortunate.

Here we go.

Abstract Algebra

One of the fundamental principles presented in Morgenstern and von Neumann's Theory of Games and Economic Behavior is the idea that the value of a monetary unit may vary based on actions of those that control the monetary system, but the underlying value itself has abstract algebraic properties favorable to prediction.  These properties, the book proposes in early chapters, permit additive modelling.

Explained another way, imagine a pile of paper bills of a particular denomination.  The selection of any one of the bills from anywhere in the deque is identical in value to the selector as any other selection of one bill.  The selection of which particular bill is also not a factor in the value remaining in the deque after the selection.

This is also true of shares, which is why serial numbers are not a trading parameter at the investor-trade interface.

For more theory behind this, the interested parties can study group theory and rings.

Critique of M&VN

This convenient additive view of value beneath monetary systems has been critiqued.  The value of assets, products, and services may be demonstrated to vary with life expectancy and philosophic divergence.  While some people fall into an investment mode, others may be divesting.

Fortunately, similar to Isaac Asimov's fictional presentation of psycho-history, the predictive mechanisms that may be developed are, in fact, not fettered by this investment versus divestment mode variance.  The volume of trading and number of traders typically hide the variance through cancellations across either side of mean behavior.

Those with considerable assets can exploit this caveat and a large number of other caveats in the additive nature, but most traders cannot.

Patterns in Stock Values

The phrase that may have started a number of investment firms down the path toward the use of machine learning in high speed training, leading to a conspicuous growth in the number of billionaires in the world, is a simple phrase from Darren Aronofsky's Pi, a 1998 movie in black and white.  The protagonist, Maximillian Cohen, looks at the scrolling LED version of a ticker tape and says, "I know these numbers," and then begins to verbally state each price just before it appears.

The patterns in stock prices have intrigued those who watch trends since the emergence of the New York Stock Exchange.  But they are not cyclic, in spite of the naive belief that they run in cycles.  They only appear to run in cycles, just as one might find a pattern they think they understand in a graphic representation of the Mendelbrot Set, but when the renderer of the set zooms in or out, the greater detail or the generality respectively reveal that the understanding was an illusion.

Attempts to use Fourier Analysis as part of a predictive strategy for anything but extremely narrowly timed speculation have been of limited value.  The reliability of the prediction of advantageous trading choice, expressed as a probability $P(C)$, dwindles to pure entropy (zero bits of actual predictive information) as the clock time of the prediction increases relative to that of the last time in the complete vector of state historic states $\vec{S}$ used for prediction.

$$\lim_{(t - t_\emptyset) \rightarrow \inf} P(A|\vec{S}) \rightarrow 0$$

In plain terms, the risk of trading based on a predictive model from past price trend increases with the staleness of trend information.

The increase in entropy is likely to be a feature of the prediction of any system as complex as the stock market even as prediction tools mature.  It is a consequence of initial input sensitivity combined with measurement inaccuracy and the inevitable unsatisfactory distributed model of decisioning.  These constraints may not be inescapable, but they are certainly resistant to brute force attacks, such as adding artificial network layers.

There are tools that can be predictive in ways better than trying to identify a spectrum using Fourier Analysis.  But let's first consider a spectrum, which is how music listeners identify the pitch, loudness, timbre, and envelope of notes.
Unlike the playing and listening to music, the distribution of pitches in the spectrum from moment to moment are not purely a representation of the spectral distribution in which value fluctuations reside.

The spectral analysis of stock prices reflects artificial bounds.  The spectrum is bounded by the market sampling rate and Nyquest's criteria on the high frequency side.  It is bound by the accumulation of additive effects that tends price fluctuation toward a Gaussian distribution on the low frequency end of the spectrum.

Three-fold Challenge of Freshness, Accuracy, and Complexity

Some may note that the intention of the question, to control (with or without human moderation) a stock portfolio to maximize profit is a planning problem that appears to have the Markov Property, provided the trader is not wealthy or trading in penny stocks, since the impact of previous trades seems unlikely to have significantly impacted market trends.  As we shall see, this appearance may not be objectively true.

Initial input sensitivity is a nearly universal quality of complex systems.  In the case of the stock market, the complexity is high.  Consider a cursory examination of the magnitude of market complexity, understanding that a single price is affected by neighboring prices in that sector and business and in the major market indicators that people watch.  A stock is almost completely isolated in terms of trading actions but inherently tied to all traders and other stocks through an interconnected network of IT pipelines, trading interfaces, news events, and brains, each containing a few billion neurons with complexities much greater than artificial cells.

A popular name for initial input sensitivity is the Buterfly Effect, which, properly explained is this.

  Whether a butterfly flaps its wings in Brazil may determine the path of a major storm in Peking a month later.

Sometimes the initial word, "Whether," is not included, which renders the explanation incorrect.  It is a statement of the effect of whether or not the perturbation occurs.  Once cannot say that the air movement from the wing flap causes the storm but rather the presence of a small perturbation may be grossly amplified over time in a complex system.  Or it may not.

Studies of chaotic systems have proven this potential of amplified effect as a characteristic of many if not most complex systems of differential equations with sufficient curvature in their behavioral surface to cause a system phenomena that could most descriptively be called, "Radically increasing bifurcations."

For those seeking greater depth of understanding, the work of Fourier, Nyquist, Cantor; G. Julia and P. Fatou; and Aleksandr Lyapunov may be of interest.

Because of initial input sensitivity, the Markov Property cannot be guaranteed for stock trading, even at low levels of investment.  Even though, technically, the Markov Property is not guaranteed, its assumption at low trading levels and in the leveraging of fresh data for short term gains, it may lead to reasonable approximations with sufficient reliability to win at the negative sum game of trading.

Trading as a Below Zero Sum Game

In a zero sum game, those trading with stale information and trying to apply ineffective predictive strategies will, over any longer period lose. Their monetary assets will pass to those with fresher information using effective predictive strategies.  With trading fees, the condition is more challenging, because the sum of all interactions is below zero by the sum of the fees.

Gaining Advantage

To gain advantage in any practical way above those who may have already employed machine learning in their trading operations or hired a funds manager that already did, there will have to be some clever application of more advanced principles in a way that drives effectiveness even higher than those already in game play.

These are some key elements.

Since the windowing over the price sequence of any LTS (liquid tradable security) to derive a spectrum will produce spectra with no particularly improved predictive power than the price sequence in the time domain, the goal must be to discover the nature of the patterns beyond spectral features.
For very short duration investments, spectral features may provide some advantage but perhaps not enough to overcome others using similar predictive strategies.
Some stocks are predictors of others in that there is a strong unidirectional or bidirectional causal relationship.  For instance, the cost of a barrel of oil impacts the future cost of toys, since plastic is a refinery product.  The development of a semantic net with causal relationships modeled and tuned with each epoch of training can produce predictive power that may exceed the market norm.  In this context, something like convolution may be advantageous, but not in the frame-horizontal-vertical-pixel-layer sense, since semantic networks have a non-cartesian topology.  The edges and vertices in a semantic net of LTSs don't make nice 90 degree angles that lend themselves to trivial applications of loops and arrays.
Attractors, another analytical concept from the world of chaos, are an analytical tool that lend themselves to trends in stock groups and possibly individual stocks, however diversification principles of investment also apply well to attractors.  The thing to study is auto-correlation in $\mathbb{R}^{(m n)}$, where $m$ is the number of features in the stock statistics for which there is a fresh historical record and $n$ is the number of LTSs in the group.  The group is not a trench, with varying risk pooled together to provide a favorable curve of return versus stability.  The group is a set of LTSs that correlated positively or negatively and with or without relative temporal shift (delay).
Leverage factors outside the exchange.  For instance, the result of a close election is a key event that impacts markets.  Terrorism is.  Experts who may know nothing about where the market is going but sound good can vastly sway trends.  Some billionaire somewhere may have rode a wave in a temporary correlation between the price of oranges and the price of gold, floating down the eastern seaboard in a yacht, not having the faintest idea why oranges correlated to gold, but analysis showed too many correlative effects to deny the tie, and they rode it until the model broke.

Project Strategy

It is recommended NOT to test with actual trades.  Build a system and create a simulated portfolio.  Do not use retirement funds and college funds to test a system pitted against some of the best trading automation in the world funded by a few hundred billionaires.  You will end up in Gamblers Anonymous that way.  Also understand that, as time passes, the bar will rise, and more people will be trading using predictive systems so that what is a profitable system today may consume all your time and energy and eventually lose everything you gained as other systems gain advantage over yours.

Also remember that asset management, in the final summary, is a zero sum game.  Trading gains are inescapably stuck inside the larger game where the pieces are placed back in the box and the box back in the closet.  This is true even if you have children or grandchildren.  Eventually, chaos has its sway.  Our great, great, great, great grandchildren, who probably won't care to know our name, might end up poorer because we were wealthy or wealthier because we were poor.  It's the Butterfly Effect again.

There are infinitely more important things than asset accumulation.  Ask any honest billionaire with an advanced terminal condition.

Layer Depth?

Yes depth.  There is no way to meaningfully model even the most banal aspects of the complexity of markets with a few layers of perceptrons.

Real Time Learning?

Reinforcement, yes, because the Q-learning algorithm class and variants are a powerful tool for re-entrant, real time learning system design.  However, be careful to not necessarily assuming the Markov Property.  Whether you can use an algorithm, right out of the box, wired directly to inputs from market data with no normalization and prevail in a sub-zero sum game is unlikely.

Innovation will be a requirement to any earned and even short term sustainable success.  Always, with the application of these powerful AI components, thoughtfully apply what is known to the design, initialization, and further development of the system into which the component fits.

Another network type worthy of consideration with time series is the LSTM network design.  This is where layer depth makes sense, however systems with multiple AI components are really the work horses of the industry today.  Very few problems give way easily to the deployment of some code generated from one algorithm specified in one paper and nothing added.

System Topology

There is likely to be multiple pipelines containing information in a system that sustainably provides growth-positive automated or semi-automated portfolio management, even among groupings of LTSs that seem to display patterned and more easily predictable trends.

If one wishes to get a general sense of the complexity of system design that may be required to beat those at the forefront of the investment community, visit a nuclear power plant or disassemble the control system of the jet engine of a passenger jet.  Still, someone seems to be getting it right.  A suspiciously large number of new billionaires have emerged since 1998 with investing given as the industry in their Forbes entry.

Additional technical information is provided in this answer.


As mentioned in the comments your assumption about independence is wrong. Here's why. To prove independence we need to show the following holds:
$$P(X=x, Y=y) = P(X=x)P(Y=y)$$
in the case of RL this becomes:
$$P(X=a, X=a') = P(X=a)P(Y=a')$$
The left hand side has the value:
$$P(X=a, Y=a') = b(A_t = a| S_t = s) p(s'|a,s) b(A_{t+1} = a'|, S_{t+1} = s')$$
while the right hand side has the value:
$$P(X=a)P(Y=a') = b(A_t = a| S_t = s)b(A_{t+1} = a'| S_{t+1} = s')$$
And hence not independent.
Now let use look at why the following expression holds:

Eq.5.14:   $\mathbb{E}[\rho_{t:T-1}R_{t+k}] = \mathbb{E}[\rho_{t:t+k-1}R_{t+k}]$

I will not derive the exact expressions, but I hope you can form the reasoning I provide. By the rules of probability we know that sum of joint probability is equal to 1 i.e.:
$$\sum_{X_1..X_n} P(X_1=a_1, X_2=a_2,...X_n = a_n) = 1$$
I have alredy showed above, the trajectory is not independent. So $R_{t+k}$ will depend on the trajectory $S_{t:t+k-1}$ where $S_{t:t+k-1}$ is a particular trajectory. At the end of this trajectory we get a reward $R_{t+k}$ and thus $R_{t+k}$ is exclusively a function of $S_{t:t+k-1}$ i.e. $R_{t+k} = f(S_{t:t+k-1})$. The trajectory after this $S_{t+k:T-1}$ irrelevant since it will always sum up of to 1. i.e once you have reached a particular state at time step $t+k-1$ you are now conditioning based on that $P(S_{t+k:T-1}|S_{t:t+k-1})$ and taking the expected value over all trajectories possible from thereon i.e. $\sum_{S_{t+k:T-1}} P(S_{t+k:T-1}|S_{t:t+k-1})$ which is 1 by probability rules. Thus, what you are really doing is:
$$P(S_{t:t+k-1})R_{t+k}(\sum_{S_{t+k:T-1}} P(S_{t+k:T-1}|S_{t:t+k-1}))$$
and hence the remaining trajectory has no contribution.
Another way of thinking this is you are taking weighted trajectories till time step $t+k-1$ weighted by rewards $R_{t+k}$ and hence you cannot sum up to 1. The rest of the trajectory after $t+k-1$ will sum up to 1.
I hope this qualitative description suffices. You can do the maths, but you must be careful with the notations and the assumptions you make.
Also all the equations are correct, I hope you can indirectly see it from my reasoning.


Maximum Q value for new state in Q-Learning never exists I'm working on implementing a Q-Learning algorithm for a 2 player board game.

I encountered what I think may be a problem. When it comes time to update the Q value with the Bellman equation (above), the last part states that for the maximum expected reward, one must find the highest q value in the new state reached, s', after making action a.

However, it seems like the I never have q values for state s'. I suspect s' can only be reached from P2 making a move. It may be impossible for this state to be reached as a result of an action from P1. Therefore, the board state s' is never evaluated by P2, thus its Q values are never being computed.

I will try to paint a picture of what I mean. Assume P1 is a random player, and P2 is the learning agent.

P1 makes a random move, resulting in state s.
P2 evaluates board s, finds the best action and takes it, resulting in state s'. In the process of updating the Q value for the pair (s,a), it finds maxQ'(s', a) = 0, since the state hasn't been encountered yet.
From s', P1 again makes a random move.

As you can see, state s' is never encountered by P2, since it is a board state that appears only as a result of P2 making a move. Thus the last part of the equation will always result in 0 - current Q value.

Am I seeing this correctly? Does this affect the learning process? Any input would be appreciated.

Thanks.

Your problem is with how you have defined $s'$.

The next state for an agent is not the state that the agent's action immediately puts the environment into. It is the state when it next takes an action. For some, more passive, environments, these are the same things. But for many environments they are not. For instance, a robot that is navigating a maze may take an action to move forward. The next state does not happen immediately at the point that it starts to take the action (when it would still be in the same position), but at a later time, after the action has been processed by the environment for a while (and the robot is in a new position), and the robot is ready to take another action.

So in your 2-player game example using regular Q learning, the next state $s'$ for P2 is not the state immediately after P2's move, but the state after P1 has also played its move in reaction. From P2's perspective, P1 is part of the environment and the situation is no different to having a stochastic environment.

Once you take this perspective on what $s'$ is, then Q learning will work as normal.

However, you should note that optimal behaviour against a specific opponent - such as a random opponent - is not the same as optimal play in a game. There are other ways to apply Reinforcement Learning ideas to 2-player games. Some of them can use the same approach as above - e.g. train two agents, one for P1 and one for P2, with each treating the other as if it were part of the environment. Others use different ways of reversing the view of the agent so that it can play versus itself more directly - in those cases you can treat each player's immediate output as $s'$, but you need to modify the learning agent. A simple modification to Q learning is to alternate between taking $\text{max}_{a'}$ and $\text{min}_{a'}$ depending on whose turn you are evaluating (and assuming P1's goal is to maximise their score while P2's goal is to minimise P1's score - and by extension maximise their own score in any zero-sum game)


Does IBM Watson use machine learning? I was reading an article on Medium and wanted to make it clear whether a bot created on IBM Watson is an intelligent one or unintelligent.

  Simply put, there are 2 types of chatbots — unintelligent ones that act using predefined conversation flows (algorithms) written by developers building them and intelligent ones that use machine learning to interact with users.

Yes, it does and at many parts of the solution. For one of the core components - intent detection - Intento did a benchmark comparing IBM Watson and other similar products: https://www.slideshare.net/KonstantinSavenkov/nlu-intent-detection-benchmark-by-intento-august-2017

Outside of intent detection, there are other areas where AI techniques help - e.g. disambiguation, bootstrapping a bot from chat logs etc. Specifically for IBM Watson, you can learn more here: https://www.ibm.com/cloud/watson-assistant/features/


How do I combine models trained on different data to increase classification accuracy? I have two trained models. One is using a LinearSVC algorithm and is trained on numerical data from medical examination from patients with diabetic retinopathy. The second one is a neural network trained on images of retina scans from patients with the same disease.

The models predict if the patient has retinopathy or not. Both are written using Python 3.6 and Keras and have accuracy around 0.84.

Is it possible to combine those two models in any way to increase the accuracy of predictions?

I'm not sure in what way it could be achievable as they are using a different kinds of data. I have tried using ensembling methods but didn't get better scores with them.

You can try using a multi-input model. Here is a recent post with a similar discussion, with the required architecture defined in the answer.

Instead of combining the separate models, you can create a model which uses image and numerical data side by side. Keras allows you to use different types of data using multi input structure via functional API. And then you can combine them to create a single machine learning model. Basic idea is like this:

This image was taken from here, where you can find further details and the code implementation as well. Actually, the discussion in the introduction of that post is almost the same as your question: What to do when the inputs are:

  Numeric/continuous values, such as age, heart rate, blood pressure
  Categorical values, including gender and ethnicity
  Image data, such as any MRI, X-ray, etc.

Also, section 5.1 of this blogpost details the same process.


How would an AI understand grids? OK, now I think an AI must view grids in a different way to computers.

For example a computer would represent a grid like this:

cells = [[1,2,3],[4,5,6],[7,8,9]] = [row1,row2,row3]

That is a grid is 3 rows of 3 cells.

But... that's not how a human sees it. A human sees a grid as made of 3 rows and 3 collumns somehow intersecting.

If an AI is built on some mathematical logic like set theory, it's like a set of rows which in turn is a set of cells.

So what would be a way to represent a grid in a computer that is more "human". And doesn't favor either rows or columns? Or is there some mathematical or programmatical description of a grid that treats rows and columns as equivalent?

Although it is common to represent a grid as two dimensional array in a computer program, this is not the only way to represent one. You could, for example, use a generalized graph structure made of linked nodes, with 4 links each. Many other representations are possible. Even if you use a 2d array, some languages would index the columns first rather than the rows (FORTRAN for instance).

That issue is really an aside though, because it conflates representation with reasoning. By choosing to represent the world in certain data structures, we can made it easier or harder for different AI algorithms to reason about them, but the reasoning can often proceed even if the representation is inefficient. That is, an AI algorithm can ask questions about the columns in the grid, even if the columns of the grid are not represented in a way that makes it programmitcally easy to group them together.


Natural language recommendation system: to pre-classify inputs or not? Does it help to "pre-classify" natural language inputs using labeled input fields? E.g., "Who," "What," "Where," "When," "Why," "How," and "How much?" Or is a single, monolithic, free-form, long-text input field equally effective and efficient for model training purposes?
Scenario 1: Without input labels

We are three research fellows, Alice, Bob and Charlie at the University of Copenhagen. We want to understand the development of the human visual system. This knowledge will help in the prevention and treatment of certain vision problems in children. Further, the rules that guide development in the visual system can be applied to other systems within the brain. Our work, therefore, has wide application to other developmental disorders affecting the nervous system. We will conduct this research in 2019 under a budget of $15,000.

Scenario 2: With input lables

Who: We are three research fellows, Alice, Bob and Charlie.
What: We want to understand the development of the human visual system.
Where: At the University of Copenhagen.
When: During the calendar year of 2019.
Why: This knowledge will help in the prevention and treatment of certain vision problems in children.
How: Further, the rules that guide development in the visual system can be applied to other systems within the brain.
How Much: The research will cost $15,000.

Use Case:
I am building an AI/ML recommendation system. Users subscribe to the system to get recommendations of research projects they might like to participate in or fund. There will be many projects from all over the globe. Far too many for a human to sort through and filter. So AI will sort and filter automatically.
Will pre-classifying input fields using labels help the training algorithm be more efficient or effective?

Likely yes!  When you split out the inputs like this, you are adding information.  How much this helps is an open question till you build your system, get some data and start training.

Of course, it would be marvelous to have a machine do all the work straight unstructured text - but you want a functional, easy-to-use website, not a research project of your own.  To that end, do everything that you can to constrain the scope of the problem, and maximize the information available to your model.  For example, you might want to see if you can add researchers using Google Scholar (so you can link to their profiles, and perhaps mine some information that way).

You'll be somewhat 'data constrained' till you've got a decent number of research proposals and user interactions to learn from.  Tools like our NLP architect may help you get more out of your text (there are some other really cool new-generation ML-for-NLP packages you should also evaluate).


Similarities and differences between UCT algorithms in (i), (ii), (iii) and (iv)? I am trying to understand the similarities and differences between: (i) the UCT algorithm in Kocsis and Szepesvári (2006); (ii) the UCT algorithm in Section 3.3 of Browne et al (2012); (iii) the MCTS algorithm in Silver et al. (2016); (iv) the MCTS algorithm in Silver et al. (2017).

I would be really grateful for some help identifying the similarities and differences in these papers, I am doing some research and really struggling right now.

(i) http://ggp.stanford.edu/readings/uct.pdf

(ii) http://mcts.ai/pubs/mcts-survey-master.pdf (Section 3.3)

(iii) https://storage.googleapis.com/deepmind-media/alphago/AlphaGoNaturePaper.pdf

(iv) https://deepmind.com/documents/119/agz_unformatted_nature.pdf

The algorithm in the 2012 survey article (your second link) is the most common / standard implementation. Whenever someone mentions using MCTS or UCT, without explicitly stating any other info, it's safe to assume that that pseudocode is what they're using.

The paper by Kocsis and Szepesvári from 2006 (your first link) is (one of) the original publication(s) on UCT. It is very similar to the "standard" implementation as described in the survey paper. If I recall correctly, the only important difference is that the algorithm as described in 2006 keeps track of at which point in time during an episode a reward is observed, and accounts for that timing in the backpropagation phase (i.e. if a reward is observed at time $t$ in an episode, credit for that reward is not assigned to states/action after time $t$). It also used a discounting factor $\gamma$ to discount the importance of reward depending on temporal distance, which is uncommon otherwise in MCTS literature.

Both of those differences are due to the original, 2006 paper having more of a "Markov decision process" or "Reinforcement Learning" flavour, whereas otherwise MCTS has especially become popular in AI for (board) games where we often by default assume that there is only a single nonzero reward (win or lose) at the end of every episode anyway, which makes those differences less meaningful.

Both of the AlphaGo papers (AlphaGo and AlphaGo Zero) use a "foundation" of MCTS that is mostly similar to the one from the 2012 survey article. The core components are all the same. Both of those systems add on a lot of (very important, and some quite complex) bells and whistles though.

Going off the top of my head (should be mostly accurate, but best details are in the original source of course!) AlphaGo (your third link) added Neural Networks trained in various ways to output policies (mapping from states to probability distributions over actions) and value functions (mapping from states to "value estimates", or estimates of win percentage). Trained policy networks were used in a variant of the "selection" phase (no longer UCB1 strategy as in UCT) to bias selection. A different (simpler function, not a deep net) policy was used to run play-outs (no long uniform at random action selection as in UCT). A combination of observed reward at end of play-out + value function estimate at start of play-out was used for backpropagation (no longer only backpropagating reward observed in play-out as in UCT).

AlphaGo Zero is, if we're purely looking at the MCTS part, quite similar to AlphaGo. The Neural Networks were a bit different (a single one, with multiple outputs for policy + value), and play-outs were no longer used at all (just immediate backpropagation of value function estimates after MCTS' selection + expansion phases). Apart from that, the primary differences going from AlphaGo to AlphaGo Zero were in the learning process used to train the Neural Networks really, not so much in the MCTS side of things.


What kind of functions can be used as activation functions? I read that functions are used as activation functions only when they are differentiable. What about the unit step activation function? So, is there any other reason a function can be used as an activation function (apart from being differentiable)?

Not completely sure your question. Do you mean

Q. why should we use activation function?

Ans: we need to introduce non-linearity to the network. Otherwise, multiple layers are no difference from single layer network. (It is obvious as we write things in matrix form, and say when we have two layers with weights $W_1$ and $W_2$, the two layer is no difference from a single layer with weight $W_2 W_1$.

Q. why they need to be differentiable?

Ans: Just for sake that we can back-propagate gradients back to earlier layers. Note that back-propagation is nothing but the chain rule in calculus. Say $f(\cdot)$ is an activation function in one layer and the output of that activation function is $\bf y$ and the input is ${\bf u}=W \bf x$, where $\bf x$ is output from the previous layer and mix with weights $W$ in current layer. Of course, the final loss $L$ will depend on ${\bf y} = f({\bf u})= f(W {\bf x})$. Say, loss $L=g(\bf y)$ somehow. To train the weights $W$, we have to find the gradient $\frac{\partial L}{\partial W}$ so that we can adjust weight $W$ to minimize $L$. But $\frac{\partial L}{\partial W}=\frac{\partial g({\bf y})}{\partial W}=\frac{\partial g({\bf y})}{\partial \bf y}\frac{\partial {\bf y}}{\partial {\bf u}}\frac{\partial {\bf u}}{\partial W}$. Each of these product terms can be computed locally and will be accumulatively multiplied as we apply backprop.  And note that the middle term $\frac{\partial {\bf y}}{\partial {\bf u}}=\frac{\partial f({\bf u})}{\partial {\bf u}}$ is just "derivative" of $f(\cdot)$, thus we require the activation function to be differentiable and "informative"/non-zero (at least most of the time). Note that ReLU is not differentiable everywhere and that is why researchers (at least Yoshua Bengio) worried about that when they first tried to adopt ReLU. You may check out the interview of Bengio by Andrew Ng for that.

Q. why step function is a bad activation function?

Ans: Note that step function is differentiable almost everywhere but is not "informative" though. For places (flat region) where it is differentiable, the derivative is simply zero. Consequently, any later layer gradient (information) will get cut off as it passes through a step function activation function.


Is input normalization built-in into mammals sensory neurons? The spectrum of human sensory inputs seems to fall within certain ranges suggesting normalization is built-in into biological NNs?

It also adapts to circumstantial conditions, e.g. people living in a city with certain factory smell eventually don't perceive the smell anymore, at least consciously (within working memory) / it adapts to a new baseline?

Yes, for many sensory inputs there is indeed something similar to normalization. But its not rally the same as in classical data analytics compared to what eg min/max normalization does or other technics.

Lets look on some examples and considerations:

mammals don't perceive heat or loudness in a linear way. This is because already many sensory receptors have chemical / physical limits. Double decibels will not perceived with double intensity. Inside your ear, the small hammer and abil will brace to protect you. --> its like normalization with logarithmic effects applied.
heat perception is more like a difference integration than a absolute temperature measurement. Its measured via H+ ions flow in mitochondria in the cell (if i recall correctly)
On the neuronal side gradual signals in the dendrites (analog signal) sum up gradually to later form an spike at the axon hill. where in turn a fire frequency is then encoded - the maximum frequency of this serves as a a natural maximum limit. I remember that grasshoppers increase axon fire frequency when objects started covering more ommatidial area on their "eye". The more of their "eyes" are covered by the shadow the more input on the neuron --> higher fire rate.
a lot of sensory input is post processed in higher cerebral areas. Eg. compared to what is expect able and heuristics are applied to compare a signal with former events.
when doing computational data analysis we may want go for accuracy and maximum comparability. Mostly on all data that could be available. --> eg. with respect to properties of a standard normal distribution. Hence we put some effort to be accurate and know the true parameters, remove outliers and so on --> big data comes into play here.
Nature in contrast strives often for efficiency with the means of reaching the minimal required with minimal resources.

Summary:
Compared to normalization in an analytical sense (eg. mean, min-max or other feature normalization techniques), nature is often only interested in the current difference between stimuli. And this only within some relevant range. Other information is not integrated. And normalization with the goal of making measurement points comparable only happens within this range of the mapping function provided by the sensor/neuron/receptor whatever!

So this should also answer your question about, why you are not smelling something in the city after a while any more. However, this for sure happens at higher cerebral regions (it might also be that your smell receptors saturate) its the same principle. Your consciousness just saves energy by not concentrating on something that is anyway not changing.

If you want to read more have a look here: https://en.wikipedia.org/wiki/Weber%E2%80%93Fechner_law


What's going on in the equation of the variational lower bound? I don't really understand what this equation is saying or what the purpose of the ELBO is. How does it help us find the true posterior distribution?

From this document, as you found here, $X$ is an observed variable and $Z$ is a hidden variable; $p(X)$ is the density function of $X$. The posterior distribution of the hidden variables can then be written as follows using the Bayes’ Theorem:

$$p(Z|X) = \frac{p(X|Z)p(Z)}{p(X)} = \frac{p(X|Z)p(Z)}{\int_Zp(X,Z)}$$

Now base on what you post, if we denote that $L= \mathbb{E}_q [\log p(X, Z)] + H[Z]$ ($q(Z)$ is a distribution we use to approximation the true posterior distribution $p(Z|X)$ in VB and $H[Z] = -\mathbb{E}_q [\log q(Z)]$), then it is obvious that $L$ is a lower bound of the log probability of the observations.
As a result, if in some cases we want to maximize the marginal probability (the log probability of the observations), we can instead maximize its variational lower bound $L$. As a real example, you can follow the "Multiple Object Recognition with Visual Attention" example in the referenced document.

Moreover, the term $L$ will be presented in KL-divergence that will be used to measuring the similarity of two distributions. Be aware that there is progress on the bound in this paper (Fixing a Broken ELBO).


How can I use deep reinforcement learning for vehicle rerouting in SUMO? I want to use deep reinforcement learning for vehicle rerouting in SUMO, but I don't know how to start training the model.

I've already created road network and vehicle routing in SUMO-XML files (mymap.net.xml and mymap.rou.xml). Currently, I'm trying to train the model on Jupyter Notebook, importing TraCI library to control the SUMO simulator and allow for a reinforcement learning approach. However, I'm still confused in training step.

Do I need any traffic data to train my agent to take actions in the environment?
How can I train based on these SUMO-XML files I created?
Is it possible to run the simulation on Windows? or I need to change to Ubuntu instead?

I would appreciate if someone could guide me. Thank you in advance.

There is a platform called Flow, sounds like its what you're looking for

https://github.com/flow-project/flow


How to add external training in chatterbot? I created a very simple bot to learn how to use chatterbot. This library already comes with a training, but I wanted extra training with an import of a corpus in Portuguese that I found in github.

from chatterbot import ChatBot

bot = Futaba(
"Terminal",
storage_adapter="chatterbot.storage.SQLStorageAdapter",
logic_adapters=[
"chatterbot.logic.MathematicalEvaluation",
"chatterbot.logic.TimeLogicAdapter",
"chatterbot.logic.BestMatch"
],

input_adapter="chatterbot.input.TerminalAdapter",
output_adapter="chatterbot.output.TerminalAdapter",
database_uri="../database.db"
)

print("Type something to begin...")

while True:
    try:
        bot_input = bot.get_response(None)
    except (KeyboardInterrupt, EOFError, SystemExit):
        break

That's all I have.

How can I import this corpus into my chatbot?

I'm not sure if this is the right one, but according to the documentation you need to create a trainer for it:

from chatterbot.trainers import ChatterBotCorpusTrainer

# Create a new trainer for the chatbot
trainer = ChatterBotCorpusTrainer(chatbot)

# Train based on the english corpus
trainer.train("chatterbot.corpus.english")

Instead of "chatterbot.corpus.english" you obviously choose your own training data.


How to design an AI that discovers more complex concepts on its own? How would I go about designing a (relatively) simple AI that discovers and invents random more complex concepts on its own?

For example, say I had a robot car. It doesn't know it's a car. It has several inputs and outputs, such as a light sensor and the drive motors. If it stays in the dark, it's score drops (bad), and if it moves into the light, it's score rises (good). It'd have to discover that it's motor outputs cause the light input to change (because it's moving closer or farther away from a light source), and that brighter light means higher score.

Of course, it'd be easier to design an AI that does specifically that, but I want its behaviour discovery system to be more generic, if that makes any sense. Like later on, it could discover a way to fight or cooperate with other robots to increase its score (maybe other robots destroy light sources when they drive over them, and they can be disabled by driving into them), but it'd have to discover this without initially knowing that another robot could possibly exist, how to identify one, what they do, and how to interact with one.

Also, I want it to be creative instead of following a 'do whatever is best to increase your score' rule. Like maybe one day it could decide that cooperating with other robots is another way to increase its score (it finds out what love is), but if it's unable to do that, it becomes depressed and stops trying to increase it's score and just sits there and dies. Or it could invent any other completely random and possibly useless behavior.

How hard would it be to make something like this, that essentially builds itself up from a very basic system, provided I give it lots of different kinds of inputs and outputs that it can discover how to use and apply to its own evolving behavior?

This is not an answer. I couldn't comment, so here are some remarks about your question: This is a very broad question, and considered The Holy Grail for building artificially intelligent systems - meaning that some scientists have been dreaming about this since time immemorial.

Some homework is warranted from your side; you could have offered some of your solutions or maybe identified the multiple layers to your query, as they invoke concepts from many fields within the study of AI (or AGI to be general).

For example, the following layers are taken up around the big-words in the question, though rhetorically -

On evolution - what registers as evolution (is only genetic mutation evolution or should it involve some form of natural selection or seeking a niche for increasing the chances of survival or etc). How would evolution of software look like? Should it be able to modify its own code in the process of evolution?
On concept - what constitutes a concept, identifying from the environment a concept by calculating its relevance (w.r.t other concepts), coming with a process of selecting the concept for use in a given environment (natural or artificial).Referring to the example in the question, is the road a relevant concept or the trees and the sky and the bees pollinating some flowers along the roadside? What is a more fundamental concept - the trees or the bees, and how does one measure that?
On rewards - for us humans, rewards are maximization of survival of genetic machinery they carry (translating to reproductive success). what reward system should we come up that machines could use to increase their chances of survival in the physical world? what value should be put on a reward for moving the car on a straight line, or moving from darkness towards light? Shouldn't moving from darkness to light be more fundamental action (concept) and therefore rewarded higher than moving along the straight line? However, given that the car has learned to move from darkness to light, shouldn't the value of the reward be lowered so that it can learn other actions/concepts?

As can be seen from this brief detailing, there are layers within layers. Therefore, it is proper to establish now that there are no simple answer to this essay-requiring soul-searching time-eating heavy-research-wanting question. However appropriate directions can be given about work that is being done by some of the most prominent scientists of our times. The thing that one is looking for are known as Universal Problems Solvers, such as Gödel Machine (by Jürgen Schmidhuber) and AIXI - Artificial Intelligence (AI) based on Solomonoff's distribution ξ (by Markus Hutter).

Here is a quote lifted from Wikipedia page on AIXI that is pretty self-explanatory on how it maximizes the rewards over time.

  AIXI is a reinforcement learning agent. It maximizes the expected
  total rewards received from the environment. Intuitively, it
  simultaneously considers every computable hypothesis (or environment).
  In each time step, it looks at every possible program and evaluates
  how many rewards that program generates depending on the next action
  taken. The promised rewards are then weighted by the subjective belief
  that this program constitutes the true environment. This belief is
  computed from the length of the program: longer programs are
  considered less likely, in line with Occam's razor. AIXI then selects
  the action that has the highest expected total reward in the weighted
  sum of all these programs.

Gödel Machine goes further - it allows the agent to modify its own code that allows it to maximise the rewards of its actions - which is to modify its own code that allows it to maximize the rewards of its actions - as so on. This is kind of a recursive definition, that simulates evolution (a rapid one) by choosing to evolve that code/state of the agent that converges towards a code that is superior to the code that the agent is currently running.

Here is a quote lifted from page on summary of Gödel machine. See that Hutter is also referenced (discovered of AIXI above).

  Our Gödel machine will never get worse than its initial problem
  solving strategy, and has a chance of getting much better, provided
  the nature of the given problem allows for a provably useful rewrite
  of the initial strategy, or of the proof searcher. The Gödel machine
  may be viewed as a self-referential universal problem solver that can
  formally talk about itself, in particular about its performance. It
  may "step outside of itself" (Hofstadter, 1979) by rewriting its
  axioms and utility function or augmenting its hardware, provided this
  is provably useful. Its conceptual simplicity notwithstanding, the
  Gödel machine explicitly addresses the `Grand Problem of Artificial
  Intelligence' by optimally dealing with limited resources in general
  environments, and with the possibly huge (but constant) slowdowns
  buried by previous approaches (Hutter, 2001, 2002) in the widely used
  but sometimes misleading O()-notation of theoretical computer science.

  The main limitation of the Gödel machine is that it cannot profit from
  self-improvements whose usefulness it cannot prove in time.

Reading the papers of the stated research would certainly be useful in finding answers to some of the layers in the question. There is also good work going on in the AGI community that somewhat aligns with the direction. Hope this helps.


Implementing experience replay in reinforcement learning I've been reading Google's DeepMind Atari paper and I'm trying to understand how to implement experience replay. My question is that whether we update the parameters $\theta$ of function $Q$ once for all the samples of the minibatch or we do that for each sample of the minibatch separately?

According to the following code from this paper, it performs the gradient descent on loss term for the $j$-th sample. However, I have seen other papers (referring to this paper) that say that we first calculate the sum of loss terms for all samples of the minibatch and then perform the gradient descent on this sum of losses.

Gradient descent should be performed using the sum (or average) of the losses in the minibatch.

This is in fact also how I read the pseudocode in your question, though I understand it can be confusing. Note that, in the pseudocode, j is not specified in detail. They do not, for example, have j ranging from 0 to the size of the minibatch.

When they say:

Sample random minibatch of transitions (phi_j, a_j, r_j, phi_{j+1}) from D

they mean multiple transitions in the minibatch (with minibatch size of 1 being a special case), and they use the index j to collectively refer to the entire set of indices in that randomly sampled minibatch. It's not one particular number / index, j is a set of indices. When further lines of code do something with index j, they actually do something with all indices j.


When are multiple hidden layers necessary? I know that my question probably seems like being asked many times, but Ill try
to be more speciffic:
Limitations to my question:

I am NOT asking about convolutional neural networks, so please, try not to mention this as an example or as an answer as long as it is possible. (maybe only in question number 3)

My question is NOT about classification using neural networks

I am asking about a "simple" neural network designed to solve the regression type of problem. Let's say it has 2 inputs and 1 output.

Preambula:
As far as I understood, from the universal approximation theorem, in such a case, even if the model is nonlinear, only one hidden layer can perfectly fit a nonlinear model, as shown here
http://neuralnetworksanddeeplearning.com/chap4.html.
Question 1
In this specific case, is there any added value in using extra layers?
(maybe the model will be more precise, or faster training?)
Question 2
Suppose in 1st question the answer was there is no added value. In such a case will the added value appear if I enlarge inputs from two inputs as described above, to some larger number?
Question 3
Suppose in 2nd question the answer was there is no added value. I am still trying to pinpoint the situation where it STARTS making sense in adding more layers AND where it makes NO sense at all using one layer.

A very wide but shallow neural network is going to be harder to train.
You can check that with the playground of tensorflow or with the MPG example in Google Colab.
The relationship between architecture and learning capabilities is not fully understood, but, empirically, thats what you see.
But making the network too deep creates more problems:

Vanishing gradients.
Less parameters for the same amount of neurons.
Exploding gradients.

It is for that reason that humans and neural networks are the one deciding a good architecture.


When would bias regularisation and activation regularisation be necessary? For Keras on TensorFlow, a layer class constructor comes with these:

kernel_regularizer=...
bias_regularizer=...
activity_regularizer=...

For example, Dense layer:
https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense#arguments_1
The first one, kernel_regularizer is easy to understand, it regularises weights, makes weights smaller to avoid overfitting on training data only.
Is kernel_regularizer enough? When should I use bias_regularizer and activity_regularizer too?

Regularizer's are used as a means to combat over fitting.They essentially create a cost function penalty which tries to prevent quantities from becoming to large.  I have primarily used kernel regularizers. First I try to control over fitting using dropout layers. If that does not do the job or leads to poor training accuracy I try the Kernel regularizer. I usually stop at that point. I think activity regularization would be my next option to prevent outputs from becoming to large. I suspect weight regularization effectively can pretty much achieve the same result.


Is the negative of the policy loss function in a simple policy gradient algorithm an estimator of expected returns? Let

$$
\nabla_\theta J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t = 0}^T \nabla_\theta \log \pi_\theta (a_t|s_t) R(\tau) \right]
$$
be the expanded expression for a simple policy gradient, where $\theta$ are the parameters of the policy $\pi$, $J$ denotes the expected return function, $\tau$ is a trajectory of states and actions, $t$ is a timestep index, and $R$ gives the sum of rewards for a trajectory.

Let $\mathcal{D}$ be the set of all trajectories used for training. An estimator of the above policy gradient is given by

$$
\hat{g} = \frac{1}{\mathcal{D}} \sum_{\tau \in \mathcal{D}} \sum_{t = 0}^T \nabla_\theta \log \pi_\theta (a_t|s_t) R(\tau).
$$
A loss function associated with this estimator, given a single trajectory with $T$ timesteps, is given by
$$
L(\tau) = -\sum_{t = 0}^T \log \pi_\theta (a_t|s_t) R(\tau).
$$
Minimizing $L(\tau)$ by SGD or a similar algorithm will result in a working policy gradient implementation.

My question is what is the proper terminology for this loss function? Is it an (unbiased?) estimator for the expected returns $J(\pi_\theta)$ if summed over all trajectories? If someone is able to provide a proof that minimizing $L$ maximizes $J(\pi_\theta)$, or point me to a reference for this, that would be greatly appreciated.

If I understand your question correctly, you are wondering whether the
policy gradient objective coincides with some real measure of progress.
This is exactly what the Policy Gradient Theorem proves (see Sutton et
al. (2000) or Sutton and Barto (2018), chapter 13). In particular,
policy gradient methods optimize the value of the start state $s_0$ under the
current policy, $v_\pi(s_0)$. Since this value is defined as an
expectation over returns, then your conclusion is correct.

Sutton, Richard S., and Andrew G. Barto. 2018. Reinforcement Learning -
an Introduction. Adaptive Computation and Machine Learning. MIT Press.
http://www.worldcat.org/oclc/37293240.

Sutton, Richard S, David A McAllester, Satinder P Singh, and Yishay
Mansour. 2000. “Policy Gradient Methods for Reinforcement Learning with
Function Approximation.” In Advances in Neural Information Processing
Systems, 1057–63.


Viola Jones Algorithm Can Viola Jones algorithm be used to detect the facial emotion. Actually it was used in creating harr-cascade file for object and facial detection, but what confused me is whether it can be used to train for emotion detection.

If not, what algorithms can I use? and what are the mathematical bases? (i.e. what mathematics should I be studying?)

An introduction to the Haar features is provided in the youTube video.  The video indicates the VJ face detector leverages a selected combination of Haar features (convolutional kernels) to detect facial features (weak classifiers), such as the nose bridge.  The binary presence of the weak classifiers are summed to determine if the window contains a face.

The ability for a VJ algorithm to detect emotion would rely on the ability to assign a set of Haar features (kernels) to recognize features associated with a particular emotion label (surprise, anger, content, fear).

It is conceivable that the initial stage of an  emotion classifier could use a the VJ algorithm to identify a face for additional stages to classify emotion.


Can I start with perfect discriminator in GAN? In many implementations/tutorials of GANs that I've seen so far (e.g. this), the generator and discriminator start with no prior knowledge. They continuously improve their performance with training. This makes me wonder — is it possible to use a pre-trained discriminator? I have two motivations for doing so:

Eliminating the overhead of training the discriminator
Being able to use already existing cool models

Would the generator be able to learn just the same, or is it dependent on the fact that they start from scratch?

If you start with perpect discriminator, loss function will be saturated, and gradient of loss will be very small, so feedback for the generator also will be small, and learning will be slow down as a result. Actually, it is allways desired for discriminator and generator to learn balancedly. Additionally, it is claimed that Wasserstein Loss take care of this problem.
You can find more information in this article. (I strongly suggest to read)
Also from the paper " Towards principled methods for training generative adversarial networks":

In theory, one would expect therefore that we would first train the discriminator as close as we can to optimality (so the
cost function on $\theta$ better approximates the JSD), and then do gradient steps on  $\theta$, alternating these
two things. However, this doesn’t work. In practice, as the discriminator gets better, the updates
to the generator get consistently worse. The original GAN paper argued that this issue arose from
saturation, and switched to another similar cost function that doesn’t have this problem. However,
even with this new cost function, updates tend to get worse and optimization gets massively unstable.

Note 1: $\theta$ is parameters of generator
Note 2: JSD: Jensen–Shannon divergence. For optimal discrimantor, loss is equal to $L(D^{ \ast }, g_{\theta})=  2 JSD(P_{r}, P_{g}) - 2log(2)$
Also from the paper of Wasserstein Loss:

The fact that the EM distance is continuous and differentiable a.e. means that
we can (and should) train the critic till optimality. The argument is simple, the more we train the critic, the more reliable gradient of the Wasserstein we get, which is actually useful by the fact that Wasserstein is differentiable almost everywhere. For the JS, as the discriminator gets better the gradients get more reliable but the true gradient is 0 since the JS is locally saturated and we get vanishing gradients ... In Figure 2 we show a proof of concept of this, where we train a GAN discriminator and a WGAN critic till optimality. The discriminator learns very quickly to distinguish between fake and real, and as expected provides no reliable gradient information. The critic, however, can’t saturate, and converges to a linear function that gives remarkably clean gradients everywhere.


Why don't people use projected Bellman error with deep neural networks? Projected Bellman error has shown to be stable with linear function approximation. The technique is not at all new. I can only wonder why this technique is not adopted to use with non-linear function approximation (e.g. DQN)? Instead, a less theoretical justified target network is used.

I could come up with two possible explanations:

It doesn't readily apply to non-linear function approximation case (some work needed)
It doesn't yield a good solution. This is the case for true Bellman error but I'm not sure about the projected one.

I have found some clues in Maei's thesis (2011): “Gradient Temporal-Difference Learning Algorithms.”

According to the thesis:

GTD2 is a method that minimizes the projected Bellman error (MSPBE).
GTD2 is convergent in non-linear function approximation case (and off-policy).
GTD2 converges to a TD-fixed point (same point as semi-gradient TD).
GTD2 is slower to converge than usual semi-gradient TD.

  It doesn't readily apply to non-linear function approximation.

No, it does.

  It doesn't yield a good solution.

No, it does. TD-fixed point is the same point for the solution of semi-gradient TD (which is generally used). There is no edge on that.

The only explanation seems to be practical convergence rate.

To quote his words:

  Some of our empirical results suggest that gradient-TD method maybe slower than conventional TD methods on problems on which conventional TD methods are sound (that is, on-policy learning problems).