Can we say: the more we increase the numbers of cross validation the less likely it is that we overfit? Based on the answer of my previous question:
How can I avoid overfitting when doing parameter tuning?

Can we say: the more we increase the numbers K of cross validation the less likely it is that we overfit?

In general, no.

There is a tradeoff between making the validation set for each fold smaller, and having more folds in total.

As an example, if you have $N$ folds for $N$ datapoints, each fold will have only a single datapoint in its validation set. The validation accuracy of a model on a single datapoint is not a reliable estimator for the test performance of the model. In fact, you can construct examples where the error is arbitrarily large.

For this reason, people sometimes use Bootstrap Validation if they need a very large number of folds. In practice though, most people just us 10 folds, and that's "good enough".


How to deal with invalid output in a policy network? I am interested in creating a neural network-based engine for chess. It uses a $8 \times 8 \times 73$ output space for each possible move as proposed in the Alpha Zero paper: Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm.

However, when running the network, the first selected move is invalid. How should we deal with this? Basically, I see two options.

Pick the next highest outputted move, until it is a valid move. In this case, the network might automatically over time not put illegal moves on top.
Process the game as a loss for the player who picked the illegal move. This might have the disadvantage that the network might be 'stuck' on only a few legal moves.

What is the preferred solution to this particular problem?

You should have a method to generate a possible moves output based on the board state. Use this as a mask before normalization in the policy head.


What are some conferences for publishing papers on Deep Learning for Human Activity recognition? What are some conferences for publishing papers on Deep Learning for Human Activity recognition? Do any of the major conferences have specific tracks for Human Activity Recognition?

If your using computer vision, the top recognised conference is CVPR (computer vision and pattern recognition).
You can also try to submit at ICML (International conference on machine learning) and NIPS (Neural Information Processing Systems), which focuses on applications of machine learning and deep learning.
I'd also recommend IJCAI (International joint conference on artificial intelligence).


How to train a neural network for a round based board game? I'm wondering how to train a neural network for a round based board game like, tic-tac-toe, chess, risk or any other round based game.
Getting the next move by inference seems to be pretty straight forward, by feeding the game state as input and using the output as the move for the current player.
However training an AI for that purpose doesn't appear to be that straight forward, because:

There might not be a rating if a single move is good or not, so training of single moves doesn't seem to be the right choice
Using all game states (inputs) and moves (outputs) of the whole game to train the neural network, doesn't seem to be the right choice as not all moves within a lost game might be bad

So I'm wondering how to train a neural network for a round based board game?
I would like to create a neural network for tic-tac-toe using tensorflow.

Great question!  NN is very promising for this type of problem: Giraffe Chess.  Lai's accomplishment was considered to be a pretty big deal, but unfortunately came just a few months before AlphaGo took the spotlight. (It all turned out well, in that Lai was subsequently hired by DeepMind, although not so well for the Giraffe engine;)

I've found Lai's approach to be quite helpful, and it is backed by solid results.

You may want to use "sequential" as opposed to "round based" since sequential is the preferred term in Game Theory and Combinatorial Game Theory, and these are the fields that apply mathematical analysis to games.

The games you list are further termed "abstract" to distinguish them from modern strategy boardgames, or games in general, which utilize a strong theme and are generally less compact than abstract games in terms of mechanics and elements. This carries the caveat that abstract games are not restricted to sequential games or boardgames, or even games specifically, as in the case of puzzles like Sudoku.

The formal name for this group of games is generally "partisan, sequential, deterministic, perfect information" with the further categorization of Tic-Tac-Toe as "trivial" (solved and easily solvable) and non-trivial (intractable and unsolved) for games like Chess and Go.


Unable to overfit using MLP I'm building a 5-class classifier with a private dataset. Each data sample has 67 features and there are about 40000 samples. Samples of a particular class were duplicated to overcome class imbalance problems (hence 40000 samples).

With a one-vs-one multi-class SVM, I am getting an accuracy of ~79% on the validation set. The features were standardized to get 79% accuracy. Without standardization, the accuracy I get is ~72%. Similar result when I tried 50-fold cross validation.

Now moving on to MLP results,

Exp 1:

Network Architecture: [67 40 5]
Optimizer: Adam
Learning Rate: exponential decay of base learning rate
Validation Accuracy: ~45%
Observation: Both training accuracy and validation accuracy stops improving.

Exp 2:
Repeated Exp 1 with batchnorm layer

Validation Accuracy: ~50%
Observation: Got 5% increase in accuracy.

Exp 3:

To overfit, increased the depth of MLP. A deeper version of Exp 1 network

Network Architecture: [67 40 40 40 40 40 40 5]
Optimizer: Adam
Learning Rate: exponential decay of base learning rate
Validation Accuracy: ~55%

Thoughts on what might be happening?

I guess you are using linear activation functions,  maybe you are not initializing your weights, or you are regularizing your model enough.

Initialize weights with glorot, insert dropout layers in between, use Relu as your activation function, stop the training process based on Early Stopping. and just experiment with one hidden layer.

side note: if you use Adam, don't mess with the learning rate.
in SGD optimizer you could use decay because there is a single learning rate for all weight updates and the learning rate does not change during training.
In Adam, the learning rate is maintained for each network weight (parameter) and separately adopted as learning unfolds.


Is this learning rate schedule increasing the learning rate? I was reading a PyTorch code then I saw this learning rate scheduler:

def warmup_lr_scheduler(optimizer, warmup_iters, warmup_factor):
    """
    Learning rate scheduler
    :param optimizer:
    :param warmup_iters:
    :param warmup_factor:
    :return:
    """
    def f(x):
        if x >= warmup_iters:
            return 1
        alpha = float(x) / warmup_iters
        return warmup_factor * (1 - alpha) + alpha

    return torch.optim.lr_scheduler.LambdaLR(optimizer, f)

and this is where the function is called:

if epoch == 0:
    warmup_factor = 1. / 1000
    warmup_iters = min(1000, len(data_loader) - 1)

    lr_scheduler = utils.warmup_lr_scheduler(optimizer, warmup_iters, warmup_factor)

As I understood it gradually increase learning rate until it reach initial learning rate. Am I correct? Why we need to increase learning rate? As I know for better learning in Neural Networks we decrease learning rate.

The higher (or smaller) the learning rate, the higher (or, respectively, smaller) the contribution of the gradient of the objective function, with respect to the parameters of the model, to the new parameters of the model. Therefore, if you progressively increase (or decrease) the learning rate, then you will accelerate (or, respectively, slow down) the learning process, so later training examples have higher (or, respectively, smaller) influence on the parameters of the model.

In your example, the function warmup_lr_scheduler returns an object of class LambdaLR, initialized with a certain optimizer and the function f, which is defined as

def f(x):
    if x >= warmup_iters:
        return 1
    alpha = float(x) / warmup_iters
    return warmup_factor * (1 - alpha) + alpha

The documentation of torch.optim.lr_scheduler.LambdaLR says that the function f should compute a multiplicative factor given an integer parameter epoch, so x is a training epoch. If the epoch x is greater than or equal to warmup_iters, then 1 is returned, but anything multiplied by 1 is itself, so, when the epoch x is greater than a threshold, warmup_iters (e.g. 1000), then the initial learning rate is unaffected. However, when x < warmup_iters, the multiplicative factor is given by

alpha = float(x) / warmup_iters
warmup_factor * (1 - alpha) + alpha

which is a function of the epoch x. The higher the epoch x, the higher the value of alpha, so the smaller (1 - alpha) and warmup_factor * (1 - alpha). Note that float(x) / warmup_iters will never be greater than 1 because x is never greater than warmup_iters. So, effectively, as the epoch increases, warmup_factor * (1 - alpha) tends to 0 and alpha tends to 1.

The learning rate can only increase if you multiply it with a constant greater than 1. However, this can only happen if warmup_factor > 1. You can verify this by solving the inequality warmup_factor * (1 - alpha) + alpha > 1.

To conclude, the initial learning rate is not being increased, but the learning process starts with a smaller learning rate than the given learning rate, for a warmup_iters epochs, then, after warmup_iters epochs, it uses the initially given learning rate (e.g. 0.002).


Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated? Many have examined the idea of modifying learning rate at discrete times during the training of an artificial network using conventional back propagation.  The goals of such work have been a balance of the goals of artificial network training in general.

Minimal convergence time given a specific set of computing resources
Maximal accuracy in convergence with regard to the training acceptance criteria
Maximal reliability in achieving acceptable test results after training is complete

The development of a surface involving these three measurements would require multiple training experiments, but may provide a relationship that itself could be approximated either by curve fitting or by a distinct deep artificial network using the experimental results as examples.

Epoch index
Learning rate hyper-parameter value
Observed rate of convergence

The goal of such work would be to develop, via manual application of analytic geometry experience or via deep network training the following function, where

$\alpha$ is the ideal learning rate for any given epoch indexed by $i$,
$\epsilon$ is the loss function result, and
$\Psi$ is a function the result of which approximates the ideal learning rate for as large an array of learning scenarios possible within a clearly defined domain.

$\alpha_i = \Psi (\epsilon, i)$

The development of arriving at $\Psi$ as a closed form (formula) would be of general academic and industrial value.

Has this been done?

Has this been done?

Difficult to prove a negative, but I suspect although plenty of research has been done into finding ideal learning rate values (the need for learning rate at all is an annoyance), it has not been done to the level of suggesting a global function worth approximating.

The problem is that learning rate tuning, like other hyperparameter tuning, is highly dependent on the problem at hand, plus the other hyperparamater values currently in use, such as size of layers, which optimiser is in use, what regularisation is being used, activation functions.

Although you may be hoping for $\Psi(\epsilon, i | P)$ to exist where P is the problem domain, it likely does not except as a mean value over all $\Psi(\epsilon, i | D, H)$ for the problem domain, where D is the dataset and H all the other hyperparameters.

It is likely that such a function exists, of ideal learning rate for best expected convergence per epoch. However, it would be incredibly expensive to sample it with enough detail to make approximating it useful. Coupled with limited applicability (not domain-specific, but linked to data and other hyperparameters), a search through all possible learning rate trajectories looks like it would give poor return on investment.

Instead, the usual pragmatic approaches are:

Include learning rate in hyperparameter searches, such as grid search, random search, genetic algorithms and other global optimisers.
Decay learning rate using one of a few approaches that have been successfully guessed and experiments have shown working. These have typically been validated by plotting learning curves of loss functions or other metrics, and the same tracking is usually required in new experiments to check that the approach is still beneficial.
Some optimisers use a dynamic learning rate parameter, which is similar to your idea but based on reacting to measurements during learning as opposed to changes based on an ideal function. They have a starting learning rate, then adjust it based on  heuristics derived from measuring learning progress. These heuristics can be based on per-epoch measurements, such as whether a validation set result is improving or not. One such approach is to increase learning rate whilst results per epoch are improving, and reduce learning rate if results are not improving, or have got worse.

I have tried this last option, on a Kaggle competition, and it worked to some extent for me, but did not really improve results overall - I think it is one of many promising ideas in ML that can be made to work, but that has not stayed as a "must have", unlike say dropout, or using CNNs for images.

Some optimisers store a multiplier per layer or even per weight - RMSProp and Adam for example track rate of change of each parameter, and adjust the rate for each weight during updates. These can work very well in large networks, where the issue is not so much needing a specific learning rate at any time, but that a single learning rate is too crude to cover the large range of gradients and differences in gradients across the index space of all the connections. With RMSProp and Adam, the need to pick specific learning rates or explore them is much reduced, and often a library's default is fine.


Has it been theoretically proven or dis-proven that a machine can generally think and communicate on its own? I am not looking for proof of AI (too broad, I know) concepts trying to solve a specific business problem, instead for a proof (or dis-proof) or a model (or a source of the same) where a machine is capable of

General thinking and decision making.
Collecting data as it see fit (use any channel at their disposal).
Communicate as it see fit (with whom, how, where and when).
Develop and evolve an agenda.

Note - I wrote an article more than 5 years ago - why a software cannot be intelligent. Since then I have not read a convincing argument against it.

Edit

Intelligence is the ability to take a decision in the unseen, unfamiliar, and unvisited circumstances. So, my definition of Artificial intelligence is - a machine doing essentially the same (doing, not replicating or simulating).

I am looking for a proof or disproof that machine is capable of doing things (written above), not the following

Instances where machine as simulated a certain aspect or property of intelligence.
Machine replicating a certain human behavior to achieve a goal.

We have not been able to create a truly intelligent AI yet, according to your definition. So we have no real life proof-of-concept that shows that it actually works.

But based on the current research, there is no known property of the human brain that cannot be modeled in software/hardware. We do not understand the human brain enough yet - and most likely lack the computational power and required storage - to create an artificial copy. But as long as we do not identify a property of the human brain that cannot be translated into a piece of software/hardware, it is most likely possible. I honestly cannot even imagine such a property within the bounds of physics that would create such a barrier.

Of course this is no formal proof, but without a clearly identified show-stopper there is no reason to believe that it is not possible to create a thinking machine.


What are examples of thought experiments against or in favour of strong AI, apart from the Chinese room argument? The Chinese Room argument against strong AI overlooks the fact that "the man in the room" is acting as a macro-scale "neurotransmitter" of the larger system in which he resides. It does not rule out strong AI, it simply reduces to an enigmatic question: where does understanding "reside" and how does it epiphenomenally emerge?

What are other examples of thought experiments against or in favor of strong AI (apart from the Chinese room argument) or extensions or refutations to known experiments?

An excellent book summarizing the development of thought in this area over several hundred years is Mind Design II, edited by John Haugeland. This book contains a collection of essays written by the major thinkers in this area through until about the 1990s. A brief summary of some of the major ideas is:

Descartes: Minds are spirits, brains are bodies. Minds attach to brains in a special part of the brain. Descartes has a number of thought experiments, but they mostly have a Theological root. Most modern scientists do not find these to be satisfying because the resolve metaphysical questions about the mind by assuming you already believe in a particular interpretation of a God.
Gilbert Ryle argues in the late 1940s that concepts like the Mind are in some sense vacuous. Asking whether a machine has a mind or not is like asking whether a field contains two cows or a pair of cows: this is a conceptual distinction that has no relation to the real world. Because of this, its study is futile. The parallel movement of Behaviorism in psychology also viewed the study of minds and mental states as a fool's errand. Rather, this movement focused on studying behaviors that could be directly measured.
Turing publishes his seminal paper in AI, in 1950. In this paper, Turing anticipates nearly all later objections to AI, and summarizes reasonable refutations of them. This paper contains a number of short thought experiments. One of the more notable ones is the Turing Test. A key observation Turing makes is that if a machine behaves in a way indistinguishable from a human, then any arguments used to reject its claim to intelligence (and even to consciousness) appear to work equally well on other humans. Searle doesn't really address this in his later argument. Basically, I have no way to know for sure whether you have subjective experiences of the same kind I have. The fact that you're made of roughly the same kind of meat as I am seems like a shaky explanation. For example, would humanoid aliens that act like us also be intelligent? What if it were found that their brains took some radically different form from ours? What if it were found that their brains were really networks of transistor-like machines?
The Cognativist revolution spanned a number of fields starting in the 1960s. This movement held that behaviorism was wrong for two basic reasons. First, Behaviorism does not account for the subjective feelings of existence or understanding. Second, Behaviorism could not explain phenomena like language, that appeared to involve reasoning logically about symbols. Notable authors in this period are Chomsky and Fodor, who argued that minds were essentially computer programs that happened to be running on brains instead of computers. These ideas dominated AI, Psychology and Linguistics for about 30-40 years. Fodor's works contain a number of thought experiments involving language and programs.
John Searle's Chinese Room argument caused problems for the cognativists, because it presents a convincing chain of logic showing that a program that does complicated tasks (like linguistics) does not subjectively understand what it is doing. Various objections are raised and Searle responds to them in his publication of the argument. Objections to Searle's arguments form the basis of several later movements in philosophy of AI.
The Connectionists argue that we can represent the behavior of the brain (and thus, the mind) with a brain-like program. This paralleled increasing interest in artificial neural networks. An example of a cognativist thought experiment is to imagine a simulation of an entire human brain, conducted with perfect fidelity (for instance, we accurately model all the quantum mechanical interactions). If such a machine were not conscious, opponents need to answer the question of where consciousness is hiding (Is it in the meat? Then see Turing. Is it in some yet-undiscovered property of neurons? Then the burden of proof is definitely on the person proposing such a hidden property, and they must explain why we cannot simulate it).
The Churchlands and others argued, much as Ryle, that minds as the public understand them are a sort of "folk theory", that appear to explain something about reality to lay people, but are actually a dead end. Concepts like "beliefs" and "experiences" are, according to these arguments, akin to aether or phlogiston: things that our current, malformed, understandings of neuroscience seem to require, but that in fact have no basis in reality, even though nearly everyone believes in them at present. This line of work contains many extensions of the kinds of thought experiments Turing hinted at.
Brooks and others advance the theory of embodied cognition, in which the mind is not confined to the body, but is actually a property of a body plus the environment it is placed in. An example thought experiment from this school (by Andy Clarke if I recall correctly) is to imagine a person with profound loss of the ability to form new long term memories. This person carries around a notebook. The cover says "Your memory". The first page contains instructions explaining that the person has lost the ability to remember things, and that they write new things they want to remember in the book. The person can remember things for, say, 15 minutes at a time, and the book is very well organized. Where is this person's mind exactly? Is it in their brain (which does not remember anything 15 minutes in the past), or is it in the brain and the book, which together can remember and reason about things essentially like a normal person. If it is in the brain and book, what if we replace the book with a computer inside the person's head?


Is a dataset of roughly 700 sentences of an average length of 15 words enough for text classification? I'm building a customer assistant chatbot in Python. So, I am modelling this problem as a text classification task. I have available more or less 7 hundred sentences of an average length of 15 words (unbalanced class).

What do you think, knowing that I have to do an oversampling, is this dataset large enough?

It depends on the number of classes; we are getting good results with about 40 training examples per class.

A good way to get an idea about this is to run a test with an increasing set of training data, evaluating the result as you go along. Obviously, with a small set (eg 3 sentences per class), it will be very poor, but the accuracy should quickly increase and then stabilise at a higher level. With larger amounts of data you will probably only find a small increase or no change at all.

Collecting this data would not only give you confidence in your conclusion, it would also be a good supporting argument when you have to ask for more training data, or have to justify the poor performance of the classifier if you do find the data set is too small.

So, set up an automated 10-fold cross validation, feed an increasing amount of your available data into it, sit back, and graph the results.


Naive Bayes Algorithm error In the attached image is the probability with the Naives Bayes algorithm of:

  Fem:dv/m/s Young own Ex-credpaid Good ->62%

I calculated the Probability so:

  P(Fem:dv/m/s | Good)*P(Young | Good)*P(own | Good)*P(Ex-credpaid | good)*P(Good) -> 1/6*2/6*5/6*3/6*0.6=0,01389

I don't Know where I failed. Could someone please tell me where is my error?

Your probability hasn't been normalized!

In this case, you are computing the probability of being good, given that the other features have a fixed value. To obtain the correct probability, you need to normalize (divide) the value from your calculation by the probability that the features have taken on those fixed values.

You can calculate this as follows:
 $$P(Fem:dv/m/s, Young, own, Ex-credpaid) = \\ \sum_{x \in \{good,bad\}} P(Fem:dv/m/s, Young, own, Ex-credpaid, x) $$

by the marginalization rule.

Then, by the chain rule, you may write:

$$\\ \sum_{x \in \{good,bad\}} P(Fem:dv/m/s, Young, own, Ex-credpaid | x) * P(x) $$

So the correct probability of 0.62 should be obtained by the equation:

$$ \frac{P(Fem:dv/m/s, Young, own, Ex-credpaid | good) * P(good)}{\sum_{x \in \{good,bad\}} P(Fem:dv/m/s, Young, own, Ex-credpaid | x) * P(x)}$$

You just need to calculate

$$P(Fem:dv/m/s, Young, own, Ex-credpaid | bad) * P(bad)$$

and it should be easy to compute the rest.


How can I stabilise a recurrent neural network used for binary classification? I’m looking for some help with my neural network. I’m working on a binary classification on a recurrent neural network that predicts stock movements (up and down) Let’s say I’m studying Eur/Usd, I’m using all the data from 2000 to 2017 to train et I’m trying to predict every day of 2018.

The issue I’m dealing with right now is that my program is giving me different answers every time I run it even without changing anything and I don’t understand why?

The accuracy during the train from 2000 to 2017 is around 95% but I’ve noticed another issue. When I train it with 1 new data every day in 2018, I thought 2 epochs was enough, like if it doesn’t find the right answer the first time, then it knows what the answer is since the problem is binary, but apparently that doesn’t work.

Do you guys have any suggestion to stabilize my NN?

Firstly, dealing with the issue that the program gives different answers every time without making any changes can be due to a couple of things.

Assigning random values to weights and bias. This can be solved by setting a seed manually at the start of the program.
Make sure you have set the model to the testing mode after training. For some frameworks, this has to be done manually.

Secondly, regarding your expected results.

To generate a proper accuracy metric, you will have to sample your dataset into training and testing data, making sure there is now overlap between them. This might be an issue as you have stated training on data till 2017 and then again training on data of 2018.

Lastly, don't expect that the model will know that the output is wrong and directly change it because it's binary classification. This is not how neural networks work. The model fits the solution better by gradually updating its weights and biases over a number of iterations. So it will take a number of epochs to learn new trends in the data for 2018.


Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity I'm working on a RL problem with the following properties:

The rewards are extremely sparse i.e. all rewards are 0 except the terminal non-zero reward. Ideally I would not use any reward engineering as that would lead to a different optimization problem.
Actions are continuous. Discretization should not be used.
The amount of stochasticity in the environment is very high i.e. for a fixed deterministic policy the variance of returns is very high.

More specifically, the RL agent represents the investor, the terminal reward represents the utility of the terminal wealth (hence the sparsity), actions represent portfolio positions (hence the continuity) and the environment represents the financial market (hence the high stochasticity).

I've been trying to use DDPG with a set of "commonly used" hyperparameters (as I have no idea have to tune them besides experimenting which lasts too long) but so far (after 10000 episodes) it seems that nothing is happening.

My questions are the following:

Given the nature of the problem I'm trying to solve (sparse rewards, continuous actions, stochasticity) is there a particular (D)RL algorithm that would lend itself well to it?
How likely is it that DDPG simply won't converge to a reasonable solution (due to the peculiarities of the problem itself) no matter what set of hyperparameters I choose?

(1) You might want look into RND (Random network distillation) which allows usage of a curiosity-based exploration bonus for the agent as an intrinsic reward. You can use the intrinsic reward to complement the sparse extrinsic reward return by the environment.

The general idea is to have a randomly initialized fixed target network which encodes the next state & a predictor network is trained to predict the output of the target network. The prediction error is used to "quantify the novelty of new experience". Stronger novelty will be a good indication for the agent that it maybe worthwhile to explore more.

The authors of this (A) paper were able to achieve SOTA performance in Montezuma's Revenge, which is notorious for it's sparse reward.

In appendix A.1, It is mentioned that: "An exploration bonus can be used with any RL algorithm by modifying the rewards used to train the
model (i.e., rt = it + et)." It is also mentioned that the authors combined this exploration bonus with PPO (which also works in continuous action space). In A.2, a pseudo code is provided.

I would also recommend this (B) paper (see section 3) if you're interested in exploring the available Bonus-Based Exploration Methods out there which may help in tackling hard exploration games with sparse rewards.

With regards to high stochasticity & variance, I found an interesting remark (on page 3, under Figure 2) in this (C) paper:

"our investigation of DDPG on different network configurations shows that for the Hopper environment, DDPG is quite unstable no matter the network architecture. This can be attributed partially to the high variance of DDPG itself, but also to the increased stochasticity of the Hopper
task."

The remark was made in the context where the authors were trying to "tune DDPG to reproduce results from other works even when using their reported hyper-parameter settings".

Have a look here for a different benchmark on how DDPG fair against other algorithms.

(2) From the information provided, I can't conclusively provide you a quantitative assessment on DDPG performance for your specific problem. However, I would recommend the following:

(a) I will encourage you to try different RL algorithms when face with a difficult problem so that you can benchmark & find out which is more suitable. Also in (A), the authors mentioned, "PPO is a policy gradient method that we have found to require little tuning for good performance."

(b) Try different sets of hyperparameters. There are many ways to tune them systematically but discussion about this will be out of scope for this question.


Feature extraction other than convolutions for images? Are there approaches other than convolutions to learn features from images? Has there been any research to use approaches such as hashing (e.g. p-hash, diff-hash etc.) in lieu?

Some more complex and better outcomes are those that include the MPEG-7 developed by the MPEG group.

These descriptors specify the video standard in question. They are split for color, shape, texture and movement.

1. Color Descriptors

Color Space. Specifies the data type of the color space in which they are expressed or work other color descriptors. Color spaces that it contemplated are: RGB (Red, Green Blue), YCrCb (Luminance, Chrominance), HSV (Hue, Saturation, Value), HMMD (Hue, Maximum, Minimum, and Difference).

Color Quantization. This descriptor defines a uniform quantization of a given color space.
Dominant Color(s). This descriptor is the most suitable to be used in images, or regions, in which a small number of colors is sufficient to characterize the information of one determined region.

Scalable Color. This descriptor consists in a color histogram in the HSV space. It is useful in image to image comparisons or searches based on color characteristics.

Color Layout. This descriptor allows representing the color spatial distribution within the images in a very compact way, so the recovery is realized with great efficiency.

Color Structure Descriptor. This descriptor characterizes the color distribution in an image. It builds a color histogram, which assigns most importance to the colors that appears most often and spread across the image. GoF/GoP Color (Group of Frames/Group of Pictures). This descriptor is an extension of the Scalable Color descriptor, which, unlike the latter, is applied to video sequences (a collection of images) instead of only an image.

2. Texture Descriptors

Homogeneous Texture. This descriptor emerged as an important tool when looking for and choosing within large collections of images of great visual similarity.

Texture Browsing. This descriptor specifies the perceptual characterization of one particular texture, and pretend be similar to the characterization that a human eye makes, according to regularity terms, coarseness and directivity.
Edge Histogram. It is a descriptor that gives us information on the type of contours or edges that appear in the image.

3. Shape Descriptors

Region Shape. The object´s shape in an image may consist of a single region or a set of regions, so this descriptor is useful for this type of characterization.
Contour Shape. It is characterized by very well represented contour features which facilitates subsequent search and retrieval; it is robust to motion, to occlusions and to different perspectives; it is extremely compact.

Shape 3D. The Shape 3D allows to describe in details the shape in 3D. This tool is very useful today due to continuous development of multimedia technologies.

This was taked from my paper: The MPEG-7 Visual Descriptors: a Basic Survey

Best regards


Why is the merged neural network of AlphaGo Zero more efficient than two separate neural networks? AlphaGo Zero contains several improvements compared to its predecessors. Architectural details of Alpha Go Zero can be seen in this cheat sheet.

One of those improvements is using a single neural network that calculates move probabilities and the state value at the same time, while the older versions used two separate neural networks. It has been shown that the merged neural network is more efficient according to the paper:

  It uses one neural network rather than two. Earlier versions of AlphaGo used a “policy network” to select the next move to play and a ”value network” to predict the winner of the game from each position. These are combined in AlphaGo Zero, allowing it to be trained and evaluated more efficiently.

This seems counter intuitive to me, because from a software design perspective this violates the principle separation of concerns. That's why I am wondering, why this merge has been proven beneficial.

Can this technique - merging different tasks in a single neural network to improve efficiency - be applied to other neural networks in general or does this require certain conditions to work?

Why has this merge proven beneficial?

If you think about the shared Value/Policy network as consisting of a shared component (the Residual Network layers) with a Value and Policy component on top rather than Separation of Concerns it makes more sense.

The underlying premise is that the shared part of the network (the ResNet) provides a high-level generalization of the input (the game states leading up to the move) that is a good input representation for both the shallow Value and Policy networks.

When that is the case, we can reduce the computational load a lot by training a single shared ResNet and using it for two much simpler networks than training two ResNets for the Value and Policy. In their case, training the two together also improves regularisation and thus creates a more robust, general representation.

Specifically, the Alpha Go Zero paper by Silver et al., Mastering the Game of Go without Human Knowledge, states that:

  Combining policy and value together into a single network slightly reduced the
  move prediction accuracy, but reduced the value error and boosted playing
  performance in AlphaGo by around another 600 Elo. This is partly due to
  improved computational efficiency, but more importantly the dual objective regularises the network to a common representation that supports multiple use cases.

Can this technique be applied in general or only in special cases?

Like common components in software libraries, it only makes sense when the problems you are trying to solve benefit from a shared representation.

You can use it if you are training classifiers for similar tasks, or training a new task with little data where you already have a classifier trained over a larger, similar dataset.

Outside of Go, it is often used in image recognition.
Deep pre-trained networks such as the ones from the ImageNet ILSVRC competitions are commonly used as a starting point. They are classifiers that have been trained (for weeks!) on over a million images.

Then, say you want to create a network to recognize you favourite brand of bicycles, you start with the general image-recognition pipeline trained on ImageNet, chop of the last layers that do the actual classification ("it's a Border Collie") and add a small new classifier to pick out only the bicycles you care about.

Since the pre-trained classifier already provides high-level image concepts that are good building blocks for image recognition (it classifies 200 categories), this saves you a lot of training and makes for a very robust classifier.

Of course there are many cases where the problems do not have useful shared representations and thus no benefit from a combined network. Nevertheless, it is a useful tool in the right situations.

Look up Transfer Learning or Multi-Task Learning to learn more about this.


Feed forward neural network using numpy for IRIS dataset I tried to build a neural network for working on IRIS dataset using only numpy after reading an article (link: https://iamtrask.github.io/2015/07/12/basic-python-network/).

I tried to search the internet but everyone was using ml libraries and found no solution using just numpy. I tried to add different hidden layers to my feed forward neural network still it wasn't converging. I tried to use backpropagation. I used sigmoid and also relu neither of which was successful.

Can someone please give me the code which will work on IRIS dataset and built only using feed forward neural networks and numpy as the only library or if it is not possible to built such a thing with these constraints then please let me know what goes wrong with these constraints.

Also tell me will it be possible to create a neural network to predict values of a matrix multiplication i.e. if we have A * B = C with matrix A as input and C as output, can we acheive substantial amount of accuracy with feed forward neural networks here?.

You can not use the ready code directly without any manipulation. Because every piece of code is written for specific datasets. In the article that you mentioned, the writer created a small dataset and then he creates an ANN architecture for it. If you want to build an ANN based on Iris dataset, you should think and create an architecture on paper maybe before coding. You should understand the Iris dataset first, I mean you should understand what is going to be your input shapes, and also the shape of weights are related to inputs' shape. After understanding these you should decide the number of layer of your Neural Network. And that tutorial that you followed is explaining all of these clearly. Do not skip and understand everything steps by step.

If you want a build ANN architecture by yourself, you must understand every mathematics behind that. (forward and backward propagation, loss and cost, gradient descent) If you do not want to deal with all of these maths, you can use a library such as Keras, then you can create an ANN easily.

See this as a source, https://ml-cheatsheet.readthedocs.io/en/latest/
And you see my Kaggle kernel, I create this network with details and only using numpy. Also I used Keras on another section. You can check; https://www.kaggle.com/erdemuysal/gender-recognition-with-lr-and-ann


What techniques to explore for dynamic clustering of documents (emails)? I have a dataset of unlabelled emails that fall into distinct categories (around a dozen). I want to be able to classify them along with new ones to come in the future in a dynamic matter. I know that there are dynamic clustering techniques that allow the clusters to evolve over time ('dynamic-means' being one of them). However, I would also like to be able to start with a predefined set of classes (or clusters/centroids) as I know for a fact what the types of those emails will be.

Furthermore, I need some guidance in terms of what vectorisation technique to use for my type of data. Would creating a term matrix using TF-IDF be sufficient? I assume that the data I am dealing with could be differentiated on the basis of keyword occurrence, but I cannot tell to what degree. Are there more sophisticated vectorisation techniques based more on the text semantics? Are they worth exploring?

It sounds like you are trying to do some kind of semi-supervised learning. In semi-supervised learning, some data points are labelled (you know which class they belong to), and others are not. There are classification algorithms designed specifically for this kind of problem, like a transductive-SVM. I personally have not found these techniques to be more effective than simply discarding the unlabelled data and treating my problem as purely supervised, but YMMV.

TFIDF remains fairly popular, as do ngram-based approaches. A more modern vectorization to consider might be word2vec, which translated something like a bag-of-words style vector into a more meaningful feature space for words.


Can someone please help me validate my MDP? Problem Statement :
I have a system with four states - S1 through S4 where S1 is the beginning state and S4 is the end/terminal state. The next state is always better than the previous state i.e if the agent is at S2, it is in a slightly more desirable state than S1 and so on with S4 being the most desirable i.e terminal state. We have two different actions which can be performed on any of these states without restrictions.
Our goal is to make the agent reach state S4 from S1 in the most optimal way i.e the route with maximum reward (or minimum cost). The model i have is a pretty uncertain one so i am guessing the agent must initially be given a lot of experience to make any sense of the environment. The MDP i have designed is shown below :

MDP Formulation :

The MDP might a look a bit messy and complicated but it basically is just showing that any action (A1 or A2) can be taken at any state (except the terminal state S4). The probability with which the transition takes place from one state to the other and the associated rewards are given below.

States : States S1 to S4. S4 is terminal state and S1 is the beginning state. S2 is a better state than S1 and S3 is a better state than S1 or S2 and S4 is the final state we expect the agent to end up in.

Actions : Available actions are A1 and A2 which can be taken at any state (except of course the terminal state S4).

State Transition Probability Matrix : One action taken at a particular state S can lead to any of the other available states. For ex. taking action A1 on S1 can lead the agent to S1 itself or S2 or S3 or even directly S4. Same goes for A2. So i have assumed an equal probability of 25% or 0.25 as the state transition probability. The state transition probability matrix is the same for actions A1 and A2. I have just mentioned it for one action but it is the same for the other action too. Below is the matrix I created -

Reward Matrix : The reward function i have considered is a function of the action, current state and future state - R(A,S,S'). The desired route must go from S1 to S4. I have awarded positive rewards for actions that take the agent from S1 to S2 or S1 to S3 or S1 to S4 and similarly for states S2 and S3. A larger reward is given when the agent moves more than one step i.e S1 to S3 or S1 to S4. What is not desired is when the agent gets back to a previous state because of a action. So i have awarded negative rewards when the state goes back to a previous state. The reward matrix currently is the same for both the actions (meaning both A1 and A2 have same importance but it can be altered if A1/A2 is preferred over the other). Following is the reward matrix i created (same matrix for both the actions) -

Policy, Value Functions and moving forward :
Now that i have defined my states, actions, rewards, transition probabilities the next step I guess i need to take is to find the optimal policy. I do not have an optimal value function or policy. From lot of googling i did, I am guessing i should start with a random policy i.e both actions have equal probability of being taken at any given state -> compute the value function for each state -> compute the value functions iteratively until they converge -> then find the optimal policy from the optimal value functions.

I am totally new to RL and all the above knowledge is from whatever i have gathered reading online. Can someone please validate my solution and MDP if I am going the right way? If the MDP i created will work ?
Apologies for such a big write-up but i just wanted to clearly depict my problem statement and solution. If the MDP is ok then can someone also help me with how can the value function iteratively converge to an optimal value? I have seen lot of examples which are deterministic but none for stochastic/random processes like mine.

Any help/pointers on this would be greatly appreciated.
Thank you in advance

The good news is that:

Your MDP appears valid, with well-defined states, actions. It has state transition and reward functions (which you have implemented as matrices). There is nothing else to add, it's a full MDP.
You could use this MDP to evaluate a policy, using a variety of reinforcement learning (RL) methods suitable for finite discrete MDPS. For instance, Dynamic Programming could be used, or Monte Carlo or SARSA.
You could use this MDP to find an optimal policy for the environment it represents, again using a variety of RL methods, such as Value Iteration, Monte Carlo Control, SARSA or Q-Learning.

The bad news is that:

All policies in the MDP as defined are optimal, with expected returns (total reward summed until end of episode) of $v(S1) = 55, v(S2) = 33.75, v(S3) = 21.25$ - solved using Dynamic Programming in case you are wondering.
The MDP is degenerate because action choice has no impact on either state transition or reward. It is effectively a Markov Reward Process (MRP) because the agent policy has been made irrelevant.
Without discounting, the best result is not going from S1-S4 directly, as you appear to want, but repeatedly looping S1-S3-S2-S1-S3-S2... (this is currently hiiden by action choice being irrelevant).

There are a few ways to fix this, but maybe the simplest is to make the rewards more straightforward (e.g. +0, +10, +20, +30 for S1-S1, S1-S2..., -10, 0, +10, +20 for S2-S1, S2-S2...) and add a discount factor, often labelled $\gamma$, when calculating values. A discount factor makes immediate rewards have higher value to the agent, so it will prefer to get a larger reward all at once and end the episode than loop around before finishing.

This whole "bad news" section should not worry you too much though. Instead it points to a different issue. The key point is here:

  The model i have is a pretty uncertain one so i am guessing the agent must initially be given a lot of experience to make any sense of the environment.

It looks like you have assumed that you need to explicitly build a MDP model of your environment in order to progress with your problem. So you are providing an inaccurate model, and expect that RL works with that, improving it as part of searching for an optimal policy.

There are a few different approaches you could take in order to learn a model. In this case as your number of states and actions are very low, then you could do it like this:

Create a 2D tensor (i.e. just a matrix) to count number of times each state, action pair is visited, initialised with all zeroes, and indexed using S, A
Create a 3D tensor to count number of times each state transition is observed, again initialised with all zeroes, indexed using S, A, S'.
Run a large number of iterations with the real environment, choosing actions at random, and adding +1 to each visited S, A pair in the first tensor, and +1 to each S, A, S' triple in the second tensor.
You now have an approximate transition function based on real experience, without needing an initial guess, or anything particularly clever, you are just taking averages in a table. Divide each count of S, A, S' by the total count of S, A to get conditional transition probability $p(s'|s,a)$. It's not really an established, named RL method, but will do.

However, if your construction of the MDP is just step 1 for running some RL policy optimisation approach, none of that is really necessary. Instead, you can use a model-free approach such as tabular Q learning to learn directly online from interactions with the environment. This is likely to be more efficient than learning the model first or alongside the policy optimisation. You don't need the explicit MDP model at all, and adding one can make things more complex - in your case for no real gain.

You probably do still need to define a reward function in your case as there is no inherent reward in the system. You want the agent to reach state S4 as quickly as possible, so you need to monitor the states observed and add a reward signal that is appropriate for this goal. As above, I suggest you modify your planned reward structure to be simple/linear and add discounting to capture the requirement to "increase" state as fast as possible (here I am assuming that being in S2 is still somehow better than being in S1 - if that's not the case, and reaching S4 is the only real goal, then you could simplify further). That's because if you make the rewards for state progression non-linear - as in your example - the agent may find loops that exploit the shape of the reward function and not work to progress states towards S4 as you want.

Beyond this very simple looking environment, there are use cases for systems that learn transition models alongside optimal policies. Whether or not to use them will depend on other qualities of your environment, such as how cheap/fast it is to get real experience of the environment. Using a learned model can help by doing more optimisation with the same raw data, using it to simulate and plan in between taking real actions. However, if the real environment data is very easy to collect, then there may be no point to that.


Why does is make sense to normalize rewards per episode in reinforcement learning? In Open AI's actor-critic and in Open AI's REINFORCE, the rewards are being normalized like so
rewards = (rewards - rewards.mean()) / (rewards.std() + eps)

on every episode individually.
This is probably the baseline reduction, but I'm not entirely sure why they divide by the standard deviation of the rewards?
Assuming this is the baseline reduction, why is this done per episode?
What if one episode yields rewards in the (absolute, not normalized) range of $[0, 1]$, and the next episode yields rewards in the range of $[100, 200]$?
This method seems to ignore the absolute difference between the episodes' rewards.

The "trick" of subtracting a (state-dependent) baseline from the $Q(s, a)$ term in policy gradients to reduce variants (which is what is described in your "baseline reduction" link) is a different trick from the modifications to the rewards that you are asking about. The baseline subtraction trick for variance reduction does not appear to be present in the code you linked to.

The thing that your question is about appears to be standardization of rewards, as described in Brale_'s answer, to put all the observed rewards in a similar range of values. Such a standardization procedure inherently requires division by the standard deviation, so... that answers that part of your question.

As for why they are doing this on a per-episode-basis... I think you're right, in the general case this seems like a bad idea. If there are rare events with extremely high rewards that only occur in some episodes, and the majority of episodes only experience common events with lower-scale rewards... yes, this trick will likely mess up training.

In the specific case of the CartPole environment (which is the only environment used in these two examples), this is not a concern. In this implementation of the CartPole environment, the agent simply receives a reward with a value of exactly $1$ for every single time step in which it manages to "survive". The rewards list in the example code is in my opinion poorly-named, because it actually contains discounted returns for the different time-steps, which look like: $G_T = \sum_{t=0}^{T} \gamma^t R_t$, where all the individual $R_t$ values are equal to $1$ in this particular environment.

These kinds of values tend to be in a fairly consistent range (especially if the policy used to generate them also only moves slowly), so the standardization that they do may be relatively safe, and may improve learning stability and/or speed (by making sure there are always roughly as many actions for which the probability gets increased as there are actions for which the probability gets decreased, and possibly by making hyperparameters easier to tune).

It does not seem to me like this trick would generalize well to many other environments, and personally I think it shouldn't be included in such a tutorial / example.

Note: I'm quite sure that a per-episode subtraction of the mean returns would be a valid, albeit possibly unusual, baseline for variance reduction. It's the subsequent division by standard deviation that seems particularly problematic to me in terms of generalizing to many different environments.


Is the working of RNNs, LSTM and GRU sequential or parallel? You take any blog or any example and all they tell you about is the given picture below.

It has 4 different matrices and 3 of whose weights are shared. So, I'm wondering how is this achieved in practice?
Please correct me:
I think the first word "hello" goes in as a one-hot encoded form and changes the Hidden matrix. And then after it, "world" goes and gets multiplied and then changes the matrix again and so on. What people make it look like is that all of the words going are in Parallel. It can't be the case because the Hidden matrix is dependent on the previous word and without changing the matric, you can not pass the current word. Please correct if my idea is wrong but I think the execution is in sequential order.

Yes, you are correct and it was one of original motivations, which inspired the invention of the Attention mechanism in seq2seq problems https://arxiv.org/pdf/1706.03762.pdf.
There is a quote from this paper:

Recurrent models typically factor computation along the symbol
positions of the input and output sequences. Aligning the positions to
steps in computation time, they generate a sequence of hidden states $h_t$,
as a function of the previous hidden state $h_{t−1}$ and the input for
position $t$. This inherently sequential nature precludes parallelization
within training examples, which becomes critical at longer sequence
lengths, as memory constraints limit batching across examples.

On the other hand, Transformer architectures have a loot loom for parallelization, because they take the whole sequence at once, and multiple heads can be executed in parallel.


When should we use separable convolution? I was reading the "Deep Learning with Python" by François Chollet. He mentioned separable convolution as following

This is equivalent to separating the learning of spatial features and
the learning of channel-wise features, which makes a lot of sense if
you assume that spatial locations in the input are highly correlated,
but different channels are fairly independent.

But I could not understand what he meant by saying "correlated spatial locations". Can some explain what he means or the purpose of separable convolutions? (except performance-related part).
Edit: Separable convolution means that first depthwise convolution is applied then pointwise convolution is applied.

Context of the question
This is a link to the text cited in the question.
It refers to the usage of SeparableConv2D (tf, keras name). A related question on StackOverflow is "What is the difference between SeparableConv2D and Conv2D layers". This answer points to this excellent article by Chi-Feng Wang:
A Basic Introduction to Separable Convolutions
Answer to the question
In image processing, a separable convolution converts a NxM convolution to two convolutions with kernels Nx1 and 1xM. Using this idea, in NN a SeparableConv2D converts a WxHxD convolution (width x height x depth, where depth means number of incoming features ) to two convolutions with kernels WxHx1 and 1x1xD.
Note the first kernel doesn't handles information across features, thus, it is "learning of spatial features". The 1x1xD kernel doesn't handles different points, it is "learning of channel-wise features".
About the phrase "spatial locations in the input are highly correlated", my understanding of what the author means is: Assume we have a channel (feature) image that each pixel measures the "distance to the background". When we pass from one pixel to a neighbors one, it is expected some continuity in the value (except for edge pixels): correlation. Instead, if we have a channel that measures "brightness" and another one that measures "distance to background" the two values for one specific pixel has little correlation.
Finally, about title question "When should we use separable convolution?" : if the final output must depend of some features of one pixel and some other features of neighbors pixels in a very unpredictable way, a complete WxHxD convolution must be used. However if, as is more usual, you can handle first spatial dependencies (neighborhood) to extract pixel features and next handle pixel-by-pixel these features to get the output, better use a WxHx1 followed by 1x1xD, saving lots of network parameters, thus, saving training time.


Which machine learning algorithm is used in self-driving cars? Which deep neural network is used in Google's driverless cars to analyze the surroundings? Is this information open to the public?

It will not be single DNN architecture, rather it will be a collection of different DNN architectures that are used together to make the final decision. Convolutions are using the images/videos from the camera. Other architectures use other sensory sources. These DNNs will be trained to compute the high-level features from their sensory sources and then those high-level features will probably be fed into an LSTM (or some other form of RNN) that is trained with some form of Reinforcement learning algorithm to compute the action (like slowing down, applying breaks etc).


If we want to classify something as either a cat/dog or neither, do we need 2 or 3 classes? Suppose one trains a CNN to determine if something was either a cat/dog or neither (2 classes), would it be a good idea to assign all cats and dogs to one class and everything else to another? Or would it be better to have a class for cats, a class for dogs, and a class for everything else (3 classes)? My colleague argues for 3 classes because dogs and cats have different features, but I wonder if he's right.

If you want to determine if something is either a

cat/dog or neither

you need 2 classes:

one for dog or cat, and
one for anything else.

However, if you assign all cats and dogs to the same class $A$, if an input is classified as $A$, then you won't be able to know whether it is a dog or a cat, you will just know that it is either a dog or a cat.
In case you wanted to distinguish between cats and dogs too (apart from neither of them), then you'll need $3$ classes.
Finally, if you specify only 2 classes:

dog, and
cat,

then your CNN will try to classify any new input as either a dog or a cat, even though it is neither a dog nor a cat (e.g. maybe it is a horse).


How does Hearthstone AI deal with random events I want to learn a lot about the AI of CCG, such as Hearthstone. And now I have known one of the main algorithms that used in this kind of games, MCTS. It analyses the most promising moves, and expands the search tree based on random sampling of the search space. But there are too many random events in this game that can cause different results to one battle. For example, a card can randomly deal X damage to a hero or other follower, and X is a random number from 0 to 30. The number of X is important for the next decision, but there will be a low accuracy by only using MCTS.

So what does the AI do to deal with these random events?

The most "standard" implementation of MCTS probably involves storing copies of game states inside nodes. This works fine for deterministic games, but not for non-deterministic games due to the reasons you mentioned.

In non-deterministic games, one of the easiest ways to make MCTS work is to take the perspective that every node in your tree should map to / represent / encode a trajectory or sequence of actions, rather than a state. This means that every node collects statistics (num wins / num visits etc.) for the trajectory of actions it represents, rather than for the state it represents.

This perspective can be implemented very easily by simply not storing any game states in nodes at all. Instead, whenever you traverse the tree (in the Selection phase of MCTS), you should simply re-simulate all the actions along that path through the tree, starting from the root state again. This way, different visits through a single node can result in different game states being "observed" in that node due to nondeterministic game state transitions. In the limit (after an infinite amount of time), every node will estimate the expected value of the trajectory of actions leading to that node, rather than the game-theoretic value of a single, deterministic resulting game state. This whole idea is sometimes referred to as "Open-Loop MCTS", for example in this paper.

What I described above is a simple way to get better performance from MCTS in games involving nondeterministic state transitions as you described (such as random amounts of damage being dealt), but there are many much more complicated ideas out there too. I suspect more complex approaches will especially be necessary to deal with more "drastic" forms of nondeterminism (resulting from random card draws, or resulting from randomization-based approaches for dealing with imperfect information such as not knowing which cards your opponent has).

There was a Hearthstone AI competition relatively recently, and some useful ideas may be found, for example, on this page of work related to that competition.


Semantic segmentation CNN outputs all zeroes I'm using MATLAB 2019, Linux, and UNet (a CNN specifically designed for semantic segmentation). I'm training the network to classify all pixels in an image as either cell or background to get segmentations of cells in microscopic images. My problem is the network is classifying every single pixel as background, and seems to just be outputting all zeroes. The validation accuracy improves a little at the very start of the training but than plateaus at around 60% for the majority of the training time. The network doesn't seem to be training very well and I have no idea why.
Can anyone give me some hints about what I should look into more closely? I just don't even know where to start with debugging this.
Here's my code:
    % Set datapath
    datapath = '/scratch/qbi/uqhrile1/ethans_lab_data';

    % Get training and testing datasets
    images_dataset = imageDatastore(strcat(datapath,'/bounding_box_cropped_resized_rgb'));
    load(strcat(datapath,'/gTruth.mat'));
    labels = pixelLabelDatastore(gTruth);
    [imdsTrain, imdsVal, imdsTest, pxdsTrain, pxdsVal, pxdsTest] = partitionCamVidData(images_dataset,labels);

    % Weight segmentation class importance by the number of pixels in each class
    pixel_count = countEachLabel(labels); % count number of each type of pixel
    frequency = pixel_count.PixelCount ./ pixel_count.ImagePixelCount; % calculate pixel type frequencies
    class_weights = mean(frequency) ./ frequency; % create class weights that balance the loss function so that more common pixel types won't be preferred

    % Specify the input image size.
    imageSize = [512 512 3];

    % Specify the number of classes.
    numClasses = 2;

    % Create network
    lgraph = unetLayers(imageSize,numClasses);

    % Replace the network's classification layer with a pixel classification
    % layer that uses class weights to balance the loss function
    pxLayer = pixelClassificationLayer('Name','labels','Classes',pixel_count.Name,'ClassWeights',class_weights);
    lgraph = replaceLayer(lgraph,"Segmentation-Layer",pxLayer);

    %% TRAIN THE NEURAL NETWORK

    % Define validation dataset-with-labels
    validation_dataset_with_labels = pixelLabelImageDatastore(imdsVal,pxdsVal);

    % Training hyper-parameters: edit these settings to fine-tune the network
    options = trainingOptions('adam', 'LearnRateSchedule','piecewise', 'LearnRateDropPeriod',10, 'LearnRateDropFactor',0.3, 'InitialLearnRate',1e-3, 'L2Regularization',0.005, 'ValidationData',validation_dataset_with_labels, 'ValidationFrequency',10, 'MaxEpochs',3, 'MiniBatchSize',1, 'Shuffle','every-epoch');

    % Set up data augmentation to enhance training dataset
    aug_imgs = {};
    numberOfImages = length(imdsTrain.Files);
    for k = 1 : numberOfImages
        % Apply cutout augmentation
        img = readimage(imdsTrain,k);
        cutout_img = random_cutout(img);imwrite(cutout_img,strcat('/scratch/qbi/uqhrile1/ethans_lab_data/augmented_dataset/img_',int2str(k),'.tiff'));
    end
    aug_imdsTrain = imageDatastore('/scratch/qbi/uqhrile1/ethans_lab_data/augmented_dataset');
    % Add other augmentations
    augmenter = imageDataAugmenter('RandXReflection',true, 'RandXTranslation',[-10 10],'RandYTranslation',[-10 10]);
    % Combine augmented data with training data
    augmented_training_dataset = pixelLabelImageDatastore(aug_imdsTrain, pxdsTrain, 'DataAugmentation',augmenter);

    % Train the network
    [cell_segmentation_nn, info] = trainNetwork(augmented_training_dataset,lgraph,options);

    save cell_segmentation_nn

Similar to other answers, I don't know Matlab that well but you could try the following steps to debug your problem.
Make sure you can overfit to a single instance
from your dataset, pull out a single image with a good amount of true positives in it. Duplicate that images B times (where B = Batch Size) and then try to train your network with only that small dataset. If you can't overfit to a single instance, then something is really wrong and you should validate all functional aspects of your network. If you can overfit, then it's probably more of an algorithmic or data imbalance issue.
Validate Functional Aspects of the Network
Make sure your input data and labels are correct.
Validate that your images are being correclty inputted into the network. You can do this by printing out any images right before they go into the training function. For the labels, make sure to manually inspect a few labels to be sure that the labels correctly match up the images.
Make sure that your loss function is correct.
Add a unit test or two to your loss function to validate that it is doing what it should be doing. Create a simiple example that you can easily validate.
Validate the rest of the non-algorithmic functionality
Anything that isn't a design choice should be validated. Make sure that the weights are the correct sizes, each layer has the correct number of weights, the intermediate features have the correct shape, etc.
Data Imbalance
If you were able to overfit to a single image, and validated the functional aspects of your network then you may be facing a data imbalance issue. Look at your dataset and see what % of your instances are true positives vs. true negatives. If you have an exteme imbalance (like 10% 90%) or something like that, then build a dataset that is more balanced and see if you can fit your data. If you fit the data with that more balanced dataset, then there's plenty of ways to fix your data imbalance issue. Google around for data imbalance and you should get a few good ideas. Some include focal loss, upsampling, etc...
Check your receptive field
The receptive field on a network is basically the area that can be looked at by the network for a specific region. This is controlled by the size of the convolution filters, stride, etc. If the data that you are segmenting doesn't fit within the receptive field of the model, then you might be saturating the loss function. tldr try playing around with kernel sizes.


How accurate are neuroevolution algorithms in modelling organism evolution? How accurate are neuro-evolution algorithms (such as NEAT) in modelling real organism evolution?

Modelling genetic code persistence, mutation, and meiosis for neurological features or any other biological features is not the greatest of challenges. Modelling the rest of the organism that can support neural growth and change, the electrochemistry of the many types of neurons, and the environment that favors the emergence of learned neural behaviors is challenging and still far beyond the current level of technology.


Understanding the On-policy state distribution for episodic tasks with $\gamma \in (0,1)$ In Sutton and Barto's Reinforcement Learning: An Introduction, section 9.2 (page 199) (here is a screenshot) describes the on-policy distribution in episodic tasks, with $\gamma =1$, as being
\begin{equation}
\mu(s) = \frac{\eta(s)}{\sum_{k \in S} \eta(k)},
\end{equation}
where
\begin{equation}
\eta(s) = h(s) + \sum_s \eta(\bar{s}) \sum_a \pi(a|\bar{s})p(s|\bar{s},a), \text{  (9.2)}
\end{equation}
is the number of time steps spent, on average, in state $s$ in a single episode.
Another way to represent this is setting $\eta(s) = \sum_{t=0}^{\infty} d_{j,s}^{(t)}$, the average number of visits to $s$ starting from $j$, and $d_{j,s}^{(t)}$ being the probability of going from $j$ to $s$ in $t$ steps under policy $\pi_{\theta}$. In particular, $d_{j,s}^{(1)} = d_{j,s} = \sum_{a \in A}[\pi_{\theta}(a|j)P(s|j,a)]$. This formulation is obtained through pag 325 proof of the Policy Gradient Theorem (PGT) and some basic stochastic processes theory.
If instead of defining $\gamma = 1$ we prove PGT using any $\gamma \in (0,1)$, we would get
\begin{equation*}
\eta_{\gamma}(s) = \sum_{t=0}^{\infty} \gamma^t d_{j,s}^{(t)}
\end{equation*}
This is not anymore the average number of visits to $s$. Here comes my first question. Would we still get a $\mu_{\gamma}$ on-policy distribution through the same trick done before? That is
\begin{equation}
\mu_{\gamma}(s) = \frac{\eta_{\gamma}(s)}{\sum_{k \in S} \eta_{\gamma}(k)},
\end{equation}
would be the on-policy distribution?
My second question is related and has to do with the phrase on page 199, that says that

If there is discounting ($\gamma <1$) it should be treated as a form of termination, which can be done simply by including a factor of $\gamma$ in the second term of (9.2).

What the authors mean by "as a form of termination"?
As inferred by my previous question, my conclusion is that having $\gamma < 1$ does not alter $\mu_{\gamma}$ being the on-policy distribution, so I don't get this last comment on page 199.

This question is really getting at the meaning of the discount factor in Markov decision processes. There are actually two, equivalent ways of interpreting the discount factor.
The first is probably what you're familiar with: the $i^{th}$ step receives a reward of $\gamma^{i}r_i$ instead of just $r_i$.
The second, equivalent interpretation, is that at every step, we have a $1-\gamma$ chance of immediately ending the episode. Assuming we don't end the episode, we receive the full $r_i$ reward. This is what Sutton and Barto mean by termination. Note that under the second interpretation, we have a $\gamma^i$ probability of reaching the $i^{th}$ step, hence the connection to the first interpretation.
Your definition of $\eta_{\gamma}(s)$ looks correct to me. Alternatively, we can write it in Sutton's, Barto's notation like
$$\eta_{\gamma}(s) = h(s) + \sum_{\bar s}\eta(\bar s)\sum_{a}\gamma \pi(a | \bar s) p(s | \bar s, a)$$


Is it theoretically possible (or impossible) that principal component analysis worsens the performance of the model? In case I had a prediction model and decided to add a PCA step prior to the model, is it theoretically possible/impossible that the number of output dimensions that is better for all tests may perform worse than the model without PCA?

My question comes from the fact that I want to add a PCA step prior to a model and hyperparameterize the PCA output dimension from 1 to N (N being the number of dimensions in the original dataset) and I wanted to know if there is any theoretical basis that there is no case in which performing this previous step could have a worse performance than the previous model.

Especially my doubt comes if the best PCA case from a selection of dimensions from 1-N is always better than the best case without PCA.

PCA can make models worse,  imagine data points scattered along two elongated parallel rectangles. The axis with the greatest variation will be parallel to the rectangles but doesn't provide any benefit in classifying the points.


Using crowdsourcing for deep learning Most companies dealing with deep learning (automotive - Comma.ai, Mobileye, various automakers etc.) do collect large amounts of data to learn from and then use lots of computational power to train a neural network (NN) from such big data. I guess this model is mainly used because both the big data and the training algorithms should remain secret/proprietary.

If I understand it correctly the problem with deep learning is that one needs to have:

big data to learn from
lots of hardware to train the neural network from this big data

I am trying to think how crowdsourcing could be used in this scenario. Is it possible to distribute the training of the NN to the crowd? I mean not to collect the big data to a central place but instead to do the training from local data on the user's hardware (in a distributed way). The result if this would be lots of trained NNs that would in the end be merged into one in a Committee of machines (CoM) way. Would such model be possible?

Of course the above stated model does have a significant drawback - one does not have control over the data that is used for learning (users could intentionally submit wrong/fake data that would lower the quality of the final CoM). This may be dealt with by sending random data samples to the central community server for review however.

Example: Think of a powerful smartphone using its camera to capture a road from vehicle's dashboard and using it for training lane detection. Every user would do the training himself/herself (possibly including any manual work like input image classification for supervised learning etc.).

I wonder it he model proposed above may be viable. Or is there a better model how to use crowdsourcing (user community) to deal with machine learning?

First thing, you need to give more credit to more reliable users. You can establish the credit from amount of data they send, and a feature, where other users can review other's feed and classify it. From there, you will have a measure of certainty to what data is good and what is not.

You will need to implement a centralized server, unless you're trying to do some kind of a peer-to-peer trust systems, but I don't think smartphones are powerful enough to do training themselves.

You will need big machines for training NNets. Don't trust users to have them. You would end up with tons of badly trained NNets, which don't make for a good CoM.


Does this $\max$ mean that we need to maximize the regret in this regret formula? I found that the regret in Online Machine Learning is stated as:
$$\operatorname{Regret}_{T}(h)=\sum_{t=1}^{T} l\left(p_{t}, y_{t}\right)-\sum_{t=1}^{T} l\left(h(x), y_{t}\right),$$
where $p_t$ is the answer of my algorithm to the question $x$ and $y_t$ is the right answer, while $h()$ is one of the hypotheses in the hypothesis space. Intuitively, as denoted in the paper, our objective is to minimize this Regret in order to optimize our algorithm, but in the following formula
$$
\operatorname{Regret}_{T}(\mathcal{H})=\max _{h^{\star} \in \mathcal{H}} \operatorname{Regret}_{T}\left(h^{\star}\right)
$$
they maximize this value. Am I interpreting the $max$ wrongly?

Yes, you're interpreting the $\max$ there wrongly. In your second formula
$$
\operatorname{Regret}_{T}(\mathcal{H})=\max _{h^{\star} \in \mathcal{H}} \operatorname{Regret}_{T}\left(h^{\star}\right) \label{1}\tag{1}
$$
The sign $=$ means "is defined as", so maybe the following notation is less confusing
$$
\operatorname{Regret}_{T}(\mathcal{H}) \triangleq \max _{h^{\star} \in \mathcal{H}} \operatorname{Regret}_{T}\left(h^{\star}\right)
$$
In fact, in section 2.1 of the same paper, there's a similar but more detailed formula that should clarify the meaning of $\max$
$$
\operatorname{Regret}_{T}(\mathcal{H})=\max _{h \in \mathcal{H}}\left(\sum_{t=1}^{T}\left|p_{t}-y_{t}\right|-\sum_{t=1}^{T}\left|h\left(\mathbf{x}_{t}\right)-y_{t}\right|\right)
$$
Note that $\operatorname{Regret}_{T}(\mathcal{H})$ is defined for $\mathcal{H}$, while your first formula is defined for $x$. So, \ref{1} is the regret for the whole hypotheses class $\mathcal{H}$, which is thus the maximum regret that you can have across all possible hypotheses $h \in \mathcal{H}$. This should make sense.


Why is baseline conditional on state at some timestep unbiased? In the homework for the Berkeley RL class, problem 1, it asks you to show that the policy gradient is still unbiased if the baseline subtracted is a function of the state at time step $t$.

$$ \triangledown _\theta \sum_{t=1}^T \mathbb{E}_{(s_t,a_t) \sim p(s_t,a_t)} [b(s_t)] = 0   $$

I am struggling through what the first step of such a proof might be.

Can someone point me in the right direction? My initial thought was to somehow use the law of total expectation to make the expectation of $b(s_t)$ conditional on $T$, but I am not sure.

Using the law of iterated expectations one has:

$\triangledown _\theta \sum_{t=1}^T \mathbb{E}_{(s_t,a_t) \sim p(s_t,a_t)} [b(s_t)] = \nabla_\theta \sum_{t=1}^T \mathbb{E}_{s_t \sim p(s_t)} \left[ \mathbb{E}_{a_t \sim \pi_\theta(a_t | s_t)}
 \left[ b(s_t) \right]\right] =$

written with integrals and moving the gradient inside (linearity) you get

$= \sum_{t=1}^T \int_{s_t} p(s_t) \left(\int_{a_t} \nabla_\theta b(s_t) \pi_\theta(a_t | s_t) da_t \right)ds_t =$

you can now move $\nabla_\theta$ (due to linearity) and $b(s_t)$ (does not depend on $a_t$) form the inner integral to the outer one:

$=
\sum_{t=1}^T \int_{s_t} p(s_t) b(s_t) \nabla_\theta \left(\int_{a_t} \pi_\theta(a_t | s_t)  da_t \right)ds_t= $

$\pi_\theta(a_t | s_t)$ is a (conditional) probability density function, so integrating over all $a_t$ for a given fixed state $s_t$ equals $1$:

$=
\sum_{t=1}^T \int_{s_t} p(s_t) b(s_t) \nabla_\theta 1 ds_t = $

Now $\nabla_\theta1 = 0$, which concludes the proof.


If IQ is used as a measure of intelligence in humans could it also be used as a measure of intelligence in machines? If IQ were used as a measure of the intelligence of machines, as in humans, at this point in time what would be the IQ of our most intelligent AI systems? If not IQ, then how best to compare our intelligence to a machine, or one machine to another?

This question is not asking if we can measure the IQ of a machine, but if IQ is the most preferred, or general, method of measuring intelligence then how does artificial intelligence compare to our most accepted method of measuring intelligence in humans. Many people may not understand the relevance of a Turing Test as to how intelligent their new car is, or other types of intelligent machines.

It depends on how the IQ test is presented:

If as for humans (effectively, as a video of the book containing the
test questions being opened etc), then all AI programs would score
zero.
If presented as the test set of a supervised learning problem (e.g. as for Bongard Problems) then one might imagine that a number of ML rule induction techniques (e.g. Learning Classifier Systems, Genetic Programming) might achieve some limited success.

So all current AI programs require the problem to be 'framed' in a suitable fashion. It doesn't take too much thought to see that removing the need for such 'framing' is actually the core problem in AI, and (despite some of the claims about Deep Learning), eliminating framing remains a distant goal.

More generally (just as with the Turing test), in order for an IQ test to be a really meaningful test of intelligence, it should be possible as a side effect of the program's capabilities, and not the specific purpose for which humans have designed it.

Interestingly, there is only one program that I'm aware of that sits between 1. and 2.:

Phaeaco (developed by Harry Foundalis at Douglas Hofstadter's research group) takes noisy photographic images of Bongard problems as input and (using a variant of Hofstadter's 'Fluid Concepts' architecture) successfully deduces the required rule in many cases.


Deep Q-learning is not performing well when there are several enemies I am playing with a deep Q-learning algorithm in my own environment. The network can perform well as long as there is only one enemy. My agent can perform the following actions:

do_nothing
prepare_for(e)
attack(e)

where e is some enemy.

In the case of two enemies, the action vector has 5 elements:

|   0       |      1          |      2      |        3         |     4      |
-----------------------------------------------------------------------------
|do_nothing | prepare_for(e1) |  attack(e1) |  prepare_for(e2) | attack(e2) |
-----------------------------------------------------------------------------

After a couple of episodes, the agent always starts picking the first do_nothing action, which is not desired. Changing reward for do_nothing action is not helping, even using significantly higher negative reward, than for other actions.

There is no problem with the environment with only one enemy. (Only using columns 0, 1, 2). I feel like my action encoding can be the issue, but I can't figure it out, how to fix it. Any suggestions?

1) Vary the number of enemies during the training process.
2) Use other generalisation methods that are appropriate for neural networks such as dropout or L1 and L2 regularisation.


How can we compare, in terms of similarity, two pieces of text? How can we compare, in terms of similarity (and/or meaning), two pieces of text (or documents)?
For example, let's say that I want to determine whether a document is a plagiarized version of another document. Which approach should I use? Could I use neural networks to do this? Or are there other more suitable approaches?

There are more than 1 way of doing this:

You can compute the bleu score between them if you are looking at the quality of machine translation. Check this link.
You can convert them into 2 vectors using doc2vec and find the similarity between the vectors using cosine similarity.
Siamese networks are something similar to what you are asking. They are neural nets that use distance metric for learning rather than a loss metric.

I don't understand why do you want to use neural network for comparing two text pieces. Generally comparisons are done by some of the distance metrics not by neural network.


In LSTM text generation can low amount of training data be compensated? I'm playing with an LSTM to generate text. In particular, this one:

https://raw.githubusercontent.com/fchollet/keras/master/examples/lstm_text_generation.py

It works on quite a big demo text set from Nietzsche and says

  If you try this script on new data, make sure your corpus
  has at least ~100k characters. ~1M is better.

This pops up a couple of questions.

A.) If all I want is an AI with a very limited vocabulary where the generate text should be short sentences following a basic pattern.

E.g.

I like blue sky with white clouds

I like yellow fields with some trees

I like big cities with lots of bars

...

Would it then be reasonable to use a much much smaller dataset?

B.) If the dataset really needs to be that big. What if I just repeat the text over and over to reach the recommended minimum? If that would work though, I'd be wondering how that is any different from just taking more iterations of learning with the same shorter text?

Obviously I can play with these two questions myself and in fact I am experimenting with it. One thing I already figured out is that with a shorter text following a basic pattern I can get to a very very low ( ~0.04) quite fast but the predicted text just turns out as gibberish.

My naive explanation for that would be that there are just not enough samples to proof against whether the gibberish actually makes sense or not? But then again I wonder if more iterations or duplicating the content would actually help.

I'm trying to experiment with these questions myself so please don't think I'm just too lazy and are aiming for others to do the work. I'm just looking for more experienced people to give me a better understanding of the mechanics that influence these things.

A.) The algorithm can only learn patterns from the data, so if you want it to align to a specific functional form of language, it would make sense to curate your data to only contain sentences of that very structure.

If you tune your hyper-parameters such that the model is not as complex as the default, it should work for datasets that are smaller. But this relies on the observation that the thing you are trying to learn isn't as complex (as opposed to learning from a large corpus of Nietzsche).

B.) What is happening is that you are overfitting the data, such that the LSTM isn't generalizing to your intended goal. In essence, overfitting means that your model is learning irrelevant details that by chance happen to predict the intended goal in the training data. To alleviate this, use a validation set.

And finally (but very importantly), these machine learning techniques don't learn the same way as humans do. To understand why you'd need to learn the math behind NNs, and I can't think of an intuitive explanation as to why.


What algorithm to use to classify data by spatial relations? Let's assume I have dataset of image-like 2D samples where values can be divided into few discrete levels (for example 1, 2, 3 and 4) like in the image below, where each color maps different value, from 1 to 4. Number of how many times given color occurs on the picture varies from sample to sample though.

I would like to classify these images into different classes but based on the spatial relations of these values between each other (not the values themselves). By spatial relations I mean basically (left, right, up, down), for example:

If blue is above and to the right of the red
Another blue is above and to the left of the same red
Yellow is to the right of one blue (same height)
One green is below red
...

My question is, what algorithm (probably some deep neural network) I should use for that task?
I would appreciate even just some keywords or clues of what might help.

The spatial relationships that you describe would correspond to features, and it's not clear that you need to use a neural network for detecting or discovering these features since you have just described them. Could you instead define a feature extractor that detects the correct patterns and returns you a vector of counts of feature occurrences across the image?
In fact gray level co-occurrance matrices might be adaptable to this problem. Since there are only four colours, each matrix would be very small, and you would define one matrix for each local spatial configuration. For example, one matrix might encode the number of times that colour x is below colour y. From these you could then extract counts of interest to use for further downstream processing.


How is creativity generated in a currently rule based neural network? I  recently became interested in how creativity is generated in NN. My understanding of NNs is that the output always known, given that it they are trained with target values, but how does one train a network to be creative, I mean in such cases would the task be to create something novel, but the actual target would not be known.  How does creativity express itself in a rule based system, such as NN?

One would need to redefine the cost-function, but how does one imply creativity in a rule based system?

Your question is a bit broad. Some examples of creativity and how it's achieved (in all of them there is an ordinary convolutional network trained via supervised learning, but applied differently):

Google Deep Dream uses backpropagation to the image to tweak the image towards maximum activation in the convolutional layer. As a result, it finds lots of "things" in the image and enhances them:

Generative Adversarial Networks use two competing networks (generator and discriminator) to generate samples from the training manifold:

Neuro style method captures the style of one image via Gram matrix of convolutional layer activations and trains (another) neural network with a special loss function that minimizes the joint distance to the incoming image (content) and to the Gram matrix (style). The result:


Examples of uses of artificial intelligence at work I'm going to give a talk, and I'm preparing the material. The purpose of the conversation is to convince companies in my region that it is possible to apply artificial intelligence in solving everyday business problems.

I would like some examples to be able to present, and so I came here to ask

Have you used artificial intelligence to solve a problem at work? What kind of problem?

This is an interesting question in that it addresses how AI becomes a part of our mundane existence.

@JadenTravnik's pithy comment is quite correct--spam filtering is a form of AI that that it so fundamental to the information eco system that it has become nearly invisible.

It's useful to be able to define what we mean by Artificial (or Algorithmic) Intelligence, and to distinguish it from computation in general.  A fundamental distinction is that AI makes decisions as opposed to merely crunching numbers.

My feeling is that applied AI is still a very specialized field, so that people using it dynamically to solve problems at work today are those who work in the field of AI, as opposed to workers in general.

You may find this related question useful: Is AI programming useful in everyday programs?

This may change in the future, where a type of expert system, for instance, could assist a worker by presenting it's own analysis and conclusions on a decision problem. But in most office settings, that type of reasoning is still the domain of humans.

My advice would be to continue on Jaden's track, looking for examples of where AI underlies everyday functions.


What is a successor function (in CSPs)? In Constraint Satisfaction Problems (CSPs), a state is any data structure that supports

a successor function,
a heuristic function, and
a goal test.

In this context, what is a successor function?

A successor function is a function that generates a next state from the current state, plus the choices that affect state changes. In e.g. the 8 queens problem, a state might be the location of 5 queens so far, the choice might be where to put the next queen, and the successor function would return the resulting state with 6 queens on the board.

Typically a solver will store/cache a current state, make a choice, and use the successor function to discover what the next state is. Then it may call the heuristic function on the next state and make a decision whether to continue that search deeper into the search tree (or recursion) or try another choice at the current state.


How fast does Monte Carlo tree search converge? How fast does Monte Carlo Tree Search converge? Is there proof that it does converge?

How does it compare to Temporal Difference learning in terms of convergence speed (assuming the evaluation step is a bit slow)?

Is there a way to exploit the information gathered during the simulation phase to accelerate MCTS?

Sorry if too many questions, if you have to choose one, please choose the last question.

Yes, Monte Carlo tree search (MCTS) has been proven to converge to optimal solutions, under assumptions of infinite memory and computation time. That is, at least for the case of perfect-information, deterministic games / MDPs.

Maybe some other problems were covered too by some proofs (I could intuitively imagine the proofs holding up for non-deterministic games as well, depending on implementation details)... but the classes of problems I mentioned above are what I'm sure about. The initial, classic proofs can be found in:

Bandit based Monte-Carlo Planning by Levente Kocsis, Csaba Szepesvári
Improved Monte-Carlo Search by Levente Kocsis, Csaba Szepesvári, Jan Willemson

Much more recently the paper On Reinforcement Learning Using Monte Carlo Tree Search with Supervised Learning: Non-Asymptotic Analysis appeared on arXiv, in which I saw it is mentioned that there may have been some flaws in those original papers, but they also seem to be able to fix it and add more theory for the more "modern" variants which combine (deep) learning approaches inside MCTS.

It should be noted that, as is typically the case, all those convergence proofs are for the case where you spend an infinite amount of time running your algorithm. In the case of MCTS, you can intuitively think of the proofs only starting to hold once your algorithm has manage to build up the complete search tree, and then on top of that had sufficient time to run through all the possible paths in the tree sufficiently often for the correct values to backpropagate. This is unlikely to be realistic for most interesting problems (and if it is feasible, a simpler breadth-first search algorithm may be a better choice).

  How does it compare to Temporal Difference learning in terms of convergence speed (assuming the evaluation step is a bit slow)?

If you're thinking of a standard, tabular TD learning approach like Sarsa... such approaches actually turn out to be very closely related to MCTS. In terms of convergence speed, I'd say the important differences are:

MCTS focusses on "learning" for a single state, the root state; all efforts are put towards obtaining an accurate value estimate for that node (and its direct children), whereas typical TD implementations are about learning immediately for the complete state-space. I suppose the "focus" of MCTS could improve its convergence speed for that particular state...
but the fact that the search tree (which can be viewed as its "table" of $Q$-values as you'd see in Sarsa or $Q$-learning) only slowly grows can also be a disadvantage, in comparison to tabular TD learning approaches which start out with a complete table that covers the complete state space.

Note that papers such as the last one I linked above show how MCTS can also actually use Temporal Difference learning for its backing up of values through the tree... so looking at it from a "MCTS vs TD learning" angle doesn't really make too much sense when you consider that TD learning can be used inside MCTS.

  Is there a way to exploit the information gathered during the simulation phase to accelerate MCTS?

There are lots and lots of ideas like that tend to improve performance empirically. It will be difficult to say much about them in theory though. Some examples off the top of my head:

All Moves As First (AMAF)
Rapid Action Value Estimation (RAVE, also see GRAVE)
Move Average Sampling Technique (MAST)
N-Gram Selection Technique (NST)
Last-Good-Reply policy
...

Many of them can be found in this survey paper, but it is somewhat old now (from 2012), so it doesn't include all the latest stuff.