What does it mean "derivative of an image"? I am reading a book about OpenCV, it speaks about some derivative of images like sobel. I am confused about image derivative! What is derived from? How can we derived from an image? I know we consider an image(1-channel) as a n*m matrix with 0 to 255 intensity numbers. How can we derive from this matrix?
EDIT: a piece of text of the book:

Derivatives   and Gradients
One   of  the most    basic   and important   convolutions    is  computing   derivatives
(or   approximations  to  them).  There   are many    ways    to  do  this,   but only    a   few
are   well    suited  to  a   given   situation.
In    general,    the most    common  operator    used    to  represent   differentiation is  the
Sobel derivative  operator.   Sobel
operators exist   for any order   of  derivative  as  well    as  for mixed   partial
derivatives   (e.g.,  ∂ 2 /∂x∂y).

The term Derivative of an Image in the context you mention has two meanings.

A matrix, image, or floating point number that is derived from an image via convolution, passing the image through a two dimensional NN, the application of an FFT analysis, or some other process.  In this context, the word Derivative implies the direction of calculation: Image B is derived from image A.
A matrix or cube that represents the rate of change at in the image being processed.  The change being measured between only two adjacent pixels in a single dimension and only one direction at a time, but the applications of this technique is very limited, and such a sequence is of differences, not at all reasonable approximations of the derivative of light.  What is more useful in real recognition systems are two dimensional or hexagonal windowing (Gausian, Hamming, Hanning, trapazoidal, cosine, ...) across space and, for video, through time.  The calculus term derivative should always reference the theoretical surface being approximated using these techniques, not the discrete matrix or cube that approximates the surface.

Such multidimensional convolution and neural network based approaches are less sensitive to capture noise and orientation nuances.  Two dimensional whole image or windowed FFT techniques have met with much success because filtering the expected frequency range of features to be detected is merely an attenuation process.  Two and three dimensional splines can also be tuned to be useful in the detection of features in an orientation independent way.

In addition to gray scale analysis, color and transparency channels can be selected for independent or parallel analysis or added to the dimension of the fitting model from which the derivative is taken.

Advances in deep networks have blossomed into a new area of image processing and recognition research, bringing new hope to robotics, automated transportation, and cybernetics in general.


What are causative and exploratory attacks in Adversarial Machine Learning? I've been researching Adversarial Machine Learning and I know that causative attacks are when an attacker manipulates training data. An exploratory attack is when the attacker wants to find out about the machine learning model. However, there is not a lot of information on how an attacker can manipulate only training data and not the test data set.

I have read about scenarios where the attacker performs an exploratory attack to find out about the ML model and then perform malicious input in order to tamper with the training data so that the model gives the wrong output. However shouldn't such input manipulation affect both the test and training data set? How does such tampering only affect the training data set and not the test data set?

When someone is able to do a causative attack it means there is a mechanism by which they are able to input data into the network. Maybe a website where people can input their images and it outputs a guess on what is in the picture and then you click if it got it right or not. If you continue to input images and lie to it it will obviously get worse and worse if they use the user input to add to the test set. Most people are careful and don't mess around with mixing new data into the testing sample. If they did something like mixed the user input training and test then resampled something like that could occur but most people don't do that. It's bad practice and even worse than leaving your NN open to tampering from malicious user input. Information isn't really added to the knowledge in the model till it is fed into the model and backpropagation occurs.

An exploratory attack is sending tons of inquiries to the model to gain information about the data set they have built into the model even to the point of extracting data about individuals pieces of data that are built into the model. Then, with this information, they could try to reconstruct the data set. They could attempt to trick the network by sending strange generated inputs.

In the paper Adversarial Machine Learning (2011), by Ling Huang et al., in section 2, the authors define these terms, under the category influence.

  Influence

  Causative - Causative attacks alter the training process through influence over the training data.

  Exploratory - Exploratory attacks do not alter the training process but use other techniques, such as probing the detector, to discover information
  about it or its training data.

They also provide other related definitions.

  Security violation

  Integrity - Integrity attacks result in intrusion points
  being classified as normal (false negatives).

  Availability - Availability attacks cause so many classification errors, both false negatives and false positives, that the system becomes effectively unusable.

  Privacy - In a privacy violation, the adversary obtains information from the learner, compromising the secrecy or privacy of the system’s users.

  Specificity (a continuous spectrum)

  Targeted - In a targeted attack, the focus is on a single or small set of target points.

  Indiscriminate - An indiscriminate adversary has a more flexible goal that involves a very general class of points, such as “any false negative.”


Is there any measure of separability of classes? I want to know if there is a measure of how well two classes in Y are separable (linearly or not) based on their features in X. Easiest way of explaining this is to compare it to correlation coefficients, the higher the correlation the higher possiblity for successful regression based on given feature (at least in theory).

Is there any measure that will tell me how well classes are separated based on input data features, before training a ML model?

Are you thinking something like Information Gain?

Information Gain basically uses the concept of information entropy to determine if splitting a variable is useful.


Can we identify only the objects in specific parts of an image with computer vision? I am studying computer vision for the past 3 months. I have come across the object identification problem, where given an image, CV would identify various parts in the image.
If I give an image, and a rectangle coordinates, can CV identify the parts' names within that rectangle? For example, can I train a model to identify the parts in the below image (mountain, river, in this case)? The model should not identify other parts like flowers, sky, etc., as they come outside the rectangle).
I tried searching but could not find similar problems. Can anyone give me a direction to solve this problem?

Giving this small rectangle to a model will probably not provide him enough information to identify what is inside this rectangle. An approach that could work would be to do image segmentation, having the identity of everything in the image, and then extract the label of the pixels in the rectangle.


Unable to replicate Figure 2.1 from "Reinforcement Learning: An Introduction" The author explains in 2.2 Action-Value Methods:

  To roughly assess the relative effectiveness of the greedy and $\varepsilon $-greedy methods, we compared them numerically on a suite of test problems. This is a set of 2000 randomly generated n-armed bandit tasks with n = 10. For each action, a, the rewards were selected from a normal (Gaussian) probability distribution with mean Q*(a) and variance 1. The 2000 n-armed bandit tasks were generated by reselecting the Q*(a) 2000 times, each according to a normal distribution with mean 0 and variance 1. Averaging over tasks, we can plot the performance and behavior of various methods as they improve with experience over 1000 plays, as in Figure 2.1. We call this suite of test tasks the 10-armed testbed.

But doing my best, the replication yields something nearer to :

I think I am misunderstanding how the author took the averages.

Here is my code:

from math import exp

import numpy
import matplotlib.pyplot as plt

def act(action, Qstar):
    return numpy.random.normal(Qstar[action], 1)

def run(epsilon):
    history = [0 for i in range(1000)]

    for task in range(1, 2000):
        Qstar = [numpy.random.normal(0, 1) for i in range(10)]
        Q = [0 for i in range(10)]
        for t in range(1, 1001):
            if numpy.random.randint(0, 100) < epsilon:
                action = numpy.random.randint(0, len(Q))
            elif t == 0:
                action = 0
            else:
                averages = [q/t for q in Q]
                action = averages.index(max(averages))

            reward = act(action, Qstar)
            Q[action] += reward

            history[t-1] += reward

    return [elem/2000 for elem in history]

if __name__ == '__main__':
    plt.plot(run(10), 'b', label="ɛ=0.1")
    plt.plot(run(1), 'r', label="ɛ=0.01")
    plt.plot(run(0), 'g', label="ɛ=0")
    plt.xlabel('Plays')
    plt.ylabel('Reward')
    plt.legend()
    plt.show()

You are calculating the average reward for each action (i.e. bandit arm) incorrectly. You cannot calculate this simply with a list comprehension, and you need to keep a second list storing the number of times each action was taken.

The correct calculation is to divide the total reward obtained from each action by the number of times that action was taken. You are dividing all actions by the total number of actions taken. Adding:

...
counts = [0 for i in range(10)]
...
else:
    averages = []
    for i in range(0, 10):
        averages.append(Q[i]/counts[i] if counts[i] > 0 else 0)
    ...
...
counts[action] += 1

results in working code, and I can generate this graph using 200 samples per method, which looks like a noisier version of the one in the figure you referenced:


Have any chatbots fooled the judges and won the Loebner Prize Gold medal yet? I've read about The Loebner Prize for AI, which pledged a Grand Prize of $100,000 and a Gold Medal for the first computer whose responses were indistinguishable from a human's.

So I was wondering whether any chatbots have fooled the judges and won a Gold Medal yet?

From their website this isn't clear (as some of the links doesn't load).

A few highlights from previous years:

2011 Loebner Prize results

  None of the AI systems fooled the judges, therefore the Turing Test has not been passed.

Loebner 2013 results:

  No chatbot fooled any of the 4 Judges.

The 2016 finals haven't started yet, they will start on Saturday, 17 September 2016.

In the 2015 finals or before that, nobody won the Gold Medal or the Silver Medal.
The most up-to-date data can be found here, where we can find both the results from 2015 and the timeline of the 2016 contest.


How do I show that uniform-cost search is a special case of A*? How do I show that uniform-cost search is a special case of A*? How do I prove this?

Yes, UCS is a special case of A*.

UCS uses the evaluation function $f(n) = g(n)$, where $g(n)$ is the length of the path from the starting node to $n$, whereas A* uses the evaluation function $f(n) = g(n) + h(n)$, where $g(n)$ means the same thing as in UCS and $h(n)$, called the "heuristic" function, is an estimate of the distance from $n$ to the goal node. In the A* algorithm, $h(n)$ must be admissible.

UCS is a special case of A* which corresponds to having $h(n) = 0, \forall n$. A heuristic function $h$ which has $h(n) = 0$, $\forall n$, is clearly admissible, because it always "underestimates" the distance to the goal, which cannot be smaller than $0$, unless you have negative edges (but I assume that all edges are non-negative). So, indeed, UCS is a special case of A*, and its heuristic function is even admissible!

To see this with an example, just draw a simple graph, and apply the A* algorithm using $h(n) = 0$, for all $n$, and then apply UCS to the same graph. You will obtain the same results.


"AI will kill us all! The machines will rise up!" - what is being done to dispel such myths? Science Fiction has frequently shown AI to be a threat to the very existence of mankind. AI systems have often been the antagonists in many works of fiction, from 2001: A Space Odyssey through to The Terminator and beyond.

The Media seems to buy into this trope as well.  And in recent years we have had people like Elon Musk warn us of the dangers of an impending AI revolution, stating that AI is more dangerous than nukes.

And apparently experts think that we will be seeing this AI revolution in the next 100 years.

However, from my (albeit limited) study of AI, I get the impression that they are all wrong. I am going to outline my understanding below, please correct me if I am wrong:

Firstly, all of these things seem to be confusing Artificial Intelligence with Artificial Consciousness.  AI is essentially a system to make intelligent decisions, whereas AC is more like the "self-aware" systems that are shown in science fiction.
Not AI itself, but intelligence and intelligent decision making algorithms is something we've been working with and enhancing since before computers have been around.  Moving this over to an artificial framework is fairly easy.  However consciousness is still something we are learning about.  My guess is we wont be able to re-create something artificially if we barely understand how it works in the real world.
So my conclusion is that no AI system will be able to learn enough to start thinking for itself, and that all our warnings of AI are completely unjustified.
The real danger comes from AC, which we are a long, long way from realizing because we are still a long way off from defining exactly what consciousness is, let alone understanding it.

So my question is, assuming that my understanding is correct, are any efforts are being made by companies or organizations that work with AI to correct these popular misunderstandings in sci-fi, the media and/or the public?

Or are the proponents of AI ambivalent towards this public fear-mongering?

I understand that the fear mongering is going to remain popular for some time, as bad news sells better than good news.  I am just wondering if the general attitude from AI organizations is to ignore this popular misconception, or whether a concerted effort is being made to fight against these AI myths (but unfortunately nobody in the media is listening or cares).

Nothing.

Its in almost everyone's favor for it to stay that way financially. Having non-technical individuals associate AI with terminators makes a perception that the field has greater capabilities than it does $\rightarrow$ this leads to grants, funding, etc...

Is there any negative? Yes. Misconceptions always have drawbacks. We see the creation of dumb ethics boards and such cough cough Elon Musk.

But if history has anything to say about this, as the field gains popularity (which it is dnagerously quick), information will spread by definition, and eventually misconceptions will be laid to rest.

Note that this answer is biased and based upon my own opinions


What is the relationship between MLE and naive Bayes? I have found various references describing Naive Bayes and they all demonstrated that it used MLE for the calculation. However, this is my understanding:

$P(y=c|x)$ $\propto$ $P(x|y=c)P(y=c)$

with $c$ is the class the model may classify $y$ as.

And that's all, we can infer $P(x|y=c)$ and $P(c)$ from the data. I don't see where the MLE shows its role.

And that's all, we can infer P(x|y=c) and P(c) from the data. I don't see where the MLE shows its role.

Maximum likelihood estimate is used for this very purpose, i.e. to estimate the conditional probability $p(x_j \mid y)$ and marginal probability $p(y)$ .

In Naive Bayes Algorithm ,using the properties of conditional probability, we can estimate the joint probability

$$
p(y, x_1, x_2, \dots, x_n) =  p(x_1, x_2, \dots, x_n \mid y) \; p(y)
$$

which is the same as
$$
p(y=c|x)\propto p(x|y=c)p(y=c)
$$

What we need to estimate in here, are the conditional $p(x_j \mid y)$ and marginal $p(y)$ probabilities, and we use maximum likelihood for this.

To be more precise: Assume that the $x$ is a feature vector $(x_1, x_2, ...x_n)$ - which means nothing more than the data point is a vector. In Naive Bayes Classification the assumption is that these features are independent of each other conditional on the class. So, in that case this term
$$p(x|y=c)$$ is replaced by $$\prod_{i=1}^{n} p(x_i|y=c)$$ -- this is because of the independence assumption that lets us simply multiply the probabilities.

The problem of estimating the class to which the data point belongs reduces to the problem of maximizing the  likelihood $$\prod_{i=1}^{n} p(x_i|y=c)$$ -- which means assigning the data $(x_1, x_2, ...x_n)$  to the class $k$ for which this likelihood is highest. This will give the same result as MLE which takes the form  of maximizing this quantity -- $$\prod_{i=1}^{n} p(x_i|\theta)$$ where $\theta$ are the assumed parameters.


Can minimax be used when both players want to increase their score? If both the players want to increase their score (by selecting the highest or best cost path), can this be done using the minimax algorithm, or are there other algorithms for this purpose?

I believe maximax is what you're looking for:

Maximax (economics, computer science, decision theory) A strategy or algorithm that seeks to maximize the maximum possible result (that is, that prefers the alternative with the chance of the best possible outcome, even if its expected outcome and its worst possible outcome are worse than other alternatives); often used attributively, as "maximax strategy", "maximax approach", and so on.SOURCE: maximax (wiki)

You may also be interested in "minimax regret":

The minimax regret approach is to minimize the worst-case regret. The aim of this is to perform as closely as possible to the optimal course. Since the minimax criterion applied here is to the regret (difference or ratio of the payoffs) rather than to the payoff itself, it is not as pessimistic as the ordinary minimax approach.
One benefit of minimax (as opposed to expected regret) is that it is independent of the probabilities of the various outcomes: thus if regret can be accurately computed, one can reliably use minimax regret. However, probabilities of outcomes are hard to estimate.
This differs from the standard minimax approach in that it uses differences or ratios between outcomes, and thus requires interval or ratio measurements, as well as ordinal measurements (ranking), as in standard minimax.
SOURCE: Minimax regret (wiki)

Apologies for just grabbing the wikis on this, but they are accurate.  Most of the references from economic sites are commercial.  If interested, when I have some more time, I could probably link to some peer-reviewed papers.  Hopefully, at the very least, this answer provides information for further research.


Have any AI's been able to decode human vision 'thoughts' I believe I saw an article about an AI that was able to decode human vision 'brain-waves' in real-time, which would create a blurry image of what the human was seeing.

This AI Decodes Your Brainwaves and Draws What You're Looking at

Is anyone aware where I can find this?

There have been studies in University of Oregon and Kyoto University to be able to visualise thoughts and dreams on a screen using voxel values of an FMRI scan as input and an estimation of an image of the thoughts as the output. Instead of linking you to these studies and papers - you could just watch this episode of mind field where both these studies are demonstrated and linked.

The idea behind this is easier to understand if you have a good understanding of generative networks such as generative adversarial networks or so. Essentially in GAN's you'd map a known latent distribution to images in pixel-space. You would be doing the same thing here, just that the latent distribution would now be the FMRI scan input and the mapping would be made in a supervised setting where they are initially showed images. A very rough understanding of the idea can be drawn on these lines.


Is recursion used in practice to improve performance of AI systems? Is there any methods by which artificial intelligence use recursion(s) to solve a certain issue or to keep up working and calculating?

To my knowledge, recursion does not play a strong role in the definition of modern AI techniques, although it does feature used in Lovasz's definition of 'Local Search' and Kurzweil is certainly an advocate.

Recursion can be seen as an elegant 'architectural factorization' - building complexity by combining the results of smaller, similar patterns previously encountered. Computationally, recursion can always be converted into iteration so this form of elegance is really mainly of use in helping to make designs more comprehensible.

GOFAI algorithms that were traditionally defined using recursion include depth- and breath- first search and means-ends analysis (used in Newell and Simon's General Problem Solver).

With respect to performance, while many functions can be very economically defined using recursion, the naive version of such definitions can be inefficient.

This page gives an example in which recursive version of the Fibonacci function, which has asymptotic execution time $n^{1.6}$, which is reduced to $n$ by the use of memoization.


Which fields of AI are actively researching consciousness? Max Tegmark discusses the topic consciousness in his book Life 3.0 and comes to the conclusion, that consciousness is substrate independent. If his analysis is correct, it should be possible to create artificial consciousness. The integrated information theory (IIT), while currently only just a theory, also points in this direction.

This leads me to the question, which fields of AI research are currently actively engaged in this domain, if any. So far I've only found research concerning consciousness in neuroscience and discussions of experts in philosophy.

Are there any projects publicly known concerning artificial consciousness or organizations who are active in this regard?

I think we still have a long way to go before any progress is made on artificial consciousness. However, researchers are taking inspiration from traits of human consciousness. One relevant paper is Machine Theory of Mind by DeepMind. They show that their model can (at least to some extent) represent the desires, beliefs, and intentions of agents that it observes. It even passes a form of the Sally-Anne test, showing that it can represent the false beliefs of an agent.


Symbolic "math" using trained networks Does anyone work out ways of relating trained networks by symbolic logic?

For example:
If I train a network on pictures of dogs, and I train a network on pictures of shirts. You could imagine that the simplest way of (without going through the process from scratch), identifying "dog AND shirt" would be to perform an AND operation on the last output of the individual cat & dog neural nets.

So "dog AND shirt" would amount to AND'ing the output of two nets (Which I believe is described here).

But this operation AND could be replaced with a more complicated operation. And in principle I could "train" a network to act as one of these operations.

For instance maybe I could figure out the net that describes some changeable output "X" being "on the shirt." This would be sort of like a "functional" in mathematics (in which we are operating are considering the behavior of a network whos input could be any network).

If I can figure out this "functional" then I would be able to use is symbolically and determine queries like "dog on the shirt"? - "cat on the shirt"?

It seems like to me there's a lot of sense to turn specific neural networks into more "abstract" objects - and that there would be a lot of power in doing so.

I think you would be interested in Neural-Symbolic Learning and Reasoning, a recent survey on the intersection between connectionist models (e.g. neural nets) and symbolic reasoning. It's a long paper, but it has a lot in it that is relevant to your question, if you're willing to dig. For example, it talks about learning networks that implement boolean logic.

  This
  network searches for satisfying solutions for the weighted conjunctive normal form (CNF):$\text{(¬X ∨ ¬Y ∨ Z) ∧ (X ∨ Y )}$.

The closest thing I could find to the "functionals" you mentioned was a discussion on learning "relations" with neural networks. For this, it cited Neural-Symbolic Cognitive Reasoning, a book also about the intersection between neural networks and symbolic reasoning. In Chapter 10, it gives the example of teaching a neural network to learn the relation of "grandparent."

  Our goal is to learn a description of grandparent from examples of the father, mother, and grandparent relations such as father(charles, william), mother(elisabeth, charles), and grandparent(elisabeth, william).

I could imagine something like this being done with the example inputs being image features from a CNN. Perhaps then you could start learning functionals like "on the shirt."


Kalman filter pre inovation I am trying to track LIDAR objects using Kalman filter. The problem is that the innovation has the value 0, which makes the Kalman gain be Infinity. Here is a link with the Kalman equations. The values with which I initialized the measurement and process covariance matrix are listed below. The update code is also shown below. When I debug the code everything is fine until the innovation becomes 0.

this->lidar_R << std_laspx_, 0, 0, 0,
    0, std_laspy_, 0, 0,
    0, 0, 0, 0,
    0, 0, 0, 0;

this->lidar_H << 1.0, 0.0, 0.0, 0.0, 0.0,
    0.0, 1.0, 0.0, 0.0, 0.0,
    0.0, 0.0, 0.0, 0.0, 0.0,
    0.0, 0.0, 0.0, 0.0, 0.0;

P_ << 1000, 0, 0, 0, 0,
    0, 1000, 0, 0, 0,
    0, 0, 1000, 0, 0,
    0, 0, 0, 1000, 0,
    0, 0, 0, 0, 1000;

 MatrixXd PHt = this->P_ * H.transpose();
 //S becomes 0
 MatrixXd S = H * PHt + R;
 //S_inv becomes INFINITY
 MatrixXd S_inv_ = S.inverse();
 MatrixXd K = PHt * S_inv_;

VectorXd y = Z - Hx;

this->x_ = this->x_ + K*y;
MatrixXd I = MatrixXd::Identity(x_.size(), x_.size());
this->P_ = (I - K * H) * this->P_;

From your matrix definitions, the issue is S always singular so it can never be inverted

I reimplemented the computation with numpy and here are the numbers

H = np.array([[1.0, 0.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0]])

P = np.identity(5) * 1000

R = np.array([[100, 0, 0, 0], [0, 100, 0, 0], [0,0,0,0], [0,0,0,0]])

S=H.dot(P.dot(np.transpose(H)))+R

and S is

array([[1100.,    0.,    0.,    0.],
       [   0., 1100.,    0.,    0.],
       [   0.,    0.,    0.,    0.],
       [   0.,    0.,    0.,    0.]])

Basically the definition of H with its all zeros lines cancels part of the information (make it unobservable) and you also do not have any observation noise component for them


What is the advantage of using Google's Coral over Nvidia's Xavier? I was reading about the possibility of using Google's Coral for deep learning-based object detection and image classification. I heard it has a good speed in terms of frames/sec.

I also read that Google's Coral is only compatible with quantized models. What does this mean? How will this affect the performance of object detection or classification in terms of accuracy and speed?

What is the advantage of using Google's Coral over Nvidia's Xavier?

Quantization is a technique used to make deep learning models smaller and faster to run.

Deep learning models are essentially collections of real-valued numbers. Because there are infinitely many real numbers, computers represent them using a format call 'floating point' numbers, which are not completely accurate. For example, a 32-bit floating point number can only represent at most $2^{32}$ distinct values. In contrast, a 64-bit floating point number can represent $2^{64}$ distinct values.

Most CPUs and GPUs cannot operate directly on large floating point numbers. This means that to do something like multiply two floating point numbers, the CPU might have to work on half of each number at a time, and do some tricky work to combine the results. Some CPUs and GPUs can operate directly on large floating point numbers, but only by using more than one core to work on a single number.

To get around this, you might chose to take a model you have that was trained with high-precision floating point weights, and reduce it to lower precision. The weights won't be exactly the same, but they'll be very close. Doing this will make the models run much faster, but you might lose some accuracy.

So the advantage of using a tool that only supports Quantized models is that models will run faster, but might be slightly less accurate.


Are the Word2Vec encoded embeddings available online? I am trying to do an NLP project and was wondering if there is anywhere online where the Word2Vec embeddings are stored (the actual n-dimmensional vectors).
I want to search up a word and see what its encoding is. I have tried looking but couldn't find anything.
Thank you

The gensim library for Python provides both a convenient way of manipulating Word2Vec embeddings, but also downloadable data sets and pretrained models. That last part is what you're looking for. Wikipedia2Vec also has multilingual embeddings. Then this guy (whom I've never heard of) has an assortement of languages I haven't seen other places. There's another 44 models hosted by the European NLPL. You can also search Kaggle, but a lot of this looks like training data, models I've already linked, or just cruft.


Could I just choose the other (non-predicted) class when the accuracy is low? I have a binary classification problem.
My neural network is getting between 10% and 45% accuracy on the validation set and 80% on the training set. Now, if I have a 10% accuracy and I just take the opposite of the predicted class, I will get 90% accuracy.
I am going to add a KNN module that shuts down that process if the inputted data is, or is very similar to the data present in the data set.
Would this be a valid approach for my project (which is going to go on my resume)?

The short answer is no, you shouldn't do that.
There is a "distribution shift" thing when you have different x-y relation on the validation set then on the train set. The distribution shift would deteriorate your model performance and you should try to avoid that. The reason it's bad - ok, you find the way to fix the model for validation data, but what about novel test data? Will it be like a train? Will it be like validation? You don't know and your model is worthless in fact.
What you can do

You should redefine the train/validation scheme. I would recommend making cross-validation with at least three splits
As mentioned in a comment, your model seems to overfit; try to apply some regularization techniques, like dropout, batch norm, data augmentation, model simplification and so on
Add some data if it's possible, that would better cover all the possible cases.


Is the minimum and maximum of a set of admissible and consistent heuristics also consistent and admissible? Let's suppose I have a set of heuristics $H$ = {$h_1, h_2, ..., h_N$}.

If all heuristics in $H$ are admissible, does that mean that a heuristic that takes the $\min(H)$ (or $\max(H)$ for that matter) is also admissible?
If all heuristics in $H$ are consistent, does that mean that a heuristic that takes the $\min(H)$ (or $max(H)$ for that matter) is also consistent?

I'm thinking about a search problem in a bi-dimensional grid that every iteration of an algorithm, the agent will have to find a different goal. Therefore, depending on the goal node, a certain heuristic can possibly better guide the agent than the others (hence the use of $\min$ and $\max$).

Yes, in both cases. Below I give two very simple proofs that directly follow from the definitions of admissible and consistent heuristics. However, in a nutshell, the idea of the proofs is that $h_{\max}(n)$ and $h_{\min}(n)$ are, by definition (of $h_{\max}$ and $h_{\min}$), equal to one of the given admissible (or consistent) heuristics, for all nodes $n$, so $h_{\max}(n)$ and $h_{\min}(n)$ are consequently admissible (or consistent).

Definitions

Consider the graph $G=(V, E, \mathcal{G})$ representing the search space, where $V$, $E$ and $\mathcal{G} \subseteq V$ are respectively the set of nodes, edges and goal nodes, and the function $w\colon E \times E \rightarrow \mathbb{R}$, which gives you the cost of each edge $e = (u, v) \in E$, where $u, v \in V$, that is, $w(e) = w(u, v) \in \mathbb{R}$ is the cost of the edge $e$.

A heuristic $h$ is admissible if $$h(n) \leq h^*(n), \forall n \in V,$$
where $h^*(n)$ is the optimal cost to reach a goal from node $n$ (that is, it is the optimal heuristic).

On the other hand, a heuristic $h$ is consistent if

\begin{align}
h(n) &\leq w(n, s) + h(s), \forall n \in V \setminus \mathcal{G}, \text{ and} \\
h(n) &= 0, \forall n \in \mathcal{G},
\end{align}
where $s$ is a successor of $n$.

Theorem 1

Given a set of admissible heuristics $H = \{ h_1, \dots, h_N \}$, then, for every $n \in V$, the heuristics $h_{\max}(n) = \max(h_1(n), \dots, h_N(n))$ and $h_{\min}(n) = \min(h_1(n), \dots, h_N(n))$ are also admissible.

Proof

Given that, $h_i(n) \leq h^*(n), \forall n \in V$ and $\forall i \in \{ 1, \dots N \}$, then $h_{\max}(n) = h_j(n) \leq h^*(n)$ (for some $j \in \{ 1, \dots N \}$)  and $h_{\min}(n) = h_k(n) \leq h^*(n)$ (for some $k \in \{ 1, \dots N \}$), so $h_{\max}$ and $h_{\min}$ are also admissible.

$\tag*{$\blacksquare$}$

Theorem 2

Given a set of consistent heuristics $H = \{ h_1, \dots, h_N \}$, then, for every $n \in V$, the heuristics $h_{\max}(n) = \max(h_1(n), \dots, h_N(n))$ and $h_{\min}(n) = \min(h_1(n), \dots, h_N(n))$ are also consistent.

Proof

Given that, for every $i \in \{ 1, \dots, N \}$,

\begin{align}
\begin{cases}
h_i(n) \leq w(n, s) + h_i(s), & \text{ if }  n \in V \setminus \mathcal{G}\\
h_i(n) = 0, & \text{ if }  n \in \mathcal{G}
\end{cases}
\end{align}

then, if $n \in V \setminus \mathcal{G}$, $h_{\max}(n) = h_j(n) \leq w(n, s) + h_j(s)$, for some $j \in \{1, \dots, N \}$, and, similarly, $h_{\min}(n) = h_k(n) \leq w(n, s) + h_{k}(s)$, for some $k \in \{1, \dots, N \}$. Similarly, if $n \in \mathcal{G}$, $h_{\max}(n) = h_{\min}(n) = h_i(n) = 0, \forall i \in \{1, \dots, N \} $. Thus, $h_{\max}$ and $h_{\min}$ are also consistent.

$\tag*{$\blacksquare$}$

Notes

$h_{\max}$ and $h_{\min}$ have been defined in the only possible reasonable way, because, given two (admissible or consistent) heuristics, one may not always dominate the other (or vice-versa), even if both are admissible and consistent.


What are some information processing models besides feedforward or multi-layered neural networks? Feedforward or multilayered neural networks, like the one in the image above, are usually characterized by the fact that all weighted connections can be represented as a continuous real number. Furthermore, each node in a layer is connected to every other node in the previous and successive layers.

Are there any other information processing models other than FFNNs or MLPs? For example, is there any system in which the topology of a neural network is variable? Or a system in which the connections between nodes are not real numbers?

Neural Network equivalents that is not (vanilla) feed forward Neural Nets:

Neural net structures such as Recurrent Neural Nets (RNNs) and Convolutional Neural Nets (CNNs), and different architectures within those are good examples.

Examples of different architectures within RNNs would would be: Long Short Term Memory (LSTM) or Gated Recurrent Unit (GRU). Both of these are well described in Colah's blog post on Understanding LSTMs

What are some alternative information processing system beside neural network

There are sooo many structures. From the top of my head: (Restricted) Boltzmann machine, auto encoders, monte carlo method and radial basis networks to name a few.

You can check out Goodfellow's Deep learning-book that is free online and get the gist of all the structures I mentioned here (most parts requires a bit of math knowledge, but he also writes about them quite intuitively).

For Recurrent Neural Nets I recommend Colah's blog post on Understanding LSTMs

Is there any system in which the topology of a neural network is variable?

Depends on what you mean with the topology of a neural network:

I think in the common meaning of topology when talking about Neural Networks is the way in which neurons are connected to form a network, varying in structure as it runs and learns. If this is what you men then the answer, in short, is yes. In multiple ways actually. On the other hand, if you mean in the mathematical sense, this answer would become a book that I wouldn't feel confortable writing. So I'll assume you mean the first.

We often do "regularization", both on vanilla NN and other structures. One of these regularization techniques are called dropout, which would randomly remove connections from the network as it is training (to prevent something called overfitting, which I'm not gonna go into in this post).

Another example for another way would be on the Recurrent Neural Network. They deal with time series, and are equipped for dealing with timeseries of different lengths (thus, "varying structure").

Does it exist neural net systems where complex numbers are used?

Yes, there are many papers on complex number machine learning structures. A quick google should give you loads of results. For example: DeepMind has a paper on
Associative Long Short-Term Memory which explores the use of complex values for an "associative memory".

Links:

Goodfellow's Deep Learning-book: deeplearningbook.org

Colah's blogpost on RNN's: colah.github.io

Paper on DeepMinds Associative LSTM: arxiv:1602.03032


What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets? Neural networks are usually evaluated by dividing a dataset into three splits:

training,
validation, and
test

The idea is that critical hyperparameters of the network such as the number of epochs and the learning rate can be tuned by testing the network on the validation data while keeping the test data completely unseen until a final evaluation that happens only after the hyperparameters have been tuned.
However, if the amount of data is very small (e.g. 10-20 examples per class), then dividing the dataset into three splits may negatively impact the model due to lack of training data, and two splits is therefore preferable. A two split approach that makes a reasonable amount of data available for training is ten-fold stratified cross validation.
My question is -- is it statistically sound to tune hyperparameters by repeatedly evaluating hyperparameter sets using cross validation? Keep in mind that there is no held-out test data in this case, as the amount of available data is too small.  I'd like some evidence/citations if possible showing that specifically for small datasets, this is the best approach for estimating the best hyperparameters that lead to the best generalizable model. Or if there is another approach that is better, I'd like to learn about that too.

I no longer really use validation that much, but rather only training and testing.  Why? Because I mostly follow Ron Kohavi's (Stanford Univ) approach to CV.  I have done a lot of validation but it seemed to be overkill, essentially causing me to ask why I have this very small-sampled parameter watch on the side from which I am supposed to learn from. You know if an ANN is overfitting when the training error diverges away from testing error -- it's an indication the model is breaking up. See my blog on this.


How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)? I'm building an agent for a racing game. In this game, there is a randomized map where there are speed boosts for the player to pick up and obstacles that act to slow the player down. The goal of the game is to reach the finishing line before the opponent.
While working on this problem, I've realized that we can almost forget about the presence of our opponent and just focus on getting the agent to the finish line as quickly as possible.
I started with a simple

$-1$ reward for every timestep
$+100$ reward for winning, and
$-100$ for losing.

When I was experimenting with this, I felt like the rewards may be too sparse, as my agent was converging to pretty poor average returns. I iterated to a function of speed and distance travelled (along with the $+100$ reward), but, after some experimentation, I started feeling like the agent might be able to achieve high returns without necessarily being the fastest to the finish line.
I'm thinking that I return to the first approach and possibly add in some reward for being in the first place (as a function of the opponent's distance behind the agent).
What else could I try? Should I try and spread the positive rewards out more for good behavior? should I create additional rewards/penalties for perhaps hitting obstacles and using boosts or can I expect the agent to learn the correlation?

Sutton and Barto state, "The reward signal is your way of communicating to the robot [agent] what you want it to achieve, not how you want it achieved." Since you stated that the goal is to reach the finish line first, then a reward of $1$ for winning, $0$ for losing, and $0$ at all other time steps seems to fit that narrative. If a draw is identical to a loss, then it should provide reward $0$; otherwise, a reward of $0.5$ seems reasonable. These rewards provide model interpretability: an expected return of $p$ (estimated with a state-value or action-value) at a certain state under the current policy would signify a $p$ chance of winning. Also, keeping the rewards in absolute value at most 1 can aid in training speed and prevent divergence, but it often isn't necessary for deep reinforcement learning problems. You most certainly can add other rewards based on partial progress towards the goal, but as it seems you found out, they may lead to incorrect results.

That being said, I would focus on the training process instead of a finely-tuned reward signal. Since there is a known goal state in the racing game (the finish line), I suggest training the RL agent by first initializing all racer agents only a few steps away from the goal state at the beginning of each episode. These episodes are shorter and therefore should provide a more dense reward signal. When your RL agent has learned a winning policy (e.g. wins more often than not), then initialize the agents slightly further from the goal state at the beginning of each episode. Also, continue to use and train the same neural network. Since the neural network presumably knows a winning policy at states near the goal state, then by initializing the agents only a few states further back, the RL agent is given an warm start and only needs to learn a policy for a few more states. The policy encoded by the neural network essentially contains a refined reward signal for the states close to the goal state since it is based on a winning policy; this helps prevent the sparsity problem caused by only supplying a reward at episode completion. You can repeat this process by initializing the agents slightly further from the goal state once the RL agent has learned a winning policy while continuing to use and train the same neural network.

Depending on your access to the environment internals, you may need other analogous approaches. For example, you could initialize the agents at the original starting line (i.e. not partway down the map) and then see which agent makes it $n$ units down the map first to determine the winner. Once a winning policy is learned by the RL agent, then gradually increase $n$ until $n$ matches the distance from the starting line to the finish line. Since it seems like you have distance traveled and distance to the opponent as features, you may instead try this method if you are unable to initialize the agents wherever you want on the map and instead can only initialize them on the starting line.

A notable benefit of the overall approach is that you can more quickly debug your algorithm on the easier environments (i.e. ones with shorter episode lengths) to be confident that the learning process is correct and focus your efforts elsewhere (e.g. the training process, including the reward signal).


Is it normal to have the root mean squared error greater on the test dataset than on the training dataset? I am new to deep learning.

I am training a model and I am getting a root mean squared error (RMSE) greater on the test dataset than on the training dataset.

What could be the reason behind this? Is this acceptable to get the RMSE greater in test data?

It is common to have root mean squared error (RMSE) greater on the test dataset than on the training dataset (this is equal to having accuracy/score higher for model in training dataset than test dataset). This normally happens because the training data are assesed on the same data that have been learnt before, while the test dataset may have data that are unknown / not common that may give more errors or misclassification when doing prediction.

But if your model shows your test dataset have way too high RMSE result rather than your training dataset RMSE result, it may indicates that overfitting happens.

If overfitting happens, there are a lot of reasons this could happen.
Referenced from https://elitedatascience.com/overfitting-in-machine-learning, some factors that causes overfitting are:

Complexity of data (e.g. there are irrelevant input features). This can be solved with removing irrelevant input features.
Not enough training data. This can be solved by training with more data (Eventhough this may not always succeed. Sometimes it may give noise towards data), etc.


How is the latent vector transforming to a feature map in DCGAN (Generator structure)? I'm working on the code trying to generate new images using DCGAN model. The structure of my code is from the PyTorch tutorial here.
I'm a bit confused trying to find and understand how the latent vector is transforming to the feature map from the Generator part (this line of code is what I'm interested in) :
nn.ConvTranspose2d(  nz, ngf * 8, 4, 1, 0, bias=False)

It means the latent vector (nz) of shape 100x1 transforming to the 512 matrices of 4x4 size (ngf=64). How  does it happen? Hence, I can't even clear to myself how the length of the latent vector influence to the generated image.
P.S. The left part of the Genarator structure is clear.
The only idea that I got is :

E.g. there is a latent vector of size 100 as input (100 random values).
We interact each value of the input latent vector with a 4x4 kernel.
In this way we get 100 different 4x4 matrices (each matrix is for one value from latent vector).
Then we summarize all these 100 matrices and get one final matrix - one feature map.
We get necessary number of feature maps taking different kernels.

Is this right? Or does it happen in other way?

Noise vector (batch_size, 100, 1, 1) is deconvoloved with filter_1 (100, 4, 4). Result is feature_map_1 (1, 4, 4). And since there are 512 filters, so there will be 512 feature maps. Output shape will be (batch_size, 512, 4, 4).
I think you need better undersanting for convolutional calculations in general. In this stack it was explained very well.


Should I just use exploitation after I have trained the Q agent? When using a trained Q-learning algorithm in an actual game, would I just use exploitation and no longer use exploration? Should I use exploration only during the training phase?

Once you have estimated the $Q$ function, you can derive the policy from it in different ways. For example, you can act greedily with respect to it (see this answer), which can be formally denoted as

$$
\pi(s) = \operatorname{argmax}_{a^*}Q(s, a), \; \forall s \in \mathcal{S}
$$
where $Q(s, a)$ is your estimated value function and $\pi$ the policy greedily derived from it.

This means that you would just exploit your current knowledge of the return. This is probably a good thing to do if you believe your value function is optimal and the dynamics of the environment don't change.

Of course, if your policy is not optimal, you may not want to always execute the greedy action. In that case, you could still perform some form of exploration (e.g. with the $\epsilon$-greedy policy).

Moreover, if the dynamics (e.g. the reward function) of your environment change over time, you could continually train your RL agent. If you are interested in continual RL, the paper Continual Reinforcement Learning with Complex Synapses (2018) could potentially be useful.


Do you need single or multiple networks to detect multiple faces? Given pictures with multiple features such as faces, can single AI algorithm detect all of them, or for better reliability is it preferred to use separate instances?

In other words I'm talking about attempt of finding all possible human faces on the same picture by a single neural network.

AFAIK, normally detection algorithms work in a sub-window of the image and not the whole of it. For example, for a specific size and orientation you slide a sub-window on the image and extract sub-images. Then you apply your algorithm on every sub-image for detection and report the size-and-orientations with positive results.

You can have a single neural network for face detection in this case or you might want to have different detectors for different orientation or any other feature, that is your decision.

There is also the technique of Combining Classifiers by which you can improve the decision of single classifiers by combining them.

Ensemble Learning is another way in which your classifiers are not trained independently but rather together. In fact, the well-known object detector of Viola and Jones uses such a technique.


What is the difference between strong-AI and weak-AI? I've heard the terms strong-AI and weak-AI used.  Are these well defined terms or subjective ones?  How are they generally defined?

The terms strong and weak don't actually refer to processing, or optimization power, or any interpretation leading to "strong AI" being stronger than "weak AI". It holds conveniently in practice, but the terms come from elsewhere. In 1980, John Searle coined the following statements:

AI hypothesis, strong form: an AI system can think and have a mind (in the philosophical definition of the term);
AI hypothesis, weak form: an AI system can only act like it thinks and has a mind.

So strong AI is a shortcut for an AI systems that verifies the strong AI hypothesis. Similarly, for the weak form. The terms have then evolved: strong AI refers to AI that performs as well as humans (who have minds), weak AI refers to AI that doesn't.

The problem with these definitions is that they're fuzzy. For example, AlphaGo is an example of weak AI, but is "strong" by Go-playing standards. A hypothetical AI replicating a human baby would be a strong AI, while being "weak" at most tasks.

Other terms exist: Artificial General Intelligence (AGI), which has cross-domain capability (like humans), can learn from a wide range of experiences (like humans), among other features. Artificial Narrow Intelligence refers to systems bound to a certain range of tasks (where they may nevertheless have superhuman ability), lacking capacity to significantly improve themselves.

Beyond AGI, we find Artificial Superintelligence (ASI), based on the idea that a system with the capabilities of an AGI, without the physical limitations of humans would learn and improve far beyond human level.


Image prediction model when data-set classes have visual similarity Lets say we have a data-set of all cats and we have to identify the cat breed based on given test image. As, the two different cat breeds have visual similarity can we use existing networks (VGG, ImageNet, GoogleNet) to solve this problem?

Should faceNet be applied here? As, the problem is similar to face detection where face characteristics of two different people are same yet it can correctly recognize a person.
What if with visual similarity in data-set we have only few example of each class? Like for a problem (random) we have good amount of data but for each class we have only few examples.

Is there any model that can be applied here?

You could use transfer learning. Find some state of the art neural net (something newer than VGG) already trained for image classification. Then remove the last layer and add a new one, training it while keeping the remaining old weights fixed. A simple softmax at the end would work.This is the simplest approach.

Because you're mostly only training the last layer, you won't need as much data.

I think FaceNet will perform poorly because it uses different features and you would need to retrain all its layers.

If you have too little data for some classes, maybe you should use an autoencoder as a features extractor. Then you could classify an image by comparing its feature vector to those for each breed, for example with cosine similarity.


What are some resources on continuous state and action spaces MDPs for reinforcement learning? Most introductions to the field of MDPs and Reinforcement learning focus exclusively on domains where space and action variables are integers (and finite). This way we are introduced quickly to Value Iteration, Q-Learning, and the like.

However, the most interesting applications (say, flying helicopters) of RL and MDPs involve continuous state space and action spaces. I'd like to go beyond basic introductions and focus on these cases but I am not sure how to get there.

What are some resources on continuous state and action spaces MDPs for reinforcement learning? And what areas do I need to know or study to understand these cases in-depth?

There is a small survey of continuous states, actions and time in reinforcement learning in my thesis proposal.

Regarding books, Reinforcement Learning: State-of-the-Art seems to be pretty up-to-date from the excerpts I've read.


How can a reinforcement learning agent generalize if it is trained against only one opponent? I started teaching myself about reinforcement learning a week ago and I have this confusion about the learning experience. Let's say we have the game Go. And we have an agent that we want to be able to play the game and win against anyone. But let's say this agent learn from playing against one opponent, my questions then are:

Wouldn't the agent (after learning) be able to play only with that opponent and win? It estimated the value function of this specific behaviour only.
Would it be able to play as good with weaker players?
How do you develop an agent that can estimate a value function that generalizes against any behaviour and win? Self-play? If yes, how does that work?

Reinforcement Learning (RL) at its core does not have anything directly to say about adversarial environments, such as board games. That means in a purely RL set up, it is not really possible to talk about the "strength" of a player.

Instead, RL is about solving consistent environments, and that consistency requirement extends to any opponents or adversarial components. Note that consistency is not the same as determinism - RL theory copes well with opponents that effectively make random decisions, provided the distribution of those decisions does not change based on something the RL agent cannot know.

Provided an opponent plays consistently, RL can learn to optimise against that opponent. This does not directly relate to the "strength" of an opponent, although usually strong opponents present a more challenging environment to learn overall.

  Wouldn't the agent (after learning) be able to play only with that opponent and win? since it estimated the value function of this specific behavior only.

If the RL has enough practice and time to optimise against the opponent, then yes the value function (and any policy based on it) would be specific to that opponent. Assuming, the opponent did not play flawlessly, then the RL would learn to play such it would win as often as possible against the opponent.

When playing against other opponents, the success of the RL agent will depend on how similar the new opponent was to the original that it trained against.

  would it be able to play as good with weaker players?

As stated above, there is not really a concept of "stronger" or "weaker" in RL. It depends on the game, and how general the knowledge is that strong players require in order to win.

In theory you could construct a game, or deliberately play strongly, but with certain flaws, so that RL would play very much to counter one play style, and would fail against another player that did not have the same flaws.

It is difficult to measure this effect, because human players learn from their mistakes too, and are unlikely to repeat the exact same game time after time, but with small variations at key stages. Humans do not make consistent enough opponents, and individual humans do not play enough games at each stage of their ability to study fine-grained statistics of their effective policies.

In practice it seems likely that the effect of weakening against new players would be there in RL, due to sampling error if nothing else. However, it seems that the "strength" of players as we measure them in any game of skill such as chess or go, does correlate with a generalised ability. In part this is backed up by consistent results with human players and Elo ratings.

Any game where you can form "rings" of winning players:

Player B consistently beats Player A
Player C consistently beats Player B
Player A consistently beats Player C

Could cause issues of the type you are concerned about when applying RL to optimise an artificial agent.

  How do you develop an agent that can estimate a value function that generalizes against any behavior and win?

If is possible to play perfectly, then a value function which estimated for perfect play would work. No player could beat it. Think of Tic Tac Toe - it is relatively easy to construct perfect play value functions for it.

This is not achievable in practice in more complex games. To address this, and improve the quality of its decisions, what AlphaGo does is common to many game-playing systems, using RL or not. It performs a look-ahead search of positions. The value function is used to guide this. The end result of the search is essentially a more accurate value function, but only for the current set of choices - the search focuses lots of computation on a tiny subset of all possible game states.

One important detail here is that this focus applies at run time whilst playing against any new opponent. This does not 100% address your concerns about differing opponents (it could still miss a future move by a different enough opponent when searching). But it does help mitigate smaller statistical differences between different opponents.

This search tree is such a powerful technique that for many successful game playing algorithms, it is possible to start with an inaccurate value function, or expert heuristics instead, which are fixed and general against all players equally. IBM's Deep Blue is an example of using heuristics.

  self-play?  if yes, how does that work?

Self-play appears to help. Especially in games which have theoretical optimal play, value functions will progress towards assessing this optimal policy, forming better estimates of state value with enough training. This can give a better starting point than expert heuristics when searching.


Speech recognition software implementation by open-source API I am making my speech recognition project for PC (working on Windows 8) and am new in this area. The project should have basic functionality, like dictation with accuracy in an e-mail, notepad, etc., and should respond to local commands of PC.
I am using sphinx4 for my speech recognition project.  I'm trying to determine if there is a better open-source API than CMU Sphinx in terms of accuracy and large vocabulary.
Does Kaldi (deep neural network-based) perform better than CMU Sphinx (HMM-based)? Are the two platforms better for different kinds of applications?
Essentially:

I want to increase my speech recognition system accuracy and vocabulary.  Which might be most optimal: Kaldi or CMU Sphinx?

Also, wondering what is the difference between a speech API and a speech Engine, and, more generally, as a developer what I will require to develop my software?
Please help me to get a clear understanding of the above questions and, if possible, provide some speech recognition developer or researcher community links.
Any and all comments, suggestions, and answers are welcome!

The difference between a Speech API and a Speech Engine is: Speech API's enable developers to integrate speech recognition technologies into developer apps. On the other hand a speech engine is software that gives your computer the ability to play back text in a spoken voice. (Source msdn library)

Below is a list of speech recognition tool-kits and their features.

Tensorflow - Although tensorflow doesn't arrive packaged with speech recognition libraries by default. You can use seq2seq models which have achieved high levels of accuracy in speech recognition. A few of the advantages of using tensorflow for speech recognition include: It comes with tensorboard which is useful in visualising and fine-tuning your network, it's architecture is highly modular which allows you to experiment with different voice libraries and finally it is portable meaning you can run it on GPU's, CPU's, servers or even mobile computing platforms.

Microsoft's CNTK - Microsofts Cognitive Toolkit delivers excellent results when it come to speech recognition. In recent times CNTK has outperformed humans in transcribing speech to text. Some of its perks are its efficient resource usage additionally it was originally built for speech recognition systems and consequently it is very effective at working with time series data.

CMU Sphinx - CMU Sphinx is a speech recognition system developed at Carnegie Mellon University. The advantages of using CMU Sphinx are: it is multilingual and supports most international languages, it has excellent commercial support, it has a light mobile version called pocketsphinx, it has a wide range of tools for different purposes i.e. keyword spotting, alignment and pronunciation evaluation. It also enjoys active support from the Carnegie Mellon University.

Kaldi - Kaldi aims to provide speech recognition software that is flexible and extensible. Kaldi has powerful features such as pipelines that are highly optimized for parallel computing i.e. training models on the GPU. Additionally it supports speaker identification and detection of errors in transcripts.

Mozilla Deep Speech - This project aims at providing speech interfaces for the web. This project achieved a word error rate on LibriSpeech's test-clean of 6.5% which is commendable.

I found this paper comparing open source speech recognition toolkits to be relevant to your question http://suendermann.com/su/pdf/oasis2014.pdf

Siraj Raval has an excellent tensorflow speech recognition tutorial. It is available here https://github.com/llSourcell/tensorflow_speech_recognition_demo


How does the Alpha Zero's move encoding work? I am a beginner in AI. I'm trying to train a multi-agent RL algorithm to play chess. One issue that I ran into was representing the action space (legal moves/or honestly just moves in general) numerically. I looked up how Alpha Zero represented it, and they used an 8x8x73 array to encode all possible moves. I was wondering how it actually works since I got a bit confused in their explanation:

A move in chess may be described in two parts: selecting the piece to move, and then selecting among the legal moves for that piece. We represent the policy $\pi(a \mid s)$ by a $8 \times 8 \times 73$ stack of planes encoding a probability distribution over 4,672 possible moves. Each of the $8 \times 8$ positions identifies the square from which to "pick up" a piece. The first 56 planes encode possible "queen moves" for any piece: a number of squares $[1..7]$ in which the piece will be moved, along one of eight relative compass directions {N, NE, E, SE, S, SW, W, NW}. The next 8 planes encode possible knight moves for that piece. The final 9 planes encode possible under-promotions for pawn moves or captures in two possible diagonals, to knight, bishop or rook respectively. Other pawn moves or captures from the seventh rank are promoted to a queen.

How would one numerically represent the move 1. e4 or 1. NF3 (and how would the integer for 1. NF3 differ from 1. f3) for example? How do you tell what integer corresponds to which move? This is what I'm essentially asking.

Let's do the code, so all the details are down.
Encoding dictionary:
codes, i = {}, 0
for nSquares in range(1,8):
    for direction in ["N", "NE", "E", "SE", "S", "SW", "W", "NW"]:
        codes[(nSquares,direction)] = i
        i += 1

You'll see that the codes dictionary will have 56 entries in it for each (nSquares,direction) pair.
The knight moves we'll encode as the long "two"-cell edge move first and the short "one"-cell edge second:
for two in ["N","S"]:
    for one in ["E","W"]:
        codes[("knight", two, one)] , i = i , i + 1
for two in ["E","W"]:
    for one in ["N","S"]:
        codes[("knight", two, one)] , i = i , i + 1

Now we should have 64 codes. As I understand, the final 9 moves are when a pawn reaches the final rank and chosen to be underpromoted. It can reach teh final rank either by moving N, or by capturing NE, NW. Underpromotion is possible to three pieces. Writing the code:
for move in ["N","NW","NE"]:
    for promote_to in ["Rook","Knight","Bishop"]:
        codes[("underpromotion", move, promote_to)] , i = i , i + 1

We get 73 codes as described.
Policy
The distribution over actions is a (8,8,73) tensor (it is not formally a "policy", since policy should also depend on state, but lets cut this corner for this discussion):
policy = np.zeros((8,8,73))

Let's also do codes for columns for convenience:
columns = { k:v for v,k in enumerate("abcdefgh")}

How would one numerically represent the move 1. e4

The first two dimensions choose the figure you are moving. So, that'd be e2 pawn. And we move north 'N' by 2 cells.
So, we put 1 into the tensor at the appropriate indices. Note that you have to subtract 1 from the row index, to make it zero-based.
e4policy = np.zeros((8,8,73))
e4policy[ columns['e'] , 2 - 1 , codes[(2 , "N")]] = 1

How would one numerically represent the move 1. NF3

The first two dimensions choose the figure you are moving. So, that'd be g1 knight. And we perform north-west N,W knight jump.
NF3policy = np.zeros((8,8,73))
NF3policy[ columns['g'] , 1 - 1 , codes[("knight", 'N' , 'W')]] = 1

Generally, the policy is a probability distribution over all possible moves, so the policy tensor would have several non-zero probability values in it. For example an opening policy that does 1.e4 or 1.Nf3 with 50/50 probability would be:
openingPolicy = (e4policy + NF3policy) / 2

Hope this clears things up.


How to run a Monte Carlo Tree Search MCTS for stochastic environment? For MCTS there is an expansion phase where we make a move and list down all the next states. But this is complicated by the fact that for some games, after making the move, there is a stochastic change to the environment. Consider the game 2048, after I make a move, random tile is generated. So the state of the world after my next move is a mix of possibilities!
How does MCTS work in a stochastic environment? I am having trouble understanding how to keep track of the expansion, do I expand all stochastic possibilities and weight the return via their chance of happening?

I am having trouble understanding how to keep track of the expansion, do I expand all stochastic possibilities and weight the return via their chance of happening?

This is indeed one option you can take. This would be very similar in spirit to the idea of "Expectimax" as a variant of minimax for non-deterministic games, in the sense that you'll include explicit "chance nodes" in your tree. When running into such a chance node later on again during a Selection phase, of a later MCTS iteration, you can just select a path of the tree to follow based on a "dice roll". Importantly, note that this option is only actually available if you have explicit knowledge of exactly when chance events occur, which states they can lead to, and with which probabilities they lead to different states. We also assume that this is feasible, i.e. that you don't have a crazy high (or infinite) number of slightly different game states you could reach.
An alternative option is to use an "open-loop" variant of MCTS. Your nodes would no longer represent game states, but only be representative of the sequence of actions leading to them. You would no longer store any game states in any nodes, but always regenerate them from scratch when traversing the tree, starting from the root node. You would no longer have any explicit chance nodes, but instead have states being representative of larger sets of states that could possibly be reached by following the corresponding path from the root node. For more on this, see my answer to this other question. The advantage of this approach is that it does not require explicit knowledge of all the possible states you can reach due to chance events, do not need explicit knowledge of the probabilities, and can just sample instead of explicitly enumerating every possible outcome.


Is a binary attribute type the same as binomial attribute type? I am not sure if I can use the words binomial and binary and boolean as synonyms to describe a data attribute of a data set which has two values (yes or no). Are there any differences in the meaning on a deeper level?

Moreover, if I have an attribute with three possible values (yes, no, unknown), this would be an attribute of type polynominal. What further names are also available for this type of attribute? Are they termed as "symbolic"?

I am interested in the realtion between the following attribute type: binary, boolean, binominal, polynominal (and alternative describtions) and nominal.

@SmallChess's answer is a good start, but there are some additional parts to the question.

binary variables or binary data consist of data with the values 0 or 1, and no other values. We usually don't talk about "binary distributions", because it's only data, variables, or outcomes that can be binary. A distribution might produce binary data, but is not itself binary because its parameters typically take on real-values.

A binomial distribution is a distribution that produces binary data. In particular, it is a random process that produces the value 1 with probability $$p$$, and the value 0 with probability $$1-p$$. Notice that although it makes binary data, it is not itself a kind of data, and is in fact charactorized by a non-binary number (p).

Boolean data takes on the values true or false. Often, but not always, these are stored as 0's and 1's. The distinction is that boolean data may not be stored numerically. There might also be different expectations about how Boolean data should be processed (for instance, $$true + true = true$$, but $1 + 1 = 2$.

I am not aware of the term polynomial being applied to data. However, multinomial distributions are probability distributions that produce 0 with probability $p_0$, 1 with probability $p_1$, 2 with probability $p_2$, and so on, producing $p_n$ with probability $1 - \sum_{i=0}^{n-1} p_i$ for $n$ different numbers. Like binomial distributions, multinomial distributions are characterized by a set of real-valued numbers, and are distinct from the kind of data they generate.

Categorical data takes on values from a set of categories. The example you give (yes, no, maybe) is not strictly multinomial data, but could be generated from a multinomial distribution by mapping the values 0, 1 and 2 onto yes, no and maybe. Note again that categorical data might be non-numeric. Operations like adding might be non-sensical.

Cardinal data isn't something you asked about, but arises when data can be nicely ordered. For example, playing cards are easily mapped to the numbers 1-13, and can have reasonable semantic meaning when represented this way (e.g. A + 2 = 3, and 1 + 2 = 3).

Nominal Data is just literal numbers that mean exactly what they purport to mean. For example, if you store the number of cans of beer a customer purchased, that would be nominal data.


How does this part of algorithm works?(K-means) I can't understand the red box. what do does that 1 do before {c(I)=j} ?
Also how does all the algorithms work? May someone explains the to steps more clearly by one example!

1_{c^(i) = j} means if c^(i) is equal to j the value is 1 and zero otherwise.

In summary, k-means try to solve Voronoi cells for each mean point, and then update mean of members of a cell and update Voronoi cells again. This step is repeated until there is no change in members of the cells.


Does there exist a resource for vetting banned words for chatbots? So, Tay the racist tweeter bot... one thing that could have prevented this would have been to have a list of watchwords to not respond to, with some logic similar to foreach (word in msg) {if (banned_words.has(word)) disregard()}.

Even if that wouldn't, what I'm getting at is obvious: I am building a chatterbot that must be kid-friendly. For my sake and for the sake of whoever finds this question, is there a resource consisting of a .csv or .txt of such words that one might want to handle? I remember once using a site-blocking productivity extension that had visible its list of banned words; not just sexually charged words, but racial slurs, too.

I have not found one other than scraping a few pages from Urban Dictionary, I built my list via crowdsourced style and got a number of interesting words I had not considered.

Start with the worst words you can think of, then try slang and accidental or on purpose misspellings of them


What is the difference between a convolutional neural network and a regular neural network? I've seen these terms thrown around this site a lot, specifically in the tags convolutional-neural-networks and neural-networks.

I know that a neural network is a system based loosely on the human brain. But what's the difference between a convolutional neural network and a regular neural network? Is one just a lot more complicated and, ahem, convoluted than the other?

TLDR:
The convolutional-neural-network is a subclass of neural-networks which have at least one convolution layer. They are great for capturing local information (e.g. neighbor pixels in an image or surrounding words in a text) as well as reducing the complexity of the model (faster training, needs fewer samples, reduces the chance of overfitting).

See the following chart that depicts the several neural-networks architectures including deep-conventional-neural-networks: .

Neural Networks (NN), or more precisely Artificial Neural Networks (ANN), is a class of Machine Learning algorithms that recently received a lot of attention (again!) due to the availability of Big Data and fast computing facilities (most of Deep Learning algorithms are essentially different variations of ANN).

The class of ANN covers several architectures including Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN) eg LSTM and GRU, Autoencoders, and Deep Belief Networks. Therefore, CNN is just one kind of ANN.

Generally speaking, an ANN is a collection of connected and tunable units (a.k.a. nodes, neurons, and artificial neurons) which can pass a signal (usually a real-valued number) from a unit to another. The number of (layers of) units, their types, and the way they are connected to each other is called the network architecture.

A CNN, in specific, has one or more layers of convolution units. A convolution unit receives its input from multiple units from the previous layer which together create a proximity. Therefore, the input units (that form a small neighborhood) share their weights.

The convolution units (as well as pooling units) are especially beneficial as:

They reduce the number of units in the network (since they are many-to-one mappings). This means, there are fewer parameters to learn which reduces the chance of overfitting as the model would be less complex than a fully connected network.
They consider the context/shared information in the small neighborhoods. This future is very important in many applications such as image, video, text, and speech processing/mining as the neighboring inputs (eg pixels, frames, words, etc) usually carry related information.

Read the followings for more information about (deep) CNNs:

ImageNet Classification with Deep Convolutional Neural Networks
Going Deeper with Convolutions

p.s. ANN is not "a system based loosely on the human brain" but rather a class of systems inspired by the neuron connections exist in animal brains.


What are the pros and cons of using sigmoid or softmax approach when dealing with 2 classes? I know that when using Sigmoid, you only need 1 output neuron (binary classification) and for Softmax - it's 2 neurons (multiclass classification). But for performance improvement (if there is one), is there any difference which of these 2 approaches works better, or when would you recommend using one over the other. Or maybe there are certain situations when using one of these is better than the other.
Any comments or shared experience will be appreciated.

Sigmoid is used for binary cases and softmax is its  generalized version for multiple classes. But, essentially what they do is over exaggerate the distances between the various values.
If you have values on a unit sphere, apply sigmoid or softmax on those values would lead to the points going to the poles of the sphere.


Is it possible to retrieve the optimal policy from the state value function? One can easily retrieve the optimal policy from the action value function but how about obtaining it from the state value function?

You can obtain the optimal policy from the optimal state value function if you also have the state transition and reward model for the environment $p(s',r|s,a)$ - the probability of receiving reward $r$ and arriving in state $s'$ when starting in state $s$ and taking action $a$.
This looks like:
$$\pi^*(s) = \text{argmax}_a [\sum_{s',r} p(s',r|s,a)(r + \gamma v^*(s'))]$$
There are variations of this function, depending on how you represent knowledge of the environment. For instance, you don't actually need the full distribution model for reward, an expected reward function and separate distribution model for state transition rules would also work.
Without at least an approximate model of the environment, you cannot derive a policy from state values. If all you have is state values, then to pick an optimal action, you absolutely need the ability to look ahead a time step at what the next state might be for each action choice.


How many GPUs can these deep learning algorithms be parallelized across (batch parallelization)? The deep learning algorithms I would to know the limits of are:

CNTK
Caffe
TensorFlow
Torch7
Theano

For example: I've heard TensorFlow is near impossible to parallelize on 8 GPUs and above. So, in this case, the limit would be 8.

Disclaimer: this answer refers solely to TensorFlow, as my knowledge of the remaining frameworks is limited.

Where did you hear that TensorFlow is near impossible to parallelize on more than 8 GPUs? With a large enough network, any number of GPUs can be used to speed-up training, as shown in the tutorial. This snippet of code introduces the multiplication of 2 constants in 2 separate GPUs. This allows you to run the graph on both simultaneously.

for d in ['/gpu:2', '/gpu:3']:
    with tf.device(d):
        a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])
        b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])
        c.append(tf.matmul(a, b))

Even if there were some sort of limit on how many GPUs TensorFlow can use per process, TF can be distributed across processes and machines. Asynchronous algorithms (like A3C) can be spread across, for example, 16 machines (what I am doing in my laboratory), where each machine uses its resources (in my case, I use CPU, but the change to a GPU would be trivial).

With a complex enough network and a well implemented solution, you can take advantage of any hardware resources you have.


How can an AI freely make decisions? Suppose a deep neural network is created using Keras or Tensorflow. Usually, when you want to make a prediction, the user would invoke model.predict. However, how would the actual AI system proactively invoke their own actions (i.e. without the need for me to call model.predict)?

Neural networks, deep learning and other supervised learning algorithms do not "take actions" by themselves, they lack agency.

However, it is relatively easy to give a machine agency, as far as taking actions is concerned. That is achieved by connecting inputs to some meaningful data source in the environment (such as a camera, or the internet), and connecting outputs to something that can act in that environment (such as a motor, or the API to manage an internet browser). In essence this is no different from any other automation that you might write to script useful behaviour. If you could write a series of tests, if/then statements or mathematical statements that made useful decisions for any machine set up this way, then in theory a neural network or similar machine learning algorithm could learn to approximate, or even improve upon the same kind of function.

If your neural network has already been trained on example inputs and the correct actions to take to achieve some goal given those inputs, then that is all that is required.

However, training a network to the point where it could achieve this in an unconstrained environment ("letting it loose on the internet") is a tough challenge.

There are ways to train neural networks (and learning functions in general) so that they learn useful mappings between observations and actions that progress towards achieving goals. You can use genetic algorithms or other search techniques for instance, and the NEAT approach can be successful training controllers for agents in simple environments.

Reinforcement learning is another popular method that can also scale up to quite challenging control environments. It can cope with complex game environments such as Defense of the Ancients, Starcraft, Go. The purpose of demonstrating AI prowess on these complex games is in part showing progress towards a longer-term goal of optimal behaviour in the even more complex and open-ended real world.

State of the art agents are still quite a long way from general intelligent behaviour, but the problem of using neural networks in a system that learns how to act as an agent has much research and many examples available online.


Why does the variational auto-encoder use the reconstruction loss? VAE is trained to reduce the following two losses.

KL divergence between inferred latent distribution and Gaussian.

the reconstruction loss

I understand that the first one regularizes VAE to get structured latent space. But why and how does the second loss help VAE to work?
During the training of the VAE, we first feed an image to the encoder. Then, the encoder infers mean and variance. After that, we sample $z$ from the inferred distribution. Finally, the decoder gets the sampled $z$ and generates an image. So, in this way, the VAE is trained to make the generated image to be equal to the original input image.
Here, I cannot understand why the sampled $z$ should make the original image, since the $z$ is sampled, it seems that the $z$ does not have any relationship between the original image.
But, as you know, VAE works well. So I think I miss something important or understand it in a totally wrong way.

The VAE uses the ELBO loss, which is composed of the KL term and the likelihood term. The ELBO loss is a lower bound on the evidence of your data, so if you maximize the ELBO you also maximize the evidence of the given data, which is what you indirectly want to do, i.e. you want the probability of your given data (i.e. the data in your dataset) to be high (because you want to use the VAE for the generation of inputs similar to the ones in your dataset). So, the idea is that you optimize both the KL term and the reconstruction (or likelihood) term jointly (i.e. the ELBO). Why? Because, as I just said, the ELBO is the Evidence Lower BOund on the given data, so, by maximizing it, you are also maximizing the evidence of your data. In other words, if you maximize the ELBO, you are finding a decoder that will have a high probability of reconstructing your inputs (i.e. the likelihood term), but, at the same time, you want your encoder to be constrained (i.e. KL term). Please, read this answer for further details.

  Here, I cannot understand why the sampled $z$ should make the original image, since the $z$ is sampled, it seems that the $z$ does not have any relationship between the original image.

The relationship is that you will be maximizing the ELBO, which implies (and you can see this implication only if you are familiar with the ELBO loss) you will be minimizing the KL divergence between your posterior and the prior to generate the samples $z$ (i.e. minimizing because there will be a minus in front of the KL term in the ELBO loss) and maximizing the probability of the reconstructed input. More precisely, $z$ is used to reconstruct the input (i.e. the decoder does this), which is then used to calculate the reconstruction loss.

In the mathematical formulations, you will see that the likelihood term of the ELBO is $p(x \mid z)$, i.e. the likelihood of the input $x$ given $z$. The $z$ is the input to the decoder, which produces a reconstruction of $x$. In practice, people will e.g. use the cross-entropy to then calculate the "reconstruction loss" (e.g. see this PyTorch implementation), which should correspond to this likelihood term $p(x \mid z)$. Why does the cross-entropy correspond to a likelihood? Because you can actually prove that the cross-entropy is equivalent to the negative log-likelihood. (Also, note that, in the ELBO loss, $p(x \mid z)$ does not appear, but the logarithm of $p(x \mid z)$ appears, but, for simplicity, I have used $p(x \mid z)$ rather than $\log p(x \mid z)$ above.)


Why does GPT-2 Exclude the Transformer Encoder? After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can only look at prior tokens.
Why does GPT-2 not require the encoder part of the original transformer architecture?
GPT-2 architecture with only decoder layers

GPT-2 is a close copy of the basic transformer architecture.
GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the MASKING in the multi-head attention block, the decoder is only allowed to glean information from the prior words in the sentence. It works just like a traditional language model as it takes word vectors as input and produces estimates for the probability of the next word as outputs but it is auto-regressive as each token in the sentence has the context of the previous words. Thus GPT-2 works one token at a time.
BERT, by contrast, is not auto-regressive. It uses the entire surrounding context all-at-once. GPT-2 the context vector is zero-initialized for the first word embedding.


Which algorithm should I choose for lead scoring I am a c# senior developer and I got a task to try and predict the potential in each new client, or maybe the worth of each customer.
I don't have experience with machine learning, but I played with accord-framework.net and got some nice results on simple task.

My data model for training is:

GeoLocation, // the country of ip when registed. iso code string
Age, // number
DateRegistered, //date time
Email, //string can be broken to vendors as catergorial (gmail, yahoo, microsoft and such)
EmailValidated, //is the email really exists. bool
PhoneNumber, //string
PhoneNumberValidated, // is the phone number really exists
CampaignName, //string (may be categirial)
UserAgent, //string should I make it categorial? (has info about browser, device, verndor, operation system and such, long string)
LandedOnPage, //string first url the customer entered from
RegisteredFromPage, //string url of the page that the user registered from
RefererUrl, //string url the client came to our site from,
NumberOfPurchases, //the amount of times the customer puschase something on our site
CustomerValueUsd, //the total amount of USD the customer spent in our site

The output shoud be CustomerValueUsd

I have a lot of data in the history, so I can back test it.

My questions:

Does it make sense to do this task even though I don't have an experience with machine learning? How complicated is this task considering I'm using a well known framework?
Assuming that I'm taking the task, which algorithm should I choose to perform this kind of task?
How should I build the training data? see my comments, do you think my comments are ok to start with? or maybe I can break the data directly?

ML is a wide and deep topic to cover.

What you need is "classification" to predict an outcome. At first, you need to convert everything to numbers (check word2vec). Then, pre-process the data (Check what is a normalization for example). Train a classifier by using any ML techniques (i.e. linear classifier, SVM, deep learning, neural classifier). Then you can use that regression function to predict further values. If you need much more detailed way to do that, it is better to buy a 'thick' book that explains all other options.


How does Q-learning work in stochastic environments? The Q function uses the (current and future) states to determine the action that gets the highest reward.

However, in a stochastic environment, the current action (at the current state) does not determine the next state.

How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?

How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?

The learned $Q$-function is not only used during training, but also after training (in what we may call "deployment", when we expect a trained agent to behave according to what it has learned).

However, the reliance on future states is only there during training, it is no longer required for deployment.

During training, we use the following $Q$-learning update rule:

$$Q(s, a) \gets (1 - \alpha) Q(s, a) + \alpha \left( R + \gamma \color{red}{\max_{a'} Q(s', a')} \right),$$

where $s'$ is the state we reach after executing $a$ in $s$. Here, the $\color{red}{\text{red}}$ part is the part where we rely on knowledge of the future $s'$. This is available in training because we can simply pick $a$, execute it in $s$, observe $s'$, and only then trigger our update step.

Outside of training (and actually also during training), we also rely on our $Q(s, \cdot)$ function for the selection of actions. We typically select an action $a$ according to $a = \arg\max_a Q(s, a)$; we select the action $a$ that maximises $Q(s, a)$ in our current state $s$. The important thing to note here is that there is no $s'$ term in this description of how we select actions: we do not require knowledge of our future state.

Note: in my answer I decided to answer the question literally as it is written, i.e. I'm explaining how $Q$-learning can still work in the described setting where actions $a$ have no influence whatsoever on the future state reached.

In practice, I would never recommend actually using $Q$-learning in such a setting, and instead refer to Neil Slater's comment about Contextual Multi-Armed Bandit algorithms likely providing a better solution.


Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval? Say I've got two Markov Decision Processes (MDPs):
$$\mathcal{M_0} = (\mathcal{S}, \mathcal{A}, P, R_0),\quad\text{and}\quad\mathcal{M}_1 = (\mathcal{S}, \mathcal{A}, P, R_1)$$
Both have the same set of states and actions, and the transition probabilities are also the same. The only difference is in the reward functions $R_0$ and $R_1$. Suppose that we've found an optimal deterministic policy $\pi^*_0$ for the problem $\mathcal{M}_0$ and we've checked that this policy is also optimal for $\mathcal{M}_1$
$$\pi_0^*(s) = \arg\max\limits_a Q^*_0(s,a)\qquad  Q_1^*(s,\pi_0^*(s)) = \max\limits_a Q^*_1(s,a)$$
Now, given the two MDPs one can build a whole family of MDPs interpolating between them:
$$\mathcal{M}_\alpha = (\mathcal{S}, \mathcal{A}, P, \alpha R_0 + (1-\alpha) R_1)$$
Where $\alpha\in[0,1]$ is the interpolation parameter between the two problems - the rewards are linearly changing from $R_0$ to $R_1$ with this parameter. My question is - in general. will $\pi_0^*$ be optimal for all MDPs in the middle of interpolation interval?
$$Q_\alpha(s,\pi_0^*(s))\stackrel{?}{=}\max\limits_aQ^*_\alpha(s,a),\; \forall\alpha\in[0,1]$$
I feel like this could be generally true due to linearity of the dependence and convexity of the optimization problem. But I cannot neither prove it, nor find a counterexample.

I believe the claim is true. Here is my attempt at a proof.
Let us consider the optimal infinite horizon value function $V_\alpha^*$ of $\mathcal{M}_\alpha$ at an arbitrary state $s \in S$.
The value $V_\alpha^*(s)$ is the expected sum of discounted rewards under an optimal policy $\pi_\alpha^*$, i.e.,
\begin{equation}
      V_\alpha^*(s) = \mathbb{E}_{\rho_\alpha}\left[\sum\limits_{t=0}^{\infty}\gamma^t\left( \alpha R_0(s_t,\pi_\alpha^*(s_t)) + (1-\alpha)R_1(s_t, \pi_\alpha^*(s_t)) \right)\middle| s_0 = s, \right],
\end{equation}
with the expectation taken with respect to the steady state distribution $\rho_\alpha$ of states under $\pi_\alpha^*$.
In the following, I drop the condition $s_0=s$ for conciseness, but you can assume it's in each expectation.
Now, break up the sum:
\begin{equation}
     V_\alpha^*(s) = \mathbb{E}_{\rho_\alpha}\left[ \alpha\sum\limits_{t=0}^{\infty}\gamma^t R_0(s_t,\pi_\alpha^*(s_t)) + (1-\alpha)\sum\limits_{t=0}^{\infty}\gamma^t R_1(s_t,\pi_\alpha^*(s_t)) \right].
\end{equation}
Then, by linearity of expectation:
\begin{equation}
      V_\alpha^*(s) = \alpha\mathbb{E}_{\rho_\alpha}\left[ \sum\limits_{t=0}^{\infty}\gamma^t R_0(s_t,\pi_\alpha^*(s_t)) \right] + (1-\alpha)\mathbb{E}_{\rho_\alpha}\left[ \sum\limits_{t=0}^{\infty}\gamma^t R_1(s_t,\pi_\alpha^*(s_t)) \right].
\end{equation}
Note that the first expectation term is the value of $\pi_\alpha^*$ in $\mathcal{M}_0$, and the second expectation term is the value of $\pi_\alpha^*$ in $\mathcal{M}_1$.
We already know that $\pi_0^*(s)$ is optimal in $\mathcal{M}_0$ with reward function $R_0$, and $\pi_1^*(s)$ is likewise optimal in $\mathcal{M}_1$ with $R_1$. Further, as per your assumption, $\pi_0^*(s) = \pi_1^*(s)$.
So $\pi_\alpha^*$ can be at most as good as $\pi_0^*$ with reward function $R_0$ (resp., with $R_1$):
\begin{equation}
      V_\alpha^*(s) \leq \alpha\mathbb{E}_{\rho_0}\left[ \sum\limits_{t=0}^{\infty}\gamma^t R_0(s_t,\pi_0^*(s_t)) \right] + (1-\alpha)\mathbb{E}_{\rho_0}\left[ \sum\limits_{t=0}^{\infty}\gamma^t R_1(s_t,\pi_0^*(s_t)) \right].
\end{equation}
Note that we know take the expectation under the steady state distribution $\rho_0$ of $\pi_0^*$ instead.
Thus, we have shown that $V_\alpha^*(s) \leq \alpha V_0^*(s) + (1-\alpha)V_1^*(s)$.
Now it remains to argue that the case with a strict less than relation is not possible.
Suppose this were the case, and we would have $V_\alpha^*(s) < \alpha V_0^*(s) + (1-\alpha)V_1^*(s)$.
But then $\pi_0^*$ would attain a higher value than $\pi_\alpha^*$ in $\mathcal{M}_\alpha$, which is a contradiction (because we assumed that $\pi_\alpha^*$ is an optimal policy for $\mathcal{M}_\alpha$).
Thus, $V_\alpha^*(s) = \alpha V_0^*(s) + (1-\alpha)V_1^*(s)$ and furthermore, acting according to $\pi_0^*$ is optimal also in $\mathcal{M}_\alpha$.


Identifying geometry in images There is a lot of information on the internet about image classification and object identification. I want to know how I can extract (or create) geometric information from an image.

For example, if the image classifier can identify that there is a sphere in the image, how can i generate 3d vertices's and edges from that knowledge?

I want to find all the edges and points on these shapes

To obtain the edges, you need to use an edge detector. Many exists, more or less efficient depending on the input image. Among the most broadly used are the Sobel and the Canny edge detectors that are implemented in many libraries for various programming languages. (Matlab, scikit-image in Python)

To obtain straight lines from these edges, I suggest that you start by using the Hough transform also available in many libraries for various programming languages. (Matlab, scikit-image in Python)

Lines output from the Hough transform are usually given in there parametric form. From there, it is easy to find the angles between the different lines. I imagine that some rules can be derived from there to identify a regular polygon that is in the image.

Regarding obtaining 3D information from an image, note that there isn't enough information in a single 2D image to reconstruct its 3D coordinates, since we are projecting information into a lower dimensional space.