Importance of starting state and player in RL for Tic Tac Toe So I am simulating a Tic Tac Toe game with a human opponent. The way the RL trains is through policy/value iterations for a fixed number of iterations all specified by the user. Now whether the human player has turn 1 or turn 2 will decide the starting state (1st move by human or empty). Now the starting states for the 1st case can differ as the human can make 9 different moves. So my questions are:

In Tic Tac Toe what is the effect of starting state in the State and Action Value function?
Does it converge to the same stable value for all starting states?
Will the value functions change if the starting players are changed? (human vs RL to RL vs human)

NOTE: I will be enumerating all states since there are approximately 20000 states which I believe is not a big number and thus convergence should not be a problem.

My understanding - from comments on the question - is that you are looking to train a Reinforcement Learning agent on the game of Tic Tac Toe (perhaps just in theory), where the agent should learn to play against a "human" opponent. In practice you may want a model of a human opponent.

In this case, the RL agent will be presented with a board state, it will take an action (to put its mark on an empty place in the grid) and either:

Win immediately, receiving a positive reward. It is common to use +1 in a game like Tic Tac Toe, so I will assume that later.
Lose on opponent's turn, receiving a negative reward (assumed -1 later in the answer), as opponent makes a move that causes it to win. This is effectively "immediately" in terms of time steps, the agent does not get to act afterwards.
Receive zero reward, and a new board state that includes the opponent's move
Receive zero reward, and the game ends in a draw

In all cases, the opponent is considered part of the environment. That makes the opponent behaviour critical to the value function and choice of optimal play. Training versus different opponents can result in very different state values.

For training to be stable, the opponent should behave with the same probability of action choices for each interim state that it observes. That includes it behaving deterministically, even optimally, purely randomly or anything in-between provided the probability distribution is fixed.

With the above context, it is possible to give sound answers to your questions:

  In Tic Tac Toe what is the effect of starting state in the State and Action Value function?

Each state of the board, or each state/action pair if you want to track action values, should converge to a value, depending on the agent's estimated, expected result at the end of the episode. As there is only a single reward possible at the end of each game, this will vary between -1 and +1. If either the agent or the opponent can make mistakes at random, then non-zero values between -1 and +1 are possible.

  Does it converge to the same stable value for all starting states?

That depends on behaviour of the opponent. In the scenario you are working with, the agent may not learn to play optimally in general, instead it will learn to gain optimal results against the supplied opponent. If the opponent can make mistakes, then moves which take advantage of that fact will have higher values.

Without a detailed description of the opponent it is not possible to make many statements about the actual state values and action values.

Against a perfect opponent, with the RL going second, it should converge to state values which are all zero and action values which are all zero or -1 for moves which would be mistakes.

Against a completely random opponent, I would have to run it to be sure, but I would expect state values to have 3 different values, all slightly positive, depending if opponent chose middle, edge or corner cases - each of these would have slightly different chance of leading to a win for the agent going forward.

  Will the value functions change if the starting players are changed?

Due to the turn-based nature of the game, all the states observed would be different depending on who was the first player. When the agent goes first it will get to score the empty grid and action values of any position it would like to make a first mark in - it gets to see state after turn 0 on time step 1, after turn 2 on time step 2, after turn 4 on time step 3 etc. When the agent goes second it will get to see and evaluate the outpt of other turns - turn 1 at t=1, turn 3 at t=2, turn 5 at t=3 etc.

That means the sets of states and state/action pairs for each case (RL first or RL second) are disjoint, and you cannot ask if one agent has the same value for a specific state as the other agent - it simply won't know about the other agent's values.

If you train a single agent, sometimes starting first, sometimes starting second, the two sets of values never interact with each other directly - in an enumerated table, as per the question, this is not at all, but if an agent uses function approximation such as neural networks, then they can affect each other.


Combining machine learning with languages like ProLog? There are mainly two different areas of AI at the moment. There is the "learning from experience" based approach of neural networks. And there is the "higher logical reasoning" approach, with languages like LISP and PROLOG.

Has there been much overlap between these (I can't find much!)?

As a simple example, one could express some games in PROLOG and then use neural networks to try to play the game.

As a more complicated example, one would perhaps have a set of PROLOG rules which could be combined in various ways, and a neural network to evaluate the usefulness of the rules (by simulation). Or even create new PROLOG rules.
(Neural networks have been used for language generation of a sort, so why not the generation of PROLOG rules, which could then be evaluated for usefulness by another neural network?)

As another example, a machine with PROLOG rules might be able to use a neural network to be able to encode these rules into some language that could be in turn decoded by another machine. And so express instructions to another machine.

I think, such a combined system that could use PROLOG rules, combine them, generate new ones, and evaluate them, could be highly intelligent. As it would have access to higher-order logic. And have some similarity to "thinking".

In reference to your exact question there is published research that attempts to bring these two areas together.

“HolStep: A Machine Learning Dataset for Higher-order Logic Theorem Proving” by Cezary Kaliszyk, François Chollet, Christian Szegedy (2017)
https://arxiv.org/abs/1703.00426

That group also has other published work related to the subject.

Regardless of their results they list several areas of logical systems that are highly suited to machine learning methods:

• Predicting whether a statement is useful in the proof of a given conjecture

• Predicting the dependencies of a proof statement (premise selection)

• Predicting whether a statement is an important one (human named)

• Predicting which conjecture a particular intermediate statement originates from

• Predicting the name given to a statement

• Generating intermediate statements useful in the proof of a given conjecture

• Generating the conjecture the current proof will lead to

It’s tough to know whether or not you can combine Higher Order Logic and Machine Learning in an effective way without needing to create a general AI. This is equivalent to wondering if an effective merging of the two areas is an AI-complete / AI-hard problem.

There are active attempts at general AI by researchers such as Ben Goertzel (many others as well but just to give a popular name for googling). Research into general AI would give you an idea of whether or not other pieces of the puzzle are needed in order to create something “highly intelligent”.


Data scan not making sense for coco dataset I am doing a simple scan to see how dataset size affects training. Basically, I took 10% of the coco dataset and trained a yolov3 net (from scratch) to just look for people. Then I took 20% of the coco dataset and did the same thing.... all the way to 100%. What is strange is that all 9 nets are getting similar loss at the end (~7.5). I must be doing something wrong, right? I expected to see an exponential curve where loss started out high and assymptotically approached some value as the dataset increased to 100%. If it didn't approach a value (and still had a noticeable slope at 100%), then that meant more data could help my algorithm.

This is my .data file:
classes= 1
train = train-run-less.txt
valid = data/coco/5k.txt
names = data/humans.names
backup = backup

I am trying to train just one class (person) from the coco dataset. Something is not making sense, and in a sanity test, I discovered that the loss drops even if the training folder only contains 1 image (which doesnt even have people in it). I thought the way this worked was that it trained on the "train" images, then it tested the neural net on the "valid" images. How is it getting better at finding people in the "valid" images if it hasnt trained on a single one??

Basically I am trying to answer the question: "how much accuracy can I expect to gain as I increase the data?"

What is strange is that all 9 nets are getting similar loss at the end (~7.5). I must be doing something wrong, right?

Yes, Please check the annotations file. Did you remove all the other classes except Person? What's the mAP of all those models?

  "how much accuracy can I expect to gain as I increase the data?"

It's hard to tell but more the data(variation), better the model.


What would be the best approach to teach an AI to learn how to "sing" along a beat? I have heard and read about HyperGAN, LSTM and a few other techniques, but I have a hard time piecing the overall concept together.

End Goal

Being able to input an instrumental and get an output of how to sing to that instrumental.

My Dataset

I have extracted pitch points from thousands of actual acapellas from real songs.

My Theory

Feed the AI a pitch point PLUS say 19 thousand points of the original song instrumental.

Illustration

The red line (on top) is the pitch viewed vertically (lower pitch down, higher pitch up) of the voice sung by the singer over time viewed horizontally.

The bottom image is the song's frequency viewed vertically (lower freq down, higher freq up) viewed horizontally over time.

We take a point in time of the instrumental, say 0 minutes 30 seconds, and extract 19k points of the FFT spectrum vertically and call this a frame.

We also take the same point in time of the voice pitch, and also refer to this as a frame.

So now we have a frame which contains 20 thousand data points, one being the pitch of the voice, and the rest being the frequencies of the songs content.

QUESTION

What kind of model could be used to teach the AI the correlation of the voice and the instrumental?

And also, I have a hard time understand how, once the AI is trained, how could just an instrumental be fed to the AI to output pitch values of how one COULD sing along to the song.

Like, training we need to input 20 thousand values, but when we want the AI to sing for us using just an instrumental, would it not still expect voice pitch input?
At what layer would the instrumental be tapped into? At the outer most right layer?

EDIT

My mind has been working on this in the background throughout the day,
and I am wondering if instead of feeding 19k points of instrumental data each frame (which would be points from the frequency domain), one could just feed the instrumental frame points (which would be points from the time domain).

Maybe that would be better, but then maybe the AI would get less "resolution" to work with, but could be trained faster (less computing power needed).

Let's say the frequency domain is fed (higher resolution), the AI could potentially find correlations from low notes, mid notes and high notes, in any combination (more computing power needed).

You have to define a function between voice and a corresponding sound, but there is no a best way to do it, it depends on what kind od output you care, for example you could create an algorithm that receives music and siging as separate "files" and then that algorithm finds the relation between them now you can give that algorithm a new music only file and it will create the singing part with the function of the previous music-singing "system".
This is sing this music as the other one was sang.

At least that is the way i would do it.


What are the top artificial intelligence journals? What are the top artificial intelligence journals?

I am looking for general artificial intelligence research, not necessarily machine learning.

This link includes various journals for artificial intelligence applied to various domains.

Some of those are:

IEEE Transactions on Human-Machine Systems
Journal of the ACM
Knowledge-based systems
IEEE Transactions on Pattern Analysis and Machine Intelligence
Journal of Memory and Language.

There are many more. You can refer to any of those journals and explore the research done by AI enthusiasts and researchers.


What will happen when you place a fake speedsign on a highway? I was wondering what will happen when somebody places a fake speedsign, of 10 miles per hour on a high way. Will a autonomous car slow down? Is this a current issue of autonomous cars?

https://www.technologyreview.com/s/530276/hidden-obstacles-for-googles-self-driving-cars/

  Google’s cars can detect and respond to stop signs that aren't on its map, a feature that was introduced to deal with temporary signs used at construction sites. But in a complex situation like at an unmapped four-way stop the car might fall back to slow, extra cautious driving to avoid making a mistake.

It's highly probable they would slow down with current technology, as they can detect temporary signs and are designed to use slow speed in complex cases.

If it was a true temporary sign (e.g. road repair), how can it make the distinction? It probably would be worse to ignore a slow down sign than slow down at a fake one.

IMHO, the problem there is with the joke in the first place, as some humans might slow down too.


Find object location (x, y) in an image I am generating images that consist of points where the object's location is where the most overlap of points occurs.
In this example, the object location is (25, 51). I am trying to train a model to just finds the location, I don't care about classification. Additionally, the shape of the overlapping points where the object is located never changes and will always be that shape.

What is a good model for this objective?
Many of the potential models I've been looking at (CNN, YOLO, and RCNN) are more concerned with classification than location. Should I search the image for the overlapping dots, create a bound box around them, then retrieve the boxes coordinates?

Thanks :)

What is a good model for this objective?

I will try to give another perspective: Solve it without machine learning model

Your problem is try to find the most overlapping point. If the image above is image that you used in your case, you can solve it directly by applying some computer vision algorithms.

Try to create some binary image based on the color of the dots. if you are not sure with the available colors on your image, you can list of pixel colors that not black and white uniquely at first. So if there are four colors you need to generate four different binary image. Create a simple condition or a complex one, for example:

if pixel[i,j]=red then
    pixel[i,j]=white
else
    pixel[i,j]=black

Get its location by searching "the white" over your image or use blob detection method (it'll be a little bit tricky if the actual image always have different axis scale). You can save it as a list of coordinate of each color.
What happen if you can't see the dot fully because it's overlapping with another dot? Find the pattern. In your image, dots appear with certain pattern. If you can find two consecutive dots horizontally and vertically, you can predict the position of all your dots.
Find the most overlapping position from your list.

Pros

The result may more accurate than using machine learning model
Faster, you don't need to train it first

Cons

Finding dot's location in image with different axis will be difficult, but it's still solvable
It'll be difficult to predict the pattern if many dots are missing because overlapped by other dots


Use AI to interpret XML? My question is more about "is it possible?" and "is this the right approach?" So let me explain what my idea is:

I think about a system which gets XML documents in various structures but with essentially the same data structure in it. For the example, lets assume each document contains data about one or more persons. So the AI would recognize a name. Somewhere else in the document there is the post address of our fictional person. The AI should now "see" the address and conclude, it belongs to our person. Anywhere else, there is a phone number in the document. Again, our AI should see the connection between our person and this phone number.

This wouldn't be a job for an AI if there wasn't a catch. If the task was merely to find and map strings like addresses and phone numbers, we could simply use a regex to match our "target strings". The catch in this scenario would be this: the XML document might contain other data, which does not belong to our person but is a valid phone number for example and thus will match an regex.

So the big question is: Would it be possible for an AI to learn this and if yes, with which framework would someone create such an AI?

Sample XML document:

<?xml version="1.0" encoding="utf-8" ?>
<document>
    <data>
        <foo>
            <bar>
                <person>
                    <name>John Doe</name>
                </person>
            </bar>
            <address>
                <street>Main street 1</street>
                <city>1111 Twilight town</city>
                <country>sample country</country>
            </address>
            <phone>+123 123 123</phone>
        </foo>
        <foo>
            <bar>
                <person>
                    <name>Jane Doe</name>
                </person>
            </bar>
            <address>
                <street>Broadway 42</street>
                <city>4521 Traverse town</city>
                <country>sample country</country>
            </address>
            <phone>+123 412123</phone>
        </foo>
    </data>
    <creator>
        <!-- Note: While this looks like a valid person, -->
        <!-- this data should not be matched by the AI -->
        <name>Sam Smith</name>
        <office>
            <street>Seaside road 5</street>
            <city>4521 Traverse town</city>
            <country>sample country</country>
        </office>
        <phone>+123 555 555</phone>
    </creator>
</document>

XML, HTML and less formal languages all respond quite nicely to being transformed or interrogated within a graph framework. XML and HTML are particularly useful in that they conform strictly to a tree-structure. That means that any good data components can be measured in terms of tree-distance to any other "good" data components.

If you extract your regex-friendly terms and keep track of where within their tree they are found, you may be able to cast those values into a general document-space vector (it might only need to be one-dimensional), allowing you to identify clusters of "good" vs anomalous sections of "bad" data based on a simple distance metric, or anomaly detection algorithm - say an isolation-forest that runs on information density for example.

This depends on your data, and how much of it you can find, ideally already tagged up containing good vs bad.

If you're looking to scrape reliable address-contents, then yes, you're likely to score hits on names, addresses, postcodes and phone-numbers all appearing as tightly connected clustered groups, all within one or two nodes-distance from one another.

Meanwhile, an annotation containing a phone number lodged somewhere else is less likely to be a match.

Different documents will have different threshold densities, and differing anomaly to conformant ratios, so you'd have a task on your hands to figure out some way to automatically tune your parameters on any given document set.

In the past, I've tried doing this against html by flattening all the content into a single string of text and a similar approach yielded half-decent results, but if you're looking at XML, it's fair to expect the structure to yield more information.


Is Windows a bad choice for DRL? I'm looking into using PPO implementations like OpenAi's SpinningUp and Baselines. However, I fear that these implementations require packages which are not available for Windows. So I'm wondering if in general Linux should be used for working with DRL algorithms. Also, I'm not planing on using OpenAI's gym environments. Environments, created with the ml-agents toolkit from Unity and its gym wrapper are going to be used.

What is your opinion on this? Should I rather work on Linux or Windows? And what do you think about alternatives such as: VM on Windows with Ubuntu or Subsystem on Windows with Ubuntu (WSL)?

TLDR Use whatever environment you are most comfortable with.

Today the choice of operating system is of much less importance than few years ago. Generally speaking, you will have more application choices under Linux, however, some of these choices will be less than spectacular :-) Windows works better with some hardware. Hence, under Linux, it is a very good idea to check the forums before buying or installing.

I have played with OpenAI gym under Linux for some time and it worked as advertised.

Early versions of many important pieces of deep learning software get released for Linux only. For instance, PyTorch and Tensorflow were available for Linux at least one year before the release of Windows ports.


Tensforflow schedule - does not change boundaries I'm trying to manipulate the learning rate with tf PiecewiseConstantDecay.
I can easily check if the algorithm switches learning rate values, because one rate is extremely low 1e-20 !!
However, NO setting of the "boundaries" causes the algorithm to switch learning rate... What am I doing wrong?
step = tf.Variable(0, trainable=False)
boundaries = [100]
values = [1e2, 1e-20]

schedule = tf.optimizers.schedules.PiecewiseConstantDecay(boundaries, values)
lr = 1e-4 * schedule(step)

optimizer = tf.keras.optimizers.Adam(learning_rate=lr)
model.compile(optimizer=optimizer, loss='mean_squared_error')

history = model.fit(x = x_train, y = y_train, validation_data=(x_val, y_val), epochs=100,batch_size=32)

Would love your input.

After 100 steps learning rate will switch from 1e-2 to 1e-24.
What exactly is the problem here? Are you confusing steps with epochs?


What is a "batch" in batch normalization? I'm working on an example of CNN with the MNIST hand-written numbers dataset. Currently I've got convolution -> pool -> dense -> dense, and for the optimiser I'm using Mini-Batch Gradient Descent with a batch size of 32.

Now this concept of batch normalization is being introduced. We are supposed to take a "batch" after or before a layer, and normalize it by subtracting its mean, and dividing by its standard deviation.

So what is a "batch"? If I feed a sample into a 32 kernel conv layer, I get 32 feature maps.

Is each feature map a "batch"?
Are the 32 feature maps the "batch"?

Or, if I'm doing Mini-Batch Gradient Descent with a batch size of 64,

Are 64 sets of 32 feature maps the "batch"? So in other words, the batch from Mini-Batch Gradient Descent, is the same as the "batch" from batch-optimization?

Or is a "batch" something else that I've missed?

The "batch" is same as in mini-batch gradient descent. The mean in batch-norm here would be the average of each feature map in your batch (in your case either 32 or 64 depending on which you use)

generally batch is used quite consistently in ML right now, where it refers to the inputs you send in together for forward/backward pass.


How do we get the true value in the prediction objective in reinforcement learning? In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto define the prediction objective ($\overline{VE}$) as follows (page 199):
$$\overline{VE}\doteq\sum_{s\epsilon S} \mu(s)[v_{\pi}(s)-\hat{v}(s,w)]^2$$
Where $v_{\pi}(s)$ is the true value of $s$ and $\hat{v}(s,w)$ is the approximation of it. Furthermore it is stated that this is "often used in plots".

How do I get the true value $v_{\pi}(s)$? And if there is a way to obtain the value, why would I need to approximate it?

The true value $v_{\pi}(s)$ is a conceptual target for the $\overline{VE}$ in the book. You often do not know it in real problems. However, it is still used in two main ways in the book:

Theoretically for analysis of different aprpoximation schemes, which can be shown to converge to minimise the $\overline{VE}$ objective, or a related one.
In toy problems when exploring the nature of approximation in Reinforcement Learning (RL), it is possible to use tabular methods guranateed to get close to zero error, and then compare them to approximate methods. There are several plots of this type in the book.

The book shows derivation of gradient descent methods that start with minimising $\overline{VE}$ as an objective, and that use samples of $v_{\pi}(s)$ such as Monte Carlo return $G_t$ or the TD target) in place of the unknown $v_{\pi}(s)$ in the loss function. These also rely on the fact that the sample distribution will be weighted by $\mu(s)$ if they are taken naturally from the environment whilst the agent is following the policy $\pi$ so that $\mu(s)$ also does not need to be explicitly known.

Outside of toy problems deliberately constructed to demonstrate that this theory is correct, you will not know $v_{\pi}(s)$ or be able to calculate $\overline{VE}$. However, you will know from the theory that if you follow the update rules derived in the book for approximate gradient descent methods, that the process should find a local minimum for $\overline{VE}$, for whatever state approximation scheme you have chosen to use.

Usually you cannot even approximate $\overline{VE}$ from raw data, as the variance in returns will add noise to the signal, and there is no way to separate variance in returns from error in approximation in the general case. However there are a couple of scenarios that do lend themselves to measuring this objective, provided you already have your estimate $\hat{v}(s,w)$ and the policy remains fixed throughout:

Simple, fast (perhaps simulated), environments which can be solved to arbitrary accuracy using tabular methods. In this case, you first calculate $v_{\pi}(s)$ using a non-approximate method, then sample many approximations by running the environment using policy $\pi$ and treating that as your data set.
Fully deterministic environments where $\pi$ is also deterministic. These have a variance of $0$ for Monte Carlo returns, so each observed return from any given state is already the true value of $v_{\pi}(s)$. Again you can just run the environment many times to get your data set to calculate  $v_{\pi}(s)$ and $\hat{v}(s,w)$ for the observed states in the correct frequencies, and thus have data to calculate $\overline{VE}$.


Classification with different approaches A few days ago I asked the question, if a NN with linear activation function can produce a function concatenated of linear functions, what actually is impossible (Can a NN with linear activation functions produce a connection of linear functions?).

Now I have here some classification examples, but I really cannot perfectly decide, which one is based on which approach:

1 -> C The perceptron does not look for the maximum separation margin.

2 -> E Neural network with linear activation function

3 -> A Linear SVM, because of the maximum separation margin.

4 -> B Because of the hyperbolic shape of the hyperplane.

5 -> D? Logisitc regression? I tought it can only linear separate?

6 -> F I guess the NN with tanh activation function, because of the no very smooth shape, which comes from the too small hidden layer size.

I actually don't get how the logistic regression classifier should be able to produce a hyperplane like in 5? What did I classify wrong here?

As far as I understand your question, you are trying to match the learning algorithm with their results.

Firstly, I don't think the answer mapping you have provided is correct.

First, we see the options and find out which one linearly seperates the data:
We have 3 candidates for the job:

A. Linear SVM

C. Perceptron

D. Logistic Regression

In the datasets only 1, 2 and 3 are linearly seperated. Since, we know linear SVM will search for decision boundary with maximum separation between 2 linearly separable data sets hence A goes with 3. Pereceptron is basically the linear version of logistic regression. Now, LR is an exponential algorithm hence the error does not appear as large as it should. Say that for a particular input vector x the y = 1.Say the hypothesis (x * transpose(theta)) your algorithm generated is 5. For perceptron error will be 5 - 1 = 4 whereas for LR it will be 1 / (1 + e^(-5)) - 1 = 0.9932. If hypothesis generates 6 error will be 5 for perceptron and about 0.9974 for LR. So we see even for very great increase in errors the cost for LR will not increase significantly (inherent property of exponentials) whereas it increases linearly for perceptron.

Hence, 1 goes with C since it tries to minimize the error, whereas 2 will go with D since even though there is a greater cost it gets scaled due to exponential function.

And now I think 5 will go with F since the curve has a smoothness associated with it, whereas due to Re-Lu's the curve in 6 will go to E (due to an associated sharpness) and finally B goes to 4 as it has a kernel of order 2 and x^2/a^2 + y^2/b^2 +xy = c is the equation of a hyperbola of order 2.

Hence final mapping:

A - 3
B - 4
C - 1
D - 2
E - 6
F - 5

I may have missed few other mathematical subtleties, and I would be grateful if someone points them out.


Action recognition using video stream data Recently, I am working on an action recognition project where my input data is from the video stream. I read some of the concepts like ConvLstm, Convolutional Lstm, etc. I am looking for someone who already those kinds of staff already and can share his work with me that will be a really good help for me?

If you want to build a stream recognition pipeline than you can use OpenCV, Kafka, and Spark together.how to build pipeline of video stream

and for Action recognition using ConvLstm. This link will help you.action recognition using Convlstm


Why is batch gradient descent performing worse than stochastic and minibatch gradient descent? I have implemented a neural network from scratch (only using numpy) and I am having problems understanding why the results are so different between stochastic/minibatch gradient descent and batch gradient descent:

The training data is a collection of point coordinates (x,y).
The labels are 0s or 1s (below or above the parabola).

As a test, I am doing a classification task.
My objective is to make the NN learn which points are above the parabola (yellow) and which points are below the parabola (purple).

Here is the link to the notebook: https://github.com/Pign4/ScratchML/blob/master/Neural%20Network.ipynb

Why is the batch gradient descent performing so poorly with respect
to the other two methods?
Is it a bug? But how can it be since the code is almost identical to the
minibatch gradient descent?
I am using the same (randomly chosen with try and error) hyperparameters for all three
neural networks. Does batch gradient descent need a more accurate
technique to find the correct hyperparameters? If yes, why so?

Assuming the problem at hand is a classification (Above or Below parabola), this is probably because of the nature of Batch gradient descent. Since the gradient is being calculated on the whole batch, it tends to work well on only convex loss functions.

The reason for why batch gradient descent is not working too great probably is because of the high number of minimas in the error manifold, ending up learning nothing relevant. You can change the loss function and observe the change in results, they might not be great (Batch GD usually isn't) but you'll be able to see differences.

You can check this out for more on the differences between the three. Hope this helped!


How to know which kind of adverb in NLP Parts of Speech (POS) tagging? There are 4 kinds of adverbs :

Adverbs of Manner. For example, slowly, quietly
Adverbs of Place. For example, there, far
Adverbs of Frequency. For example, everyday, often
Adverbs of Time. For example, now, first, early

nltk, spacy and textblob only tag a token as an adverb without specifying which kind it is.

Are there any libraries which tag including the type of adverb?

Generally POS taggers only use broad categories, especially if they work statistically: the more fine-grained the tagset is, the more training data you need. And unless there are differences in the distribution, the accuracy of the tagger would not be improved by adding more categories, eg

  I ran [there/fast/often/early].

all work -- so nothing is gained by having subcategories of adverbs.

I suggest the easiest solution for this would be to have a list of adverbs by category, and simply look them up after the tagging. Just pick all words tagged as adverbs and check if they are in your list. Unless there are ambiguities (ie some adverbs can belong to multiple categories) it should be very straight forward, and would also work with any tagger/tagset that marks adverbials.

You might have to consult a grammar book to get a list of adverbial classes, but that again should be easy to do.


What are examples of optimization problems that can be solved using genetic algorithms? I'm trying to learn how genetic algorithms can solve optimization problems. I have already learned how genetic algorithms can solve the knapsack, TSP and set cover problems. I'm looking for some other similar optimization problems, but I have not found any.
Would you please mention some other famous optimization problems that can be solved by using genetic algorithms?

There are numerous problems that can be solved with genetic algorithms or, more generally, with evolutionary algorithms (which includes also genetic programming and evolutionary strategies), even though they may not necessarily be the most efficient approach.
Here are a few examples.

Evolution of the topology of neural networks. This is called neuroevolution.
Automatic test case generation (in particular, for self-driving cars). AsFault is one specific example.
Design of novel quantum computing algorithms. Specifically, genetic programming has been used to solve this problem (see this reference for more details).
As an alternative to reinforcement learning algorithms to solve RL problems. Specifically, evolution strategies have been successfully used in this case (see this).

There is a Wikipedia article that lists many other applications of genetic algorithms: List of genetic algorithm applications.


A2C Critic Loss Interpretation I'm working on an Advantage A2C implementation, and I just finished creating the value network $\hat{V_{\phi}}$. I train this network with the standard MSE loss of discounted rewards-to-go:$$\|\hat{V_{\phi}}(s_{t'}) - \sum_{t=t'}^{T}\gamma^{t-t'} r(s_t, a_t)\|^2$$

I would like to be able to evaluate and assess the performance and ability of the value network as I train, especially to see how that relates and interacts with the changes and improvements in the policy, however I'm not sure how to do this.

My first instinct was to track the loss after each batch of experiences, but this doesn't work. As the policy improves, episodes last longer, and the value loss increases. I understand why this happens, as it is harder to predict the rewards-to-go when the length of the future is more undetermined.

To fix this, I tried dividing the loss that I'm computing by the total number of steps in the batch of episodes. However, the loss is still increasing as the policy improves (as the episodes get longer). Why is this still happening? Then, is there anything I can do to get a better assessment of the quality of the value network?

Unfortunately no, the way to go is track the total reward and see if it's increasing and converging eventually. Value loss isn't a useful metric as the loss can be 0 when the value network always predicts 0 and the agent doesn't collect any reward, meaning very poor behavior.


What are the techniques for detecting and preventing overfitting? I'm worrying that my neural network has become too complex. I don't want to end up with half of the neural network doing nothing but just take up space and resources.
So, what are the techniques for detecting and preventing overfitting, to avoid such problems?

Usually you keep track of training loss and validation loss and apply proper regularization technique (such as L1, L2, dropout, DropConnect, etc.).

The more interesting technique is to observe your validation loss with respect to the number of parameters in the network (often controlled by the number of layers/feature maps). If the validation starts dropping with raising your model's complexity, then your optimization might be bad or simply the model remembers all of the training samples and overfits badly.


Rollout algorithm like Monte Carlo search suggest model based reinforcement learning? From what I understand, Monte Carlo Tree Search Algorithm is a solution algorithm for model free reinforcement learning (RL).

Model free RL means agent doesnt know the transition and reward model. Thus for it to know which next state it will observe and next reward it will get is for the agent to actually perform an action.

my question is:
if that is the case, then how come the agent knows which state it will observe during the rollout, since rollout is just a simulation, and the agent never actually perform that action ? (it never really interact with the environment: e.g it never really move the piece in a Go game during rollout or look ahead, thus cannot observed anything).

It can only assume observing anything when not actually interacting with environment (during simulation) if it knows the transition model as I understand it. The same arguments goes for the rewards during rollout/ simulation.

in this case, doesnt rollout in Monte Carlo Tree Search algorithm suggests that the agent knows the transition model and reward model and thus a solution algorithm for model based reinforcement learning and not model free reinforcement learning ?

** it makes sense in Alphago, since the agent is trained to estimate what it would observed. but MCTS (without policy and value netwrok) method assumes that agent knows what it would observed even though no additional training is included.

From what I understand, Monte Carlo Tree Search Algorithm is a solution algorithm for model free reinforcement learning (RL).

Monte Carlo Tree Search is a planning algorithm. It can be considered part of RL, in a similar way to e.g. Dyna-Q.

As a planning algorithm MCTS does need access to a model of the environment. Specifically it requires a sampling model, i.e. a model that can accept a state and action, then return a single next state and reward with the same probabilities as the target system. The alternative model, used by other RL techniques such as Value Iteration, is a distribution model which provides the full distribution of probabilities for rewards and next states.

  if that is the case, then how come the agent knows which state it will observe during the rollout, since rollout is just a simulation, and the agent never actually perform that action ?

It is not the case. The agent knows what it will observe during the simulation because the simulation is a sampling model.

  In this case, doesnt rollout in Monte Carlo Tree Search algorithm suggests that the agent knows the transition model and reward model and thus a solution algorithm for model based reinforcement learning and not model free reinforcement learning ?

Yes. This is how most planning algorithms work.

The simulation can be driven purely by sampling from previous experience, which is how Dyna-Q works*. I would still consider that a model-based approach, and its success depends a lot on how well such a model can be learned. In many cases, variance due to errors in the learned model adversely affects learning. So MCTS works best in environments that can be accurately sampled, because they are strongly rules-driven. For example, board games.

* Functionally DynaQ is almost identical to experience replay. So much so, that whether you consider it a planning algorithm or experience replay added to basic Q learning is more a matter of how you present the design of the learning agent - e.g. perhaps a designer wants to focus on the model-learning side more, so has code that explicitly represents the learned model.


Do deep learning algorithms represent ensemble-based methods? According to the Wikipedia article on deep learning:

Deep learning is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and non-linear transformations.

Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition and bioinformatics where they have been shown to produce state-of-the-art results on various tasks.

Can deep neural networks or convolutional deep neural networks be viewed as ensemble-based method of machine learning, or are they different approaches?

You should think of them as different approaches. A deep neural net is a single independent model, whereas ensemble models are ensembles of many independent models.

The primary connection between the two is dropout, a particular method of training deep neural nets that's inspired by ensemble methods.


Approaches to an algorithm for crossing a road I want to write an algorithm which indicates to a robot the first point in time when it is reasonably safe to cross a road.  Assume that the robot's goal is to travel to a location that requires a road crossing and that the robot is ready to cross.

Simple algorithms and decision making will probably not suffice.  What features and capabilities must the algorithm have to provide crossing safety?  What existing AI methods might be useful to consider for this endeavor?

For the first iteration, we can assume the traffic pattern is normal in that no vehicles are driving over the curb and there are no high speed chases or other safety related abnormalities.

Any AI algorithm depends on the environment, and available actuators and sensors. In our case, the environment is a road, street, etc. The primary actuator includes wheels (or legs) of the robot. Sensors include a camera, sonar system, etc.
A simple Model-based reflex algorithm can work in your case:

function MODEL-BASED-REFLEX-AGENT(percept) returns an action
    persistent: state, agents' current conception of the world state
                model, description of how hte next state depends on current state and action
                rules, a set of condition-action rules
                action, most recent action, initially none
    state <-- UPDATE-STATE(state, action, percept, model)
    rule <-- RULE-MATCH(state, rules)
    action <-- rule.ACTION
    return action

Most of the terms and functions are self-explanatory and I will try to explain the important points. This implementation helps the robot to keep track of the external environment by maintaining internal state that depends on the percept history. Updating the internal state requires how the environment works without our agent in it. For e.g., Cars stop at red light and start moving on the green signal. Another thing that is required is how the actions of our agent will affect the world. Since your robot is only trying to cross the road there are not many cases here. Simple things include, stopping motors will stop the bot and starting them will move it forward.
The algorithm above shows how the current percept combined with the old internal state result in generating the updated description of the world, based on agent's model of how the world works. Thus, agent's model of how the world works is the most important part. UPDATE-STATE is responsible for creating new internal state description. The actual implementation of this will depend on the environment and technology that is being used.
The above algorithm has been taken from the book, Artificial Intelligence: A Modern Approach, for agents working in the partially observable environment. Implementation of many functions also depends on how complex you want your robot to be. For e.g., we haven't considered the case of looking up while crossing the road. This is possible that something might be falling from the sky, highly improbable, but possible. The list like this never stops in real life environments.


Why does Google Translate produce two different translations for the same Chinese text? I don't understand why Google Translate translates the same text in different ways.
Here is the Wikipedia page of the 1973 film "Enter the Dragon". You can see that its traditional Chinese title is: 龍爭虎鬥. Google translates this as "Dragons fight".
Then, if we go to Chinese Wikipedia page of this film, and search for 龍爭虎鬥 using Ctrl-F, it will be found on several places:

But if we try to copy the hyperlink of Chinese page into Google translate, it will be the word "tiger" from somewhere:

Even more, if we try to translate Chinese page into English using build-in Chrome translate, it will be sometimes translated as "Enter the Dragon", in English manner:

Why it gives different translations for the same Chinese text here?

It's not quite clear what you are asking. So I'll answer in separate parts.

  Why is the translation different from the official title?

It could be simply because machine translation is not perfect, or our human translator took some creative liberties when translating. In this case it seems to be both.

Note that 龍爭虎鬥 properly translated doesn't mean either "Dragons fight" or "Enter the dragon". Literally translated, it means "dragon compete tiger fight". It belongs a family of well-formed idioms called "Chengyu", which describes a situation where there is fierce fighting or competition.

So you can see that neither translations fit.

  Why does Google Translate give me different translations on the same phrase?

Context matters when we read! So translating a phrase in isolation doesn't guarantee that the same phrase has the same meaning/translation in all other parts of the text.

For example, green is a color, but the word "green" can also be used as like: "Alice is green with envy" or "Bob has green thumbs", of which neither instances of the word "green" refers to the color.

Considering the technical side of things, Google translate probably uses some kind of RNN in its pipeline. RNN's are influenced by past states, meaning that what it outputs now as a function of what it reads in is dependent on the RNN's past state. Which is similar to the issue addressed above.


In Batch Normalisation, are $\hat{\mu}$, $\hat{\sigma}$ the mean and stdev of the original mini-batch or of the input into the current layer? In Batch Normalisation, are the sample mean and standard deviation we normalise by the mean/sd of the original data put into the network, or of the inputs in the layer we are currently BN'ing over?

For instance, suppose I have a mini-batch size of 2 which contains $\textbf{x}_1, \textbf{x}_2$. Suppose now we are at the $k$th layer and the outputs from the previous layer are $\tilde{\textbf{x}}_1,\tilde{\textbf{x}}_2$. When we perform batch norm at this layer would be subtract the sample mean of $\textbf{x}_1, \textbf{x}_2$ or of $\tilde{\textbf{x}}_1,\tilde{\textbf{x}}_2$?

My intuition tells me that it must be the mean,sd of $\tilde{\textbf{x}}_1,\tilde{\textbf{x}}_2$ otherwise I don't think it would be normalised to have 0 mean and sd of 1.

Your intuition is correct. We will be normalizing the inputs of the layer under consideration (just right before applying the activation function).

So, if this layer receives an input $\mathrm{x}=\left(x^{(1)} \ldots x^{(d)}\right)$, the formula for normalizing the $k^{th}$ dimension of $\mathrm{x}$ would look as follows:
$$\widehat{x}^{(k)}=\frac{x^{(k)}-\mathrm{E}\left[x^{(k)}\right]}{\sqrt{\operatorname{Var}\left[x^{(k)}\right]}}$$

Note that in practice a constant $\epsilon$ is also added under the square root in the denominator to ensure stability.

Source: The original Batch Normalization paper (Section 3).

Andrew Ng's video on this topic might also be useful for illustration.


Do convolutional neural networks also have recurrent connections? I asked my self this simple question while reading "Comment Abuse Classification with Deep Learning" by Chu and Jue. Indeed, they say at the end of the that

  It is clear that RNNs, specifically LSTMs, and CNNs are state-of-the-art  architectures for sentiment analysis

To my mind CNNs were only neurons arranged so that they correspond to overlapping regions when paving the input field. It wasn't that recurrent at all.

You are right. I think you are just misinterpreting the part of the sentence ('specifically LSTMs'). LSTMs are an example of a popular type of RNN.
RNNs and CNNs are different architectures but they can be used together.

Here is another sentence with the same structure:

  It is clear than dogs, specifically corgis, and cats are very common in online memes.


Can the optimal value of discount factor in Deep Reinforcement Learning be between 0.2 to 0.8? I'm now reading a book titled as Hands-On Reinforcement Learning with Python, and the author explains the discount factor that is used in Reinforcement Learing to discount the future reward, with the following:

A discount factor of 0 will never learn considering only the immediate rewards; similarly, a discount factor of 1 will learn forever looking for the future reward, which may lead to infinity. So the optimal value of the discount factor lies between 0.2 to 0.8.

The author seems to be not going to explain further about the figure, but all the tutorials and explanations I have ever read write the optimal (or at least widely used) discount factor between 0.9 an 0.99. This is the first time I have seen such a low-figure discount factor.
All the other explanations the author makes regarding the discount factor are the same as I have read so far.
Is the author correct here or does it depend on cases? If it is, then what kind of problems and/or situations should I set the discount factor as low as such figure at?

EDIT
I just found the following answer at Quora:

Of course. A discount factor of 0 will never learn, meanwhile a factor near of 1 will only consider the last learning. A factor equal or greater than 1 will cause the not convergence of the algorithm. Values usually used are [0.2, 0.8]
EDIT: That was the learning factor. The discount factor only affect how you use the reward. For a better explanation:
State-Action-Reward-State-Action - Wikipedia
See influences of variables .

I don't know what is written in the question as it in not visible in Quora, but it seems that the 0.2 to 0.8 figure is used for learning factor, not discount factor. Maybe the author is confused with it...? I'm not sure what the learning factor is, though.

The discount factor is not something you should be optimising. It is typically part of the problem statement.

For practical purposes, you may set it below 1.0 for continuous problems when in fact you care about best long-term reward. Another option to avoid infinities on continuous problems is to re-formulate the problem as optimising average reward. A high discount factor of e.g. 0.99 or 0.999 should produce a similar policy as one based on average reward.

  Is the author correct here or does it depend on cases?

The author appears to be either completely wrong, or just poor at explaining themselves on this part.

  If it is, then what kind of problems and/or situations should I set the discount factor as low as such figure at?

A low discount factor is for when you care much more about immediate rewards. You set it that low when that is the case. You decide what you care about when you set the learning problem. The value of the discount factor is part of the setup that decides what the optimal policy is. You never set it low "to help with optimising" because changing the value could change the optimal policy.


Training a convolutional network to recognize object location I am beginning an image analysis project to recognize images with a particular object centered on the image. If the object is at the center, I give the image a positive label, and if it is anywhere else, or simply not in the image, I give the image a negative label. The object, itself, has a complex pattern, such that statistical methods and basic image processing techniques are not able to detect it. The human eye, however, has no trouble detecting this object. Therefore, I am opting to develop a convolutional network that can parse the complexity of this pattern. The only issue, however, is that convolutional networks are inherently designed to be spatially invariant. Therefore, is it even possible to train the network to focus on the importance of the object being at the center simply by feeding the network many negative examples containing the object anywhere else but the center? Furthermore, is there perhaps a better or more direct way to go about incorporating this spatial aspect into the network's functionality?

To use a convolutional net that isn't spatially invariant, you can make the convolution matrix of size equal to your input image size. Afterwards, just use any desired number of fully connected layers and your network should be able to learn your dataset.


Are the shortcomings of neural networks diminishing? Having worked with neural networks for about half a year, I have experienced first-hand what are often claimed as their main disadvantages, i.e. overfitting and getting stuck in local minima. However, through hyperparameter optimization and some newly invented approaches, these have been overcome for my scenarios. From my own experiments:

Dropout seems to be a very good regularization method.

Batch normalization eases training and keeps signal strength consistent across many layers.

Adadelta consistently reaches very good optima

I have experimented with scikit-learn's implementation of SVM alongside my experiments with neural networks, but I find the performance to be very poor in comparison, even after having done grid-searches for hyperparameters. I realize that there are countless other methods, and that SVM's can be considered a sub-class of NN's, but still.
So, to my question:
With all the newer methods researched for neural networks, have they slowly - or will they - become "superior" to other methods? Neural networks have their disadvantages, as do others, but with all the new methods, have these disadvantages been mitigated to a state of insignificance?
I realize that oftentimes "less is more" in terms of model complexity, but that too can be architected for neural networks. The idea of "no free lunch" forbids us to assume that one approach always will reign superior. It's just that my own experiments - along with countless papers on awesome performances from various NN's - indicate that there might be, at the least, a very cheap lunch.

Neural Networks have other short comings as well.

It takes much longer and far more resources to train a neural network than something like a random forest. So if you need speed of training or are resource constrained in anyway, you probably should not look at Neural Networks first. Evaluation of a trained deep NN can be much more expensive than competing techniques too.
The effort involved in learning how to architect and train a NN is still much higher than competing methods, like an SVM. People who are just starting out in Data Science should probably use other techniques to learn about the nuances of fitting data before getting involved in neural networks. And although simple NNs with only one or two hyperparameters are often available in
many data science libraries, they don't perform any better than other techniques so are just another ML black-box technique really.
While we have made a lot of progress in understanding how neural networks do their magic, they are still less accessible and dissectible than most competing methods. So while NNs might solve the problem, they might not give you as many insights as easily as other techniques do.

Looking forward to what other people have to say here.


How is weighted average computed in Deep Q networks I was going through the Sutton book and they said the update formula for Q learning comes from the weighted average of the returns
I.e
New estimate= old estimate +alpha*[returns- old estimate]
So by the law of large numbers this will converge to the optimal true q value
Now when we go to Deep Q networks,how exactly is the weighted average computed, all they simply did was try to reduce the error between the target and the estimate, and keep in mind this isn’t the true target, it’s just an unbiased estimate,since it’s an unbiased estimate how is the weighted average computed , which is the expectation?
Can someone help me out here??
Thanks in advance

Let's say $Q$ is the old estimate, $Q'$ the new estimate, and $R$ is the return.
We have
$$
Q' = Q + \alpha(R-Q)
$$
This can be rewritten as
$$
Q' = (1-\alpha)Q + \alpha R
$$
When $\alpha$ is a constant, this is an exponential weighted average of returns. If $n$ is the number of samples we get and $\alpha=1/n$ (so it decreases with each sample), we get
$$
Q' = \frac{n-1}{n}Q + \frac{1}{n}R
$$
This simply represents the average return. So, playing with $\alpha$ tunes the weighting of the estimate.


Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking action $a$ in the given state $s$) as the average of all rewards recorded for being in state $s$ and taking action $a$?
In many examples, I've seen the value of a state computed as the expected return computed as the cumulative sum of rewards multiplied by a discount factor:
$V^π(s)$ = $\mathbb{E}(R \mid s)$ (the value of state s, if we follow policy π is equal to the expected return given state s)
So, $V^π(s)$ = $\mathbb{E}(r_{t+1}+ γr_{t+2}+ (γ^2)_{t+3} + ...    \mid s) = {E}(∑γ^kr_{t+k+1}\mid s)$
as $R=r_{t+1}+ γr_{t+2}+ {γ^2}r_{t+3}, + ...   $
Would it not make more sense to compute the value of a state as the following:
$V^π(s)$ = $(r_{t+1} + γr_{t+2} + (γ^2)_{t+3}, + ...    \mid s)/k = {E}(∑γ^kr_{t+k+1}\mid s)/k $ where k is the number of elements in the sum, thus giving us the average reward for being in state s.
Reference for cumulative sum example: https://joshgreaves.com/reinforcement-learning/understanding-rl-the-bellman-equations/

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?

That is the definition of return.
In fact when applying a discount factor this should formally be called discounted return, and not simply "return". Usually the same symbol is used for both ($R$ in your case, $G$ in e.g. Sutton & Barto).
There are also other variations, such as truncated return (sum up to a given time horizon). They all share the feature that a return is a sum of reward values. You cannot really change that and keep the formal term "return", that's how it has been defined.
You can however define the value function to be something other than the expected return. Rather than looking for alternative definitions of return as your title suggests, you could be looking for alternative metrics to use as value functions.
You do go on to ask about computing "the value of a state" without mentioning the word "return", but it is not 100% clear whether you are aware that the way to resolve this is to not use return, but something else.

Would it not make more sense to compute the value of a state as the following:
$V^π(s)$ = $(r_{t+1} + γr_{t+2} + (γ^2)_{t+3}, + ...    \mid s)/k = {E}(∑γ^kr_{t+k+1}\mid s)/k $ where k is the number of elements in the sum, thus giving us the average reward for being in state s.

Your example would nearly always result in zero for long-running or non-episodic problems, as you are summing a decreasing geometric series possibly up to very large $k$, then dividing by the maximum $k$. Notation-wise you are also using $k$ to be an iterator and the maximum value of the same iterator, that would need fixing.
However, this is very close to a real value metric used in reinforcement learning, called the average reward setting.
The expected average reward value function for a non-episodic problem is typically given by
$$V^\pi(s) = \mathbb{E}[\lim_{h \to \infty}\frac{1}{h}\sum_{k=0}^{h}r_{t+k+1}|s_t = s]$$
Note there is no discount factor, it is not usually possible to combine a discount factor with the average reward setting.
Sutton & Barto point out in Reinforcement Learning: An Introduction chapter 10, section 10.4, that when using function approximation on continuing tasks, then a discount factor is not a useful part of the setting. Instead average reward is a more natural approach. It is also not so different, and quite easy to modify the Bellman equations and update rules. However, many DQN implementations still use discounted return to solve continuing tasks. That is because with high enough discount factor $\gamma$, e.g. $0.99$ or $0.999$, then the end result is likely to be the same optimal solution - the discount factor has moved from being part of the problem formulation to being a solution hyperparameter.


How can we prevent AGI from doing drugs? I recently read some introductions to AI alignment, AIXI and decision theory things.
As far as I understood, one of the main problems in AI alignment is how to define a utility function well, not causing something like the paperclip apocalypse.
Then a question comes to my mind that whatever the utility function would be, we need a computer to compute the utility and reward, so that there is no way to prevent AGI from seeking it to manipulate the utility function to always give the maximum reward.
Just like we humans know that we can give happiness to ourselves in chemical ways and some people actually do so.
Is there any way to prevent this from happening? Not just protecting the utility calculator physically from AGI (How can we sure it works forever?), but preventing AGI from thinking of it?

You make a lot of assumptions about AGI, namely that 'we need a computer to compute the utility and reward AGI'. It not clear to me that (1) we can achieve AGI, (2) AGI will be on a computer as we know it and (3) AGI will work with a utility / reward function as we know them.
One thing I am sure though is that ML is known for "cheating" (see for exemple). Avoiding such cheating is part of the building process. So, when you assume that we can achieve AGI, it means that you assume we can build an AGI that do not "cheat". Thus, the answer is mostly contained in your assumptions.
Whether we are able to build an AGI, what "cheating" we would have to overcome to do so and how we will be able to do so is mostly undetermined.


"Vocal captcha" for robots on the phone? With all the Google I/O stuff coming out, how can I verify that I have an actual human on the phone using only voice? Are there still vocal things humans can, but robots can't do?

Conditions: the person on the phone is a stranger (so personal questions won't work), and the verification must be voice only.

(Also, I understand Google Duplex may be just an overhyped demo that will turn out to flop like the Pixel Buds. But eventually such a bot would be created, right? If so, what's the best verification?)

your question is a very similar to "turing-test". you could narrate a simple story and ask questions based on that , considering the state-of-art algorithms in "question-answering" are still far beyond human skills.


How can I simulate responses from the distribution of human intelligence? I'm eventually looking to build an algorithm that will process answers from humans that are given questions. But first I have to setup an experiment to determine the variety of responses.

Specifically, humans will be asked a multiple choice question that has a single correct answer. I want to understand what kinds/ranges of responses I would get from the bell curve distribution of human intelligence.

Is there any way I can have, say, 1000 "humans" be asked a prompt, repeated 100 times (the same question) and then compile the responses? My concern is that I'll have to build some algorithm or process for each dumb, average, smart "human" to follow but then I would introduce bias in how smart they are or limit how they may respond. I'm guessing I'll have to give them a data sort to work from.

To clarify, it's not the number of times a single user gets a question right that makes them smart, they have to be programmed dumb, smart etc. before the simulation starts. So dumb users could get some right and smart can get some wrong.

I'm not sure the Monte Carlo method is useful here but some type of simulation where I can specify the distribution (normal) and then bound the responses would be helpful.

I have access to Excel, Minitab, and Python. Any ideas how to set up an experiment like this? I really am open to any technique to measure this.

The data you want to collect cannot be reliably simulated at this time. There is no current realistic simulator for a human performing reading comprehension.

The actual error rates and specific wrong answers chosen on the questions will depend to a large degree on specific humans, and the nature of questions. As you are hoping to get realistic results, the only method that will work for your ground truth data is to present 1000+ real humans with your sample questions. In addition, if you want to categorise your humans into "smart", "dumb" etc, you will need to run additional tests on them, such as an IQ test, in order to create those categories.

Depending on context, such as the nature of the questions you want to assess, you may be able to obtain some anonymised data from real-world exams that could help, instead of trying to generate that data yourself. In that case you could make an approximate model for humans answering multiple-choice questions - perhaps by training an LSTM-based natural language model. For best accuracy on your questions you would want the training set to include similar kinds of question. There is still a caveat that NNs do not really do logic or reasoning, they make statistical fits, so can easily get answers logically wrong or select nonsense answers. The best general NLP models still fail badly at semantics.

If you have specific set of questions to evaluate - and are not willing to ignore the content of the questions - then no machine can currently match distribution of human behaviour on such a task without extensive training data and significant effort.

If you don't care about the content of the questions, or assessing their difficulty, or indeed any feedback about the specific questions, then you could maybe use a dataset from any multi-choice questionnaire to get statistics of correct answer accuracy. With this approach your simulation could just be a simple distribution over "correct", "most plausible incorrect" etc answers where you choose "most plausible incorrect" either manually or randomly (but consistently). This will get you a distribution of responses similar to known real-world data. It could be used to unit-test a scoring system perhaps, or demonstrate some stats or visualisation software. But the question and answer text may as well be gibberish at that point.


How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network? I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience.

I'm using the following equations to calculate the gradients for weights and biases as well as the equations to find the derivative of the cost with respect to a hidden layer neuron:

The issue is, during backpropagation, the gradients keep cancelling each other out because I take an average for opposing training examples. That is, if I have two training labels being [1, 0], [0, 1], the gradients that adjust for the first label get reversed by the second label because an average for the gradients is taken. The network simply keeps outputting the average of these two and causes the network to always output [0.5, 0.5], regardless of the input.

To prevent this, I figured a softmax function would be required for the last layer instead of a sigmoid, which I used for all the layers.

However, I have no idea how to implement this. The math is difficult to understand and the notation is complicated for me.

The equations I provided above show the term: σ'(z), which is the derivative of the sigmoid function.

If I'm using softmax, how am I supposed to substitute sigmoid with it?

If I'm not mistaken, the softmax function doesn't just take one number analogous to the sigmoid, and uses all the outputs and labels.

To sum it up, the things I'd like to know and understand are:

The equation for the neuron in every layer besides the output is: σ(w1x1 + w2x2 + ... + wnxn + b). How am I supposed to make an analogous equation with softmax for the output layer?
After using (1) for forward propagation, how am I supposed to replace the σ'(z) term in the equations above with something analogous to softmax to calculate the partial derivative of the cost with respect to the weights, biases, and hidden layers?

One of the best ways to learn is use reference of others.

Have a look at Peter Sadowski - Notes on Backpropagation (Page 3):

There is also a great blog post by Eli Bendersky - The Softmax Function and Its Derivative.


Multi-param LSTM input Lets say you install your LSTM machine on a road between London and Oxford. And it makes observations. A car with 3 people inside drives past it in one direction 21 sec after previously observed car (in any direction) : the input is {3, LO, 21}. A bus with 43 people drives in other direction 11 sec later : {43, OL, 11}

I cant find a definitive general explanation (without references to existing ML packages code) - how your LSTM layer structure should look like to accept those params (there are 3 of them and they are vastly different between themselves but presenting them together at each step is very significant (as opposed to splitting them to 3 streams and feeding then into 3 LSTMs and then pooling the results).

Could someone explain it in a single formula or a drawing?

So you want your network to represent those 3 values at each step as single composite value? I can't think of any better way than utilizing 3 LSTM units but attaching them to same write and read nodes of the enclosing network. In other words your assumption that it makes sense to keep all those 3 values together, gets hardcoded into your network by making 6 connections (3 connected to read and 3 to write) share their weights. Usually researchers leave it to the network to decide whether to keep such composite values together or separate (by learning read and write weights through backpropagation) and sometimes network chooses to keep them together all on its own like in https://youtu.be/93rzMHtYT_0?t=531 (it can be seen how network resets whole bunch of LSTM units simultaneously ).


Can we evolve 0 and 1? Is it possible to combine or create conditional statements of 0 and 1, and optimize with an evolutionary algorithm (given that all computers use a binary system)?

There may be an algorithm that maps input and output to 0 and 1, or a conditional statement that edits a conditional statement.

An example of a binary conditional statement is if 11001 then 01110.

Just as molecules are combined to form living beings, we could begin with the most fundamental operations (0 and 1, if then) to develop intelligence.

Is it possible to combine or create conditional statements of 0 and 1, and optimize with an evolutionary algorithm (given that all computers use a binary system)?

Yes, evolutionary algorithms are very general, and can be used to modify almost any source data structure, including logic trees or even executable code, provided you have some measure of fitness to optimise for.

This can be taken to a highly abstract level, such as the paper Evolution of evolution: Self-constructing Evolutionary Turing Machine case study where an evolutionary algorithm is used to optimise other evolutionary algorithms which solve tasks using generic models of computing.

However, there are two important caveats:

There needs to be a measurement phase to establish the fitness of the algorithm. This can be very complex, depending on the problem you attempt to solve.
Genetic algorithms may be very general optimisers (capable of finding optimal solutions for problems when other algorithms may fail), but may also be very inefficient and slow depending on the size of the allowed genomes, and how the search space is structured relative to available genetic operations.

  begin with the most fundamental operations (0 and 1, if then) to develop intelligence.

Provided the fitness measure allows for the expression of intelligence, then this seems theoretically possible - in the sense that there are no known reasons why a sufficiently complex logical machine could not be intelligent by any measure we have of abstract intelligence (excluding measures deliberately constructed to exclude computational models such as "intelligence is the capability of a living system . . .")

However, such a project faces some barriers which currently look insurmountable:

There is no formal measure of general intelligence to use as a fitness function. This could be worked around using an e-life approach of providing a suitable rich virtual environment and allowing agents to compete for resources in the hope that the most competitive agents would exhibit intelligent behaviour - but that begs the question of how you could recognise and select those agents through any objective measure.
Any environment rich enough to select for general intelligence, whilst simulating low-level agent logic is likely to require a lot of computation.
Our one example of evolving basic building blocks into beings we call intelligent took billions of years, whilst processing billions upon billions of separate evolving entities at any one time.

These last two points imply a computational cost far beyond current technology, so there is no route to actually running this experiment for real.


Choosing an instance in AWS or GCP for training a CNN I'm new to the whole world of AI and neural networks, and I'm looking to get started with training a CNN on a cloud service like AWS or GCP but I'm totally overwhelmed by the choices for VM instances. In GCP, there are options that seem to have the same amount of memory but are classifed as standard, highmem, or highcpu -- what's the difference? As well what would be some reasons to opt for having a GPU as well?

I can't do a better job at explaining than Jeremy Howard's team did.

Follow carefully and systematically all the steps put together here.

You can read more about the different types of machines here, about the possible choices of GPUs here and about the regions and zones here.

Follow the fastai course too.
I can't emphasize how good is it.
You can find the lessons in the left panel.

I would recommend you use Colab for experiments as it has a Tesla-K80 and the cloud computing for something that requires more power.

Hope it helped!


Why is the policy implied by Q-learning deterministic, when it always chooses the action with highest probability? Q-learning uses the maximizing value at each step, which implies that there is a probability distribution and it happens to choose the one with the highest probability. There is no direct mapping between a particular state to ONLY a particular action but a bunch of actions with varying probabilities. I don't understand.

Q-learning uses the maximizing value at each step,

Mostly true. The target policy that Q-learning learns the action values of is the one with the maximum value. While training, Q-learning will take one action randomly (typically with a high probability of taking the action with maximum value), whilst it makes updates to value estimates assuming it will always take the maximising action next.

which implies that there is a probability distribution and it happens to choose the one with the highest probability.

Not true. This is not implied at all. The action values that Q-learning estimates are not probabilities, but expected sums of future reward. The probability distribution for exploring actions is added outside of the Q table or Q function estimate. There is no implied probability distribution of actions in the target policy. When implementing the target policy, you may decide to break ties randomly, but that is not an important detail.
It is worth noting that even if you were working with a table of action probabilities, then an agent that always chose the one with the maximum probability would be deterministic. A list of numbers from a table or non-stochastic function (which is how a Q table or Q function is implemented, and also how policy functions are implemented for methods that do process probabilities), even if it represents probabilities, cannot be random in itself. Instead, it must be interpreted by a process that decides how to use them. A process that includes a random number generator can use the numbers to generate a probability distribution that it samples from.
In Q-learning, the process that updates the Q table estimates does not include a random number generator, so as a consequence it is deterministic. However, the choice of which actions to explore is often random, and it is sometimes the case that the environment includes randomness in state transitions or reward values. So Q-learning taken as a whole is a random process.


Reinforcement Learning with long term rewards and fixed states and actions I have read a lot about RL algorithms, that update the action-value function at each step with the currently gained reward. The requirement here is, that the reward is obtained after each step.

I have a case, where I have three steps, that have to be passed in a specific order. At each step the agent has to make a choice between a range of actions. The actions are specific for each step.

To give an example for my problem:
I want the algorithm to render a sentence of three words. For the first word the agent may choose a word out of ['I', 'Trees'], the second word might be ['am', 'are'] and the last word could be chosen from ['nice', 'high']. After the agent has made its choices, the reward is obtained once for the whole sentence.

Does anyone know which algorithms to use in this kind of problem?

To give a bit more detail on what I already tried:

I thought that using value iteration would be an reasonable approach to test. My problem here is that I don't know how to assign the discounted reward for the chosen operations.

For example after the last choice I get a reward of 0.9. But how do I update the value for the first action (choosing out of I and Trees in my example)?

You don't need to have a reward on every single timestep, reward at the end is enough. Reinforcement learning can deal with temporal credit assignment problem, all algorithms are designed to work with it. Its enough to define a reward at the end where you, for example, give a reward of 1 if sentence is satisfactory or -1 if it isn't. Regular tabular Q-learning would easily solve the toy problem that you gave as an example.


Are there any novel quantum machine learning algorithms that are fundamentally different from "classical" ones? Generally, if one googles "quantum machine learning" or anything similar the general gist of the results is that quantum computing will greatly speed up the learning process of our "classical" machine learning algorithms. However, "speed up" itself does not seem very appealing to me as the current leaps made in AI/ML are generally due to novel architectures or methods, not faster training.

Are there any quantum machine learning methods in development that are fundamentally different from "classical" methods? By this I mean that these methods are (almost*) impossible to perform on "classical" computers.

*except for simulation of the quantum computer of course

Generally, if one googles "quantum machine learning" or anything similar the general gist of the results is that quantum computing will greatly speed up the learning process of our "classical" machine learning algorithms.

This is correct. A lot of machine learning methods involve linear algebra, and it often takes far fewer quantum operations to do things in linear algebra than the number of classical operations that would be needed. To be more specific, for a matrix of size $N\times N$, if a classical computer needs $f(N)$ operations to do some linear algebra operation, such as diagonalization which can take $f(N)=\mathcal{O}(N^3)$ operations on a classical computer, a quantum computer would often need only $\log_2 f(N)$ operations, which inn (in and only in) the language of computational complexity theory, means exponential speed-up. The "only in" part is there because we have made here an assumption that "fewer quantum operations" means "speed-up", which for now is something we only know to be true in the world of "computational complexity theory".

However, "speed-up" itself does not seem very appealing to me as the current leaps made in AI/ML are generally due to novel architectures or methods, not faster training.

I disagree. Take vanilla deep learning for example (without GANs or any of the other things that came up in the last decade). Hinton and Bengio had been working on deep learning for decades, so why did interest in deep learning suddenly start growing so much from 2011-2014 after a roughly monotonic curve from 1988-2010? Not that this rise started before newer advances such as GANs and DenseNet were developed:

Notice also the similarity between the above graph and these ones:

These days pretty much everyone doing deep learning uses GPUs if they have access to GPUs, and what is possible to accomplish is extremely tied to what computing power a group has. I don't want to undermine the importance of new methods and new algorithms, but GPUs did play a big role for at least some areas of machine learning, such as deep learning.

Are there any quantum machine learning methods in development that are fundamentally different from "classical" methods?

I think you mean: "Most quantum machine learning algorithms are simply based on classical machine learning algorithms, but with some sub-routine sped-up by the QPU instead of a GPU -- are there any quantum algorithms that are not based on classical machine learning algorithms, and are entirely different".
The answer is yes, and more experts might be able to tell you more here.
One thing you might consider looking at is Quantum Boltzmann Machines.
Another things I'll mention is that a child prodigy named Ewin Tang who began university at age 14, discovered at around the age of 17 some classical algorithms that were inspired by quantum algorithms rather than the other way around, and
the comments on the Stack Exchange question Quantum machine learning after Ewin Tang might give you more insight on that. This is related to something called dequantization of quantum algorithms.

By this I mean that these methods are (almost*) impossible to perform on "classical" computers. *except for simulation of the quantum computer of course

Unfortunately quantum computers can't do anything that's impossible for classical computers to do, apart from the fact that they might be able to do some things faster. Classical computation is Turing complete, meaning anything that can be computed can be computed on a big enough classical computer.


Image comparison algorithm, trying to figure out how similar two "binary" forms are I'm a student I'm completely new to this technology maybe my approach could be completely wrong, I want to create an algorithm that compares the similarity between two binarized images.

I'll explain:
I have 2 pictures as input. The RGB colors of these images can only be 0 or 255

(R = G = B = 255) or (R = G = B = 0). I take these two letters as an example.

1.  2.

I thought so: the 255 value is the background of the image which is white.
The 0 value is the shape (letter) formed in the image. So I thought of creating a matrix with 0 and 1 values where value 0 represents the background and value 1 represents the shape.

So now i would like to create an algorithm that understands the shape created in the two matrices and that returns a similarity percentage.

Update: I'm creating this app that tries to recognize the font of a text in a image (https://github.com/Sirvasile/Typefont), I want to create this algorithm to improve the comparison between the input letters and the alphabet of my fonts in the database.

I think you look for the meaning of the image (pic of 'a' = a) rather than the regular image similarity. For example, you would like the two images of the 'a' you posted to be classified to the same label instead of, for example, to be classified as similar to an 'o' image which would give a smaller Hamming distance or mean square error.
So as @Jaden Travnik suggested, MNIST digits classification is a good start. It classifies handwritten digits and has human-level solutions implemented by any existing CNN framework. (TensorFlow, Caffe, Theano, to name a few).


Why did the openai's gym website close? Openai's gym website redirects to the GitHub repository.  Why did the openai's gym website close?

According to Open AI's Greg Brockman, the Gym website never had a big impact and so was never maintained. This is the reason he gives for shutting down the website.

A read only export of the site was archived at https://gym.openai.com/read-only.html and if you attempt to access the old website through the url https://gym.openai.com you will be redirected to the Open AI gym github repository.

For a while a static copy of the Open AI leader-board was maintained on the below url https://scoreboard-site-1764008611.us-west-2.elb.amazonaws.com/ . However as of present the static site is also unreachable.

The Canadian AI startup montreal.ai had offered to maintain the gym website. However this offer was not taken up by open ai.

This issue was discussed in depth on the following threads

https://www.reddit.com/r/MachineLearning/comments/6zvlm2/d_openai_closing_down_gym_toolkit_website/?st=jca6ia3n&sh=48a0d8f3

https://github.com/openai/gym/issues/718


How to determine the number of hidden layers and units of a deep auto-encoder? I am using a deep autoencoder for my problem. However, the way I choose the number of hidden layers and hidden units in a hidden layer is still based on my feeling.
The size of the model that indicates the number of hidden layers and units should not be too much or too few for the model can capture useful features from the dataset.
So, how do I choose the right size of the deep autoencoder model is enough to good?

You are right!
1- the number of hidden layers shouldn't be too high! Because of the gradient descent when the number of layers is too large, the gradient effect on the first layers become too small! This is why the Resnet model was introduced.
2- the number of hidden layers shouldn't be too small to extracts good features. It's proved that in CNN networks the first layers extract very simple elements like lines and curves but last layers extracts more complex features.
3- number of hidden units is a hyper-parameters and usually you should find it by testing or based on your background knowledge.
But what can you do at all?
As you can tests different parameters and compare their results there is some other options! One option is grid search you can check this tutorial https://towardsdatascience.com/grid-search-for-model-tuning-3319b259367e