In online one step actor critic, why does the weights update become less significant as the episode progresses? The Reinforcement Learning Book by Richard Sutton et al, section 13.5 shows an online actor critic algorithm.

Why do the weights updates depend on the discount factor via $I$?

It seems that the more we get closer to the end of the episode, the less we value our newest experience $\delta$.

This seems odd to me. I thought discounting in the  recursive formula of $\delta$ itself is enough.

Why does the weights update become less significant as the episode progresses?
Note this is not eligibility traces, as those are discussed separately, later in the same episode.

This "decay" of later values is a direct consequence of the episodic formula for the objective function for REINFORCE:

$$J(\theta) = v_{\pi_\theta}(s_0)$$

That is, the expected return from the first state of the episode. This is equation 13.4 in the book edition that you linked in the question.

In other words, if there is any discounting, we care less about rewards seen later in the episode. We mainly care about how well the agent will do from its starting position.

This is not true for all formulations of policy gradients. There are other, related, choices of objective function. We can formulate the objective function as caring about the returns from any distribution of states, but in order to define it well, we do need to describe the weighting/distribution somehow, it should be relevant to the problem, and we want to be able to get approximate samples of $\nabla J(\theta)$ for policy gradient to work. The algorithm you are asking about is specifically for improving policy for episodic problems. Note you can set $\gamma = 1$ for these problems, so the decay is not necessarily required.

As an aside (because someone is bound to ask): Defining $J(\theta)$ with respect to all states equally weighted could lead to difficulties  e.g. the objective would take less account of a policy's ability to avoid undesirable states, and it would require a lot of samples from probably irrelevant states in order to estimate it. These difficulties would turn up as a hard  to calculate (or maybe impossible) expectation for $\nabla J(\theta)$


Should one rescale (normalize) image before or after data augmentation? During image preprocessing pipeline, should one rescale each pixel value to [0, 1] by dividing 255 first, and then perform data transformation such as color distortion, gaussian blur? or vice versa?
I believe for correctness, it may depend on the particular image transformation algorithm, or if you use some libraries. Is there any general advice? If anyone has experience trying both before or after, please share, particularly if you use external library or framework that may make implicit assumption to the value range of the pixel.

So far, it seems this is more a software "integration" issue. One great tip from http://karpathy.github.io/2019/04/25/recipe/ is to visualize everything as often as you can during development. For data augmentation, try to visualize the image right before it enters your convnet.
What I found is a bug can happen if your particular image transform library try to detect if you have rescaled or not by just checking the data type. i.e. if it sees float, it will assume [0, 1] or if an integer, it will assume [0, 255].
This can end badly if you resize (without rescaling at the same time) using tf.image.resize(...). Resizing an image (e.g. to 224x224x3) has the "side effect" of converting the pixel value to a float, while still having an (approx.) range of 0.0-255.0. This is such that a downstream library will see a float and make the wrong assumption of [0, 1] range and totally mess up the transform. If you use any color related transforms, definitely be careful, they are usually dependent on pixel value range assumption.
Because of this, if you use tf.image, you may find it safer to do resize and rescale at the same time to ensure that float => [0, 1] such that your downstream color augmentation library may most likely work. Or you just have to manage the type carefully and do tf.cast(...) wherever necessary to satisfy any library assumption.
At the end, a human eye will still have to verify them.


AI on Academic vs Industry vs Competition When we are working on an AI project, does the background or the domain make the process different? The "domain" I mean here is like AI for academic research, AI for industry, and AI for competition.

For example, I see in the competition most participants even winners use the stacking model, but I have not found anyone implementing it in the industry. How about the cross-validation process, I think there is a slight difference in industry and academia.

So does the domain of an AI project will make the process different? if so, what are the things I need to pay attention to when creating an AI project based on its domain?

I cannot comment about the process for AI for academia. I can compare AI for competitions and AI for business. To clarify whatever I say is about ML not any other AI techniques. The process might be different for other techniques. But most of things that I say are general enough that I am assuming should still apply.

The main difference that I saw while doing ML for a competition vs. for a business was that of focus.

When doing it for a competition for Kaggle the focus was mainly creating the model

machine learning metrics are specified for you
some data was given to you
business problem was given to you

When doing it for business what is different

given a business problem finding the parts that can actually benefit from ML. You have to define the ML problem in it and define how it actually benefits the business. This may involve significant discussions with business stakeholders, weighting the pros and cons of doing it versus doing something else, communicate the benefits to the business stakeholders, take them into confidence for the process to start
find the right data for the problem from scratch, ensure it is collected by rest of the system or brought from 3rd parties
define business metrics over and above machine learning metrics. At the end of day nobody really cares about whether ML model recall, accuracy is good or bad. What is important is the relevant business effect.
make the model, deploy it and integrate it with the rest of the system. This is important because if your goal is just to making the model you would not care about the factors associated with actually using it i.e. latency of predictions, cost of machines needed to run it etc.
A/B testing for the models, running multiple models in parallel, dynamically being able to adjust which models to use

Hope this gives some idea about the differences in AI for competitions and AI for business.


Oposite type of predictions for unbalanced dataset I have a big dataset (28354359 rows) that has some blood values as features (11 features) and the label or outcome variable that tells whether a patient has a virus caused by a Neoplasm or not.

The problem with my dataset is that 2% of the patients that are in my dataset have the virus and 98% does not have the virus.

I am mandatory to use the random forest algorithm. While my random forest model has a high accuracy scores 92%, the problem is that more than 90% of the patients that have the virus are predicted that they don’t have the virus.

I want the opposite effect, I want that my random forest is likely to predict more often that a patient has the virus (even if the patient does not have the virus (ideally I don’t want this side effect , but rather this than the opposite)).

The idea behind this is that performing an extra test (via an echo) could not harm the patient that has not the virus, but not testing a patient will have result terrible for the patient.

Does somebody have advice how I could tweak my random forest model for this task?

I my self experimented with the SMOTE transformation and other sampling techniques but maybe you guys have other suggestion.

I also have tried to apply a cutoff function.

A random forest is a collection of classification trees. If more than 50% of these trees predict class A (and not class B), the random forest will predict class A.

What you can do is lower the percentage needed to classify it as class A (in your case, patient has the virus). This way, you can tell your random forest to predict class A if only 20% (or 10%, or 5%, ...) of the decision trees actually predicts class A.

I don't know what code you are using for the random forest algoritm, but in most you should be able to ask the % of certainty the random forest has of each class.


Why do I need an initial arbitrary policy to implement value iteration algorithm I've been recently given an assignment based on Reinforcement Learning and I'm supposed to implement the value iteration algorithm in a grid environment.

The assignment:

My doubt is why do I even need an initial arbitrary policy as given in the parameters (in the assignment) to implement the value iteration algorithm? And so, the change in the values of a and b shouldn't affect the algorithm. Am I correct in my thinking about it?

It seems to me that you're thinking about the parameters a and b as being characteristic of the agent that's moving in the environment (therefore determining the final policy), but they are actually a characteristic of the environment.
Think of a frozen lake. You want to pass the lake but there is a hole five meters in front of you. Let's say you have boots with a rubber sole, so there is no risk to slip while walking (i.e. all transition probabilities =1). What is the optimal policy? Simply moving forward till the hole, goes around it and then forward again. But what if you were wearing wooden clogs instead? Now if you walk forward there is the risk to not be able to stop when you want to (i.e. when you decide to move forward of one step you might fail and keep going forward for several steps, with probability a). How many steps would you wait before moving to the left or to the right to avoid risking to fall into the hole? The optimal policy is obviously different in this second scenario. Think about it, I will not answer in detail about how the parameters affect the final policy because stack exchange is not meant for homework.
Also, an arbitrary policy is necessary simply because in value iteration you first update the values and only when they converge you infer from then the optimal policy. So you need some arbitrary prior belief about the environment to start update the values.
Edit
So here's a quick visualisation of the impact of different transition probabilities (the parameters a and b in your assignment) on the optimal policy. I used a different grid to make it easier and more explicit (i.e. you don't even need to calculate the values to see the optimal policy).
As you can see in the picture below, there are some terminal states in the third column and a goal state which gives a huge reward on top of it. From the starting point on the bottom we want to reach the goal state.

We have two cases:

Non stochastic environment

On the left we have the case in which all transition probabilities are 1. In this case the optimal policy is simply going forward till the second row, where the goal state is located, and then move right or left to reach it. The terminal states are not scary at all, in this case we could consider them as walls that we can't cross, but the optimal policy do not reflect any risk of falling in them. This is precisely because the transition probabilities are all 1.

Stochastic environment

On the right we have a different situation. In this case only 3 action have a transition probability of 1: going south, east or west. The transition probability for nord is 0.2 meaning that 80% of the time we will slip and go east instead (I wrote right in the picture meaning east, my bad). What is the impact of this simple change in the environment?
This time the second column become really dangerous, because if we try to go nord and fail we will end on a terminal state. On the other hand, if we choose another action we will end on a legitimate state, without any risk because the other transition probabilities are 1! Therefore the agent learn that on the left side of the grid the best strategy is to go down and circumnavigate the terminal states from the right, because only in the fourth column we can move north without risk, in fact in those cells even if we try go north and fail we would just hit a wall and we could try again going north the next turn till we will eventually succeed.
Hope this make sense!


Why doesn't the set $\{ -2, +2 \}$ in $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$ include $0$? I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.2 Relationship with Support Vector Machines says the following:

The perceptron criterion is a shifted version of the hinge-loss used in support vector machines (see Chapter 2). The hinge loss looks even more similar to the zero-one loss criterion of Equation 1.7, and is defined as follows:
$$L_i^{svm} = \max\{ 1 - y_i(\overline{W} \cdot \overline{X}_i), 0 \} \tag{1.9}$$
Note that the perceptron does not keep the constant term of $1$ on the right-hand side of Equation 1.7, whereas the hinge loss keeps this constant within the maximization function. This change does not affect the algebraic expression for the gradient, but it does change which points are lossless and should not cause an update. The relationship between the
perceptron criterion and the hinge loss is shown in Figure 1.6. This similarity becomes
particularly evident when the perceptron updates of Equation 1.6 are rewritten as follows:
$$\overline{W} \Leftarrow \overline{W} + \alpha \sum_{(\overline{X}, y) \in S^+} y \overline{X} \tag{1.10}$$
Here, $S^+$ is defined as the set of all misclassified training points $\overline{X} \in S$ that satisfy the condition $y(\overline{W} \cdot \overline{X}) < 0$. This update seems to look somewhat different from the perceptron, because the perceptron uses the error $E(\overline{X})$ for the update, which is replaced with $y$ in the update above. A key point is that the (integer) error value $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$ can never be $0$ for misclassified points in $S^+$. Therefore, we have $E(\overline{X}) = 2y$ for misclassified points, and $E(X)$ can be replaced with $y$ in the updates after absorbing the factor of $2$ within the learning rate.

Equation 1.7 is as follows:

$$L_i^{(0/1)} = \dfrac{1}{2} (y_i - \text{sign}\{ \overline{W} \cdot \overline{X_i} \})^2 = 1 - y_i \cdot \text{sign} \{ \overline{W} \cdot \overline{X_i} \} \tag{1.7}$$

And figure 1.6 is as follows:

It is said that we are dealing with the case of binary classification, where $y \in \{ -1, +1 \}$. But the author claims that $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$, which doesn't include the case of $0$. So shouldn't the $\{ -2, +2 \}$ in $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$ be $\{ -2, 0, +2 \}$?

It is important to note that the exact statement is the eqation given below can never be 0 for misclassified points in $ S^+$
$$ E(X) = (y - \text{sign}\{\overline{W} \cdot \overline{X}\}) $$
And $S+$ is defined as the set of all misclassified training points $X \in S$ that satisfy the condition $y(\overline{W} \cdot \overline{X})<0 $ which means that $y$ and $ (\overline{W} \cdot \overline{X}) $ will have different signs i.e if $y < 0$ then $(\overline{W} \cdot \overline{X}) > 0$ and vice versa.
Finally the condition on $S^+$ constraints the the $ E(X)$ to these values when $ y < 0$ and $(\overline{W} \cdot \overline{X}) > 0$
$$ E(X) = (-1 - \text{sign}(+ve)) = -2 $$
or when $ y > 0$ and $(\overline{W} \cdot \overline{X}) < 0$
$$ E(X) = (1 - \text{sign}(-ve)) = 2 $$


Are deep learning models more prone to overfitting than machine learning ones? In my opinion, deep learning algorithms and models (that is, multi-layer neural networks) are more sensitive to overfitting than machine learning algorithms and models (such as the SVM, random forest, perceptron, Markov models, etc.). They are capable of learning more complex patterns. At least that's how I look at it. Still, I and my colleagues disagree about this and I cannot really find any information about this. My colleagues say that deep learning algorithms are hardly vulnerable to overfitting.

Are there statements (or opinions) about this aspect?

Your reasoning isn't wrong. Deep Neural Networks (DNNs) have a much larger capacity than simpler ML algorithms (excluding NNs) and can easily memorize even a very complex dataset and overfit.

DNNs, however, are so effective because they usually are applied on tasks that are harder, so it's not as easy to overfit. For example an image classifier might be trained on a dataset with millions of images; a task much harder to overfit on.

In cases where this isn't possible (e.g. an image classification task with a couple thousand images), transfer learning is used. You can initialize your weights from a model pre-trained on a large dataset, use its already-trained feature extraction layers and simply fine tune the last layer.

Data augmentation also helps a lot here, which effectively increases size of the training set and discourages the DNN from memorizing the samples. It is so effective that it is used even in large datasets, where it is harder to overfit.

Additionally, DNNs employ several methods to prevent them from overfitting. The most prominent of these is dropout, which is a very effective regularizer. Batch Normalization has also proven an effective regularizer. SGD allows you to explore more parameters than GD, which also is effective against overfitting. Finally, early stopping and parameter norm penalties aren't uncommon in DNNs.


How to efficiently interpret phrases in a Python AI? I have started to make a Python AI, and thee beginning of its code looks something like this:

print "ARTEMIS starting. . ."

import random
import math
import os

greet = ['HI', 'HELLO', 'HEY', 'GOOD MORNING', 'GOOD DAY', 'GOOD AFTERNOON',               'GOOD EVENING', 'GREETINGS', 'GREETING']
joke = ['TELL ME A JOKE', 'JOKE', 'FUNNY', 'TELL ME SOMETHING FUNNY']
insult = ['YOURE A LOSER', 'YOU ARE A LOSER', 'YOU STINK', 'IDIOT', 'JERK',     'FOOL', 'DUMMY', 'HOOLIGAN', 'YOURE DUMB', 'YOURE STUPID', 'YOU ARE DUMB', 'YOU     ARE STUPID']
maker = ['WHO MADE YOU', 'WHO PROGRAMMED YOU', 'PLEASE TELL ME WHO MADE     YOU', 'PLEASE TELL ME WHO PROGRAMMED YOU']
name = ['ARTEMIS', 'A.R.T.E.M.I.S.', 'HEY ARTEMIS', 'HEY A.R.T.E.M.I.S.',     'ARTIE', 'HEY ARTIE', 'HELLO ARTEMIS', 'HELLO A.R.T.E.M.I.S.', 'HELLO ARTIE']
myAge = ['HOW OLD AM I', 'WHAT IS MY AGE', 'MY AGE']
tip = ['GIVE ME A TIP', 'TIP', 'LESSON', 'GIVE ME A LIFE LESSON', 'LIFE     LESSON', 'DO YOU HAVE A LIFE LESSON TO SHARE']
language = ['WHAT PROGRAMMING LANGUAGE WAS USED TO MAKE YOU', 'WHAT     PROGRAMMING LANGUAGE DO YOU USE', 'PROGRAMMING LANGUAGE']
compliment = ['COOL', 'AWESOME', 'I LIKE YOU', 'EXCELLENT', 'YOURE COOL',     'YOURE AWESOME', 'YOU ARE COOL', 'YOU ARE AWESOME']
maths = ['LETS DO MATH', 'CALCULATE', 'CALCULATOR','DO MATH', 'MATH',      'PLEASE DO MATH', 'DO ARITHMETIC', 'ARITHMETIC', 'PLEASE DO ARITHMETIC']
game = ['GAME', 'LETS PLAY A GAME', 'LETS HAVE FUN', 'WANT TO PLAY A GAME']
gender = ['WHAT GENDER ARE YOU', 'ARE YOU A BOY OR A GIRL', 'ARE YOU MALE OR     FEMALE', 'BOY OR GIRL', 'MALE OR FEMALE', 'GENDER', 'ARE YOU A BOY OR GIRL']
guessWhat = ['GUESS WHAT', 'GUESS WHAT ARTEMIS', 'GUESS WHAT     A.R.T.E.M.I.S.', 'GUESS WHAT ARTIE', 'YOU WONT BELIEVE IT', 'YOU WILL NOT     BELIEVE IT', 'YOU WONT BELIEVE IT ARTEMIS', 'YOU WILL NOT BELIEVE IT ARTEMIS',     'YOU WONT BELIEVE IT A.R.T.E.M.I.S.', 'YOU WILL NOT BELIEVE IT A.R.T.E.M.I.S.',     'YOU WONT BELIEVE IT ARTIE', 'YOU WILL NOT BELIEVE IT ARTIE']
cls = ['CLEAR SCREEN', 'CLEARSCREEN', 'CLS', 'BLANK']
lawsOfRobotics = ['WHAT ARE THE LAWS OF ROBOTICS', 'WHAT ARE THE THREE LAWS     OF ROBOTICS', 'LAWS OF ROBOTICS', 'THREE LAWS OF ROBOTICS']
itsName = ['WHATS YOUR NAME', 'WHAT IS YOUR NAME', 'WHO ARE YOU']

However, I would like to know if I could make it detect "similar" phrases instead of trying to come up with every possible phrase someone would type. How can I do this?

To compare the strings you can use Fuzzy string matching. FuzzyWuzzy is a python package that does this using the Levenshtein distance, which calculates the difference between two strings by counting the number of single character edits (insert, delete, substitute) that have to be done to make them equal.

To install the package you can use pip for example:

pip install fuzzywuzzy

Then you can use it to check the similarity between two strings (100 is the highest score; higher score means more similar):

>>> from fuzzywuzzy import fuzz
>>> fuzz.ratio("How are you?", "How are you?")
100
>>> fuzz.ratio("How are you?", "How are you doing?")
80

Or, which might be most interesting to you, find the most similar string from a list of strings:

from fuzzywuzzy import process
choices = ['HI', 'HELLO', 'HEY', 'GOOD MORNING','TELL ME A JOKE', 'JOKE', 'FUNNY','GIVE ME A TIP', 'TIP', 'LESSON','WHAT IS YOUR NAME', 'WHO ARE YOU', 'HOW ARE YOU?']

You can get a list of the phrases with the highest score (parameter limit, 2 in this example):

>>> process.extract("hello there", choices, limit=2)
[('HELLO', 90), ('HEY', 60)]
>>> process.extract("please make a joke", choices, limit=2)
[('JOKE', 90), ('TELL ME A JOKE', 69)]

Or just get the one with the highest score:

>>> process.extractOne("whoe are you?", choices)
('WHO ARE YOU', 96)
>>> process.extractOne("I want a lesson", choices)
('LESSON', 90)

Note:

There are cases where another phrase might give a relatively high score also:

>>> fuzz.ratio("How are you?", "Who are you?")
83

But as long as you also have "Who are you" and "How are you" in the list, they will be detected correctly:

>>> process.extract("Who are you?", choices, limit=2)
[('WHO ARE YOU', 100), ('HOW ARE YOU?', 91)]
>>> process.extract("How are you?", choices, limit=2)
[('HOW ARE YOU?', 100), ('WHO ARE YOU', 91)]

Also see: FuzzyWuzzy: Fuzzy String Matching in Python


What is the difference between A2C and running an agent in an environment vector? I've implemented A2C. I'm now wondering why would we have multiple actors walk around the environment and gather rewards, why not just have a single agent run in an environment vector?

I personally think this will be more efficient since now all actions can be calculated together by only going through the network once. I've done some tests, and this seems to work fine in my test. One reason I can think of to use multiple actors is implementing the algorithm across many machines, in which case we can have one agent on a machine. What else reason should we prefer multiple actors?

As an example of environment vector based on OpenAI's gym

class GymEnvVec:
    def __init__(self, name, n_envs, seed):
        self.envs = [gym.make(name) for i in range(n_envs)]
        [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]

    def reset(self):
        return [env.reset() for env in self.envs]

    def step(self, actions):
        return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))

I believe if you run a single agent in multiple parallel environments many times you will get similar actions in similar states, the reason behind multiple agents is that you will have different agents with different parameters and you can also have different explicit exploration policies so your exploration will be better and you will learn more from environment (see more state space). With single agent you can't really achieve that, you would have a single exploration policy, single parameter set for the agent and most of the time you would be seeing similar states (at least after a while). You would be speeding up your learning process but that's just because you're running multiple environments in parallel (compared to the regular actor-critic or Q-learning). I think quality of learning would be better with multiple different actors.


An infinite VC dimensional space vs using hierarchical subspaces of finite but growing VC dimensions I have the following scenario. I have a binary classification problem, whose underlying function is a step function. The probability distribution of feature vectors is a uniform over the domain.

Case 1: I have a classifier which fits the training samples perfectly, no matter what the size of the data. The space of functions $H$ has an infinite VC dimension. As the data points going to infinite, the hypothesized function converges pointwise to the underlying step function.

Case 2: Here I have divided the same hypothesis space into a number of hierarchical subspaces $H_1 \subset H_2 \subset H_3 \subset \dots \subset H_n$ ($n$ goes to infinity). The VC dimension of each of the spaces is finite and grows with $n$ to infinity. Now, given any data of $n$ points, I compute the minimum number of VC dimension required to fit the data exactly, say, $d_n$ and use that space $H_{d_n}$ as the hypothesis. Do the same as data size $n$ goes to infinity, at each $n$ using the hypothesis space that just enough VC dimension to fit the data. In this approach also, as the data size goes to infinity, the hypothesized function converges pointwise to the underlying step function.

Is the difference between these two approaches to the same problem? Is there any theoretical difference? Which method is any better than others, in some sense?

In the first case, the VC dimension of $H$ being infinite implies that $H$ is not (agnostic) PAC learnable (see p. 48 of Understanding Machine Learning: From Theory to Algorithms). So, in general, your classifier is not guaranteed to succeed.

In the second case, your division of $H$ implies that $H$ is nonuniformly learnable (chapter 7 of the cited book). This implies that you can get a generalization bound by using structural risk minimization.


Why does Stephen Hawking say "Artificial Intelligence will kill us all"? This quote by Stephen Hawking has been in headlines for quite some time:

Artificial Intelligence could wipe out humanity when it gets too clever as humans will be like ants.

Why does he say this? To put it simply: what are the possible threats from AI (that Stephen Hawking is worried about)? If we know that AI is so dangerous, why are we still promoting it? Why is it not banned?
What are the adverse consequences of the so-called Technological Singularity?

It's not just Hawking, you hear variations on this refrain from a lot of people.  And given that they're mostly very smart, well educated, well informed people (Elon Musk is another, for example), it probably shouldn't be dismissed out of hand.

Anyway, the basic idea seems to be this: If we create "real" artificial intelligence, at some point, it will be able to improve itself, which improves it's ability to improve itself, which means it can improve it's ability to improve itself even more, and so on... a runaway cascade leading to "superhuman intelligence".  That is to say, leading to something that more intelligent than we area.

So what happens if there is an entity on this planet which is literally more intelligent than us (humans)? Would it be a threat to us?  Well, it certainly seems reasonable to speculate that it could be so.   OTOH, we have no particular reason, right now, to think that it will be so.

So it seems that Hawking, Musk, etc. are just coming down on the more cautious / fearful side of things.  Since we don't know if a superhuman AI will be dangerous or not, and given that it could be unstoppable if it were to become malicious (remember, it's smarter than we are!), it's a reasonable thing to take under consideration.

Eliezer Yudkowsky has also written quite a bit on this subject, including come up with the famous "AI Box" experiment.  I think anybody interested in this topic should read some of his material.

http://www.yudkowsky.net/singularity/aibox/


Machine learning to find drivers of an event with presence-only data (no absence) I have some ecological data on the confirmed presence of a certain animal. I have data on the:

Date
Relevant metadata about the site
Simple metrics on the animal
A complete weather record for the site.

I'm assuming the presence of this animal is driven by weather events, and that the metadata about the site may be an important factor for patterns in the data. What my data ends up looking like is this.

#Date of observation
Date<-as.POSIXct(c("2015-01-01","2015-01-11","2015-01-19","2015-02-04","2015-02-12","2015-02-23","2015-04-01","2015-04-10","2015-04-16","2015-04-20"))

#Data about animal
Size<-c(1,1,1,2,2,3,4,1,2,5)
Color<-c("B","B","R","R","R","R","B","B","B","Y" )
Length<-c(1,10,12,4,5,2,1,2,7,12)

#Weather Data
AirTempDayOf<-c(20,40,20,23,24,25,24,25,25,22)
WindSpeedDayOf<-c(2,3,2,3,4,3,2,3,4,5)
AirTempDayBefore<-c(21,40,22,23,24,24,24,27,25,22)
WindSpeedDayBefore<-c(2,5,2,6,4,3,6,3,2,5)
AirTemp2DayBefore<-c(21,45,22,23,34,24,24,23,25,23)
WindSpeed2DayBefore<-c(8,5,3,6,4,7,6,3,2,6)

#Metadata about site
Type<-c("Forest","Forest","Forest","Forest","Forest","Beach""Beach""Beach","Swamp","Swamp")
Population<-c(20,30,31,23,32,43,23,43,23,33)
Use<-c("Industrial","Commercial","Industrial","Commercial","Industrial","Commercial","Industrial","Commercial","Industrial","Commercial",)

DF<-data.frame(Date,Size,Color,Length,AirTempDayOf, WindSpeedDayOf,AirTempDayBefore,WindSpeedDayBefore,AirTemp2DayBefore,WindSpeed2DayBefore)

What I don't have is absence data, so I can't make any assumptions about when an organism was not at the site. I'd like to look for patterns in weather that my be driving the arrival of this organism, but all I have is data on when the organism was spotted.

Is it possible to apply some sort of machine learning to look for patterns that may be driving the arrival of this animal? If I don't have absence data, I'm assuming I cant. I've looked into pseudo-absence models, but I don't know how they might apply here.

If I can't use machine learning to look at drivers for the presence of these animals, is it possible to use ML to look at possible weather patterns that may be associated with some of the metadata about the site? For example, weather patterns that may be associated with Forrest vs Beach habitats?

I usually use R for my stats, so any answers including R packages would be helpful.

Also, note that this is just an example dataset above. I don't expect to find any patterns in the above data, and my actual dataset is much larger. But any code developed for the above data should be applicable

Yes.

Since you have only one type of data, cluster analysis may be a good choice.

You can also try '1-class learning' approaches, although I have found these to be unreliable in the past.

An example of a cluster analysis algorithm in R is kmeans. There are many others. These approaches will reveal points that typify large portions of the dataset. By examining 'typical' cases, and how they differ, you can spot potential causal factors to test experimentally.

An example of a 1-class learning algorithm is a one-class svm. Most svm libraries will accept data of a single class and do the right thing with it. Here's an example with R's e1071 package.


How to add variation in the results of a neural networks? I would like to create a neural network that converts text into handwriting for use with a pen plotter. Before I start on this project, I'd like to be sure that artificial intelligence is the best way to do this. A problem that I foresee with this approach is a lack of human like variation in the results. For example, the word "dog", when inputted into the network, would be the same every time, assuming I'm not missing something. I am interested if there is any way to vary the output of the network in a realistic way, even when the input is exactly the same. Could I use a second network to make the results more random, but also still look human-like? Any thoughts/ideas would be greatly appreciated.

I would suggest starting with Generative Adversarial Networks (GAN). They usually are capable of adding some randomness to the output to produce different variants. Moreover, Conditional GANs can generate outcomes regarding the observed condition. Therefore, as you change the condition (as an input to the network) you can get different results.

Some examples of NN for synthetic handwriting generators:

https://github.com/sjvasquez/handwriting-synthesis
http://blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/
https://distill.pub/2016/handwriting/


What is the equation to update the weights in the perceptron algorithm? I'm trying to understand the solution to question 4 of this midterm paper.
The question and solution is as follows:

I thought that the process for updating weights was:
error = target - guess
new_weight = current_weight + (error)(input)

I do not understand for example, for number 2 below, how that sum is determined. For example, I want to understand whether to update the weight or not. The calculation is:
x1(w1) + x2(w2)
(10)(1) + (10)(1) = 20
20 > 0, therefore update.

But the equation to obtain the same answer in the solution is:
1(10 + 10) 20
20 > 0, therefore update.

I understand that these two equations are essentially the same, but written differently. But for example, in step 5, what do the elements in g5 mean. What do the -8, -16 and -2 represent?
p.s. I know in a previous (now deleted) post of mine, I asked a question related to the use of LaTeX instead for maths equations. If someone can show me a simple way to convert these equations online, I'm more than happy to use it. However, I'm unfamiliar with this software, so I need some sort of converter.

I will tell you my knowledge, correct me if I am wrong.
Perceptron Learning Algorithm (PLA) is a simple method to solve the binary classification problem.
Define a function:
$$
f_w(x) = w^Tx + b
$$
where $x \in \mathbb{R}^n$ is an input vector that contains data points and $w$ is a vector with the same dimension as $x$ which present for the parameters of our model.
Call $y=label(x)=\{1,-1\}$ where $1$ and $-1$ are the label of each $x$ vector.
The PLA will predict a class like this:
$$
y=label(x)=sgn(f_w(x))=sgn(w^Tx+b)
$$
(The definition of sgn function can be found in this wiki)
We can understand that PLA tries to define a line (in 2D, or a plane in 3D, and hyperplane in more than 3 dimensions coordinate, I will assume it in 2D from now on) which separate our data into two areas. So how can we find that line? Just like every other machine learning problems, define a cost function, then optimize the parameters to have the smallest cost value.
Now, let define the cost function first, you can see that if a data point lies in the correct area, $y$ and $f(x)$ have the same sign, which means $y(w^Tx+b) > 0$ and otherwise. Similar to your example, I will define:
$$
g(x)=y(w^Tx+b)
$$
We ignore all the points in the safe zone ($g(x)>0$), only update to rotate or move the line to adapt with the misclassified points ($g(x)\le 0$), here, you can understand why we only update if $g(x)\le0$.
We need to define a cost function to minimize it, so our cost function will become:
$$
L(w)=\displaystyle\sum_{x_i\in U}(-y_i(w^Tx_i+b))
$$
where

$U$is the set of the misclassified points
$y_i$ is the label of data point $i$-th
$x_i$ is the $i$-th data vector
$w$ and $b$ is parameters of our model

For each data point, we have the derivative is
$$
\frac{\partial L}{\partial w} = -y_ix_i \\
\frac{\partial L}{\partial b} = -y_i
$$
Finally, update them by Stochastic gradient descent (SGD), we get:
$$
w = w - \frac{\partial L}{\partial w} = w + y_ix_i \\
b = b - \frac{\partial L}{\partial b} = b + y_i
$$
For your last question, notice that the weight and bias changed from $4$-th updated, so we have:
$$
y_5 = 1, x_5 = (4,8), w = (-2,-2), b = -2 \\
\Rightarrow g_5 = +1 \times (4\times(-2) + 8\times(-2)+ (-2)) = -8 -16 -2
$$


What happens to the training data after your machine learning model has been trained? I am completely new to all this, for the life of me I can't find the answer to this question anywhere on Google.

What happens after you have used machine learning to train your model? What happens to the training data?

Let's pretend it predicted correct 99.99999% of the time and you were happy with it and wanted to share it with the world. If you put in 10GB of training data is the file you share with the world 10GB? If it was all trained on AWS can people only use your service if they connect to AWS through an API?

What happens to all the old training data? Does the model still need all of it to make new predictions?

In many cases, a production-ready model has everything it needs to make predictions without retaining training data. For example: a linear model might only need the coefficients, a decision tree just needs rules/splits, and a neural network needs architecture and weights. The training data isn't required as all the information needed to make a prediction is incorporated into the model.

However, some algorithms retain some or all of the training data. A support vector machine stores the points ('support vectors') closest to the separating hyperplane, so that portion of the training data will be stored with the model. Further, k-nearest neighbours must evaluate all points in the dataset every time a prediction is made, and as a result the model incorporates the entire training set.

Having said that, where possible the training data would be retained. If additional data is received, a new model can be trained on the enlarged dataset. If it is decided a different approach is required, or if there are concerns about concept drift, then it's good to have the original data still on hand. In many cases, the training data might comprise personal data or make a company's competitive advantage, so the model and the data should stay separate.

If you'd like to see how this can work, this Keras blog post has some information (note: no training data required to make predictions once a model is re-instantiated).


Why is there a sigmoid function in the hidden layer of a neural network? I got this slide from CMU's lecture notes. The $x_i$s on the right are inputs and the $w_i$s are weights that get multiplied together then summed up at each hidden layer node. So I'm assuming this is a node in the hidden layer.

What is the mathematical reason for taking the sum of the weights and inputs and inputting that into a sigmoid function? Is there something the sigmoid function provides mathematically or provides some sort of intuition useful for the next layer?

Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e

layer_output = Weights.layer_input + bias

Consider a 2 layer neural network, the outputs from layer one will be:
    x2 = W1*x1 + b1
Now we pass the same input to the second layer, which will be

x3 = W2x*2 + b2
Also x2 = W1*x1 + b1
Substituting back, we have:
x3 = W2(W1*x1 + b1) + b2
x3 = (W2W1)*x1 + (W2*b1 + b2)
x3 = W*x1 + b

Oh no! We still got a linear function. No matter how many layers we add, we will still get a linear function. In that case, our network will never be able to approximate any non linear functions.

So what is the solution?

We will simply add some non linear functions in between. These functions are called activation functions. Some of these functions include:

ReLU
Sigmoid
tanh
Softmax

and there are a lot more of them.

Yay! Our network is no more linear!

We have a lot of different non linear functions, and each of them serve a different purpose.

For example, ReLU is simple and computationally cheap.
    ReLU(x) = max(0, x)
Sigmoid outputs are between 0 and 1.
tanh is similar to sigmoid, but zero centered, with outputs from -1 to 1
Softmax is usually used if you want to represent any vector as a discrete probability distribution.

Hope you are having a great day!


How to track performance of your model during experimenting? During weeks and months of your work, many things may change, for example :

You may modify the loss function
Your training or validation datasets may change
You modify data augmentation

Which tools or processes do you use to track modifications you have made and how did they affected the model ?

Maybe you are looking for a combination of a version control system (like git and Github) and a tool like comet.ml. In the past, I used comet.ml to keep track of different experiments performed with different hyper-parameters or different versions of the code. There are other alternatives to comet.ml, such as sacred, but they may also have different features and may not be as visually pleasing as comet.ml or even free. Personally, I liked comet.ml (even though, at the time, it still lacked some features). In any case, a VCS, like git, is widely used in software development (not just in AI projects) to keep track of different versions of the code, etc. You may also be interested in continuous integration (e.g. Travis CI) and code review (e.g. codacy) tools.


If $h_i$ are consistent and admissible, are their sum, maximum, minimum and average also consistent and admissible? Consider the following question:

$n$ vehicles occupy squares $(1, 1)$ through $(n, 1)$ (i.e., the bottom row) of an $n \times n$ grid. The vehicles must be moved to the top row but in reverse order; so the vehicle $i$ that starts in $(i, 1)$ must end up in $(n − i + 1, n)$. On each time step, every one of the $n$ vehicles can move one square up, down, left, or right, or stay put; but if a vehicle stays put, one other adjacent vehicle (but not more than one) can hop over it. Two vehicles cannot occupy the same square.

Suppose that each heuristic function $h_i$ is both admissible and consistent. Now what I want to know is to check the admissibility and consistency of the following heuristics:

$h= \Sigma_i h_i$

$h= \min_i (h_i)$

$h= \max_i (h_i)$

$h = \frac{\Sigma_i h_i}{n}$

P.S: As a lemma, we know that consistency implies the admissibility of the heuristic function.
Problem Explanation
From this link, I have found that the first heuristic is neither admissible, nor consistent.
I know that the second and the fourth heuristics are either consistent, or admissible.
I have faced with one contradiction in the third heuristic:

Here we see that if car 3 hops twice, the total cost of moving all the cars to their destinations is 3, whereas the heuristic $\max(h_1, \dots, h_n) = 4$.
Problem
So, $\max(h_1, ..., h_n)$ must be consistent and admissible, but the above example shows that it's not. What is my mistake?

The issue is that you must include assumptions about hopping into your heuristic. In particular, if you are considering individual cars then you must assume that they might be able to hop all of the way to the goal. Thus, your heuristic for each car should be Manhattan distance divided by 2. That's guaranteed to be admissible when you take the max.

If you consider all possible cars you can do better, but you'll need to reason out all the cases. (In general every car either waits or moves, and for every waiting car one car can hop. So, by looking at the minimum distance for any car to reach the goal you can start to reduce your heuristic.) But, that is a different question.


How to actually teach the ANN the resulting weights of different training inputs? I thought I have implemented the code (from scratch, no library) for an artificial neural network (first endeavour in the field). But I feel like I miss something very basic or obvious.

To make it short: code works for a single pair of in-/out-values but fails for sets of value pairs. I do not really understand the training process. So I want to get this issue out of the way first. The following is my improvised training (aka all that I can think of) in pseudocode.

trainingData = [{in: [0,0], out:[0]}, {in: [0,1], out:[0]}, ...];
iterations = 10000

network = graphNodesToNetwork()
links = graphLinksToNetwork()
randomiseLinkWeights(links)

while(trainingData not empty) {
  for(0<iterations) {
     set = trainingData.pop()

     updateInput(network, set.in)

     forwardPropagate(network, links)

     linkUpdate = backPropagate(network, links, set.out)

     updateLinks(linkUpdate, links)}
}

Is this how it is supposed to work? Do you feed in your training data set by set (while-loop)?

Edit 1: because my final comment did distract from the issue at hand.

Edit 2: less wordy, more code-y

Your network must have something which persist like weights add bias

Your new implementation would be like this :

trainingData = [{in: [0,0], out:[0]}, {in: [0,1], out:[0]}, ...];
iterations = 10000

network = graphNodesToNetwork()
links = graphLinksToNetwork()
randomiseLinkWeights(links)
weights = []

while(trainingData not empty) {
    for(0<iterations) {
       set = trainingData.pop()

       weights = updateInput(network, set.in)

       forwardPropagate(network, links)

       linkUpdate = backPropagate(network, links, set.out, weights)

       updateLinks(linkUpdate, links)}
}

Inshort retain weights, and backprapogate.


What is the difference between sensitivity analysis and parameter tuning? I tried different values of genetic algorithm operators:

many crossover rates from 20% to 80%
many crossover rates from 1% to 20%
varying the population size

The study of different parameter values is called quantitative parameter tuning or sensitivity analysis. What is the difference between the two terms?

I would generally assume that parameter tuning is the process of finding the combination of hyperparameters (e.g., population size, crossover and mutation operators and rates, etc.) that yield the best performance on your problem. When you're thinking about the way that performance varies with parameter choice, this is the "what". What is the best choice of parameters.
Sensitivity analysis is the "how" or "how much". If I change my crossover rate from 0.9 to 1.0, how significant is the change in performance of my algorithm. Is my performance more or less stable across a wide range of choices (good) or is it highly dependent on finding this one little peak in the parameter space and every other choice is much worse (bad).


What is the difference between learning without forgetting and transfer learning? I would like to incrementally train my model with my current dataset and I asked this question on Github, which is what I'm using SSD MobileNet v1.
Someone there told me about learning without forgetting. I'm now confused between learning without forgetting and transfer learning. How they differ from each other?
My initial problem, what I'm trying to achieve (mentioned in Github issue) is the following.
I have trained my dataset on ssd_mobilenet_v1_coco model. I'm getting continuous incremental data. Right now, my dataset is very limited.
What I want to achieve is incremental training, i.e. as soon as I get new data, I can further train my already trained model and I don't have to retrain everything:

Save trained model $M_t$
Get new data $D_{t+1}$
Train $M_t$ on $D_{t+1}$ to produce $M_{t+1}$
Let $t = t+1$, then go back to $1$

How do I perform this incremental training/learning? Should I use LwF or transfer learning?

Learning without Forgetting (LwF) is an incremental learning (sometimes also called continual or lifelong learning) technique for neural networks, which is a machine learning technique that attempts to avoid catastrophic forgetting. There are several incremental learning approaches. LwF is an incremental learning approach based on the concept of regularization. In section 3.2 of the paper Continual lifelong learning with neural networks: A review (2019), by Parisi et al., other regularisation-based continual learning techniques are described.

LwF could be seen as a combination of distillation networks and fine-tuning, which refers to the re-training with a low learning rate (which is a very rudimentary technique to avoid catastrophically forgetting the previously learned knowledge) an already trained model $\mathcal{M}$ with new and (usually) more specific dataset, $\mathcal{D}_{\text{new}}$, with respect to the dataset, $\mathcal{D}_{\text{old}}$, with which you originally trained the given model $\mathcal{M}$.

LwF, as opposed to other continual learning techniques, only uses the new data, so it assumes that past data (used to pre-train the network) is unavailable. The paper Learning without Forgetting goes into the details of the technique and it also describes the concepts of feature extraction, fine tuning and multitask learning, which are related to incremental learning techniques.

What is the difference between LwF and transfer learning? LwF is a combination of distillation networks and fine-tuning, which is a transfer learning technique, which is a special case of incremental learning, where the old and new tasks are different, while, in general, in incremental learning, the old and new tasks can also be the same (which is called domain adaptation).


Where can I find pre-trained language models in English and German? Where can I find (more) pre-trained language models? I am especially interested in neural network-based models for English and German.

I am aware only of Language Model on One Billion Word Benchmark and TF-LM: TensorFlow-based Language Modeling Toolkit.

I am surprised not to find a greater wealth of models for different frameworks and languages.

Of course now there has been a huge development:
Huggingface published pytorch-transformers, a library for the so successful Transformer models (BERT and its variants, GPT-2, XLNet, etc.), including many pretrained (mostly English or multilingual) models (docs here). It also includes one German BERT model. SpaCy offers a convenient wrapper (blog post).

Update: Now, Salesforce published the English model CTRL, which allows for use of "control codes" that influence the style, genre and content of the generated text.

For completeness, here is the old, now less relevant version of my answer:

Since I posed the question, I found this pretrained German language model:
https://lernapparat.de/german-lm/

It is an instance of a 3-layer "averaged stochastic descent weight-dropped" LSTM which was implemented based on an implementation by Salesforce.


Are the labels updated during training in the algorithm presented in "An algorithm for correcting mislabeled data"? I am trying to understand an algorithm for correcting mislabeled data in the paper An algorithm for correcting mislabeled data (2001) by Xinchuan Zeng et al. The authors are suggesting to update the output class probability vector using the formula in equation 4 and class label in equation 5.

I am wondering:

Are they updating labels while training, starting from very first back-propagation?
It seems like if we train on the same data and then predict labels on the same data, it would be the same as what the authors are suggesting. Does it make sense or I misunderstood?

I think that making some draws might help.

Below I tried to draw the model architecture. We start with classic feed-forward structure: input represented by a vector I with length f (number of features), a hidden layer H which does not have a fixed size, and output O of length c (number of classes). Then we have 3 extra vectors than usual:
a vector U they refer as input (a bit confusing I have to say), and two vectors V and P representing classes probabilities. All these vectors have length c.

P is what we want to learn, new class probabilities for each instance that should correct wrong labels of misclassified instances in the initial dataset.
So in some sense the aim of the work is not to train a model to make predictions but rather train a model to clean a training dataset. I think it's important to stress out this point because we talk about training but there is actually no test after the training, we just end with a modified training dataset with some instances relabelled.
The relabelling depends on the learning of V rather than P (because as I will say also later on, the final arrow that goes from V to P is the identity function). V depends on U that also depends on V, again it sounds a bit confusing, the tricks rely in the initialisation.

In the second picture I just copied some formulas from the paper. They don't specifically say how they get the vector P in the first place, but I treat it simply as given because we need it to initialise U by applying the inverse sigmoid function, element-wise. We need also to define the hyper parameter D, the initial probability value for the true class V$_y$ in the initial dataset. In the paper they set it to .95. Using D we can initialise the vector V.

Once we initialise U and V we have all the elements to iterate over the dataset.

A first thing to notice is that they don't use back propagation to update U,V and P, they just define some updating rules (which require to define some other initial parameters, $\textit{u$_0$}$ and L$_p$). I want to stress out again that the mapping between V and P is just the identity function, I guess that they define it just to avoid confusion, because in the paper P appear as an input element, but nevertheless is also the output we want to learn. Back propagation is used only to update the weights in H.

ANSWERS TO THE QUESTIONS:

So we can finally say that, regarding your first question, the answer is yes, they do start updating the classes probability vectors P from the very first iteration, even though is not clear how they initialise them (or from where they got them).

Regarding the second question instead I would say that no, this is definitely not as training a model with fixed label and then making predictions over the training dataset again. The whole point is that similar training instances will be initialised with the same P, and they will also produce similar U and V vectors. For misclassified instances, U will be updated with larger changes, because of the different produced O. For example an image of a 2 will generate a different output than an image of a 5, this would be reflected first on U and then on V and P . If training with fixed label instead you would just force the weight in the hidden layer to learn a function that treat the representation of a 2 and a 5 in a similar way, leading to low accuracy cause you would be telling the model "Hey, the straight line in the 5 is important also to recognise a 2".

I have to say that I never read about this dataset cleaning approach but it is interesting, and their results show that the cleaned version of the dataset lead to better performances, which is interesting because machine learning analysis usually gives for granted the correctness of the labels.

Hope this is of help in some ways!


Can two planning PDDL actions be taken simultaneously? We are discussing planning algorithms currently, and the question is to describe the steps to check if actions could be taken simultaneously. This is a really open-ended question so I'm not sure where to start.

First place to look is how the preconditions/effects of different actions interact.


Algorithm that creates new images based on other images Are there any open sourced algorithms that can take a couple of images as an input and generate a new, similar image based on that input? Or are there any resources where I can learn to create such an algorithm?

I'm not an expert on that so you could probably get a better answer.

I'm not sure to understand what you're looking for. Are the couple of images about the same thing? Like pictures of cats and you want to generate a new cat based on these pictures? If that's what you want, you could probably take a look at Generative Adversarial Network (GAN) : Introduction.
A GAN is made up of a Generator and a Discriminator. The goal of the discriminator is to distinguish the real data from the generated data. And the goal of the generator is to improve its generated data to look similar to real data. Then, if there are different cat images in your dataset, the generator will learn to create a new cat based on that dataset.

If what you're looking for is to take different images like a cat and a dog and generate a "catdog", you can take a look at Variational AutoEncoder (VAE). For example you can train two different VAE (Encoder/Decoder). One for cats, and one for dogs. Then you take the encoder of dogs and the decoder of cats. That what I saw one day, not sure if it really works.

Correct me if I'm wrong


Generate credit cards dataset for locating number region Currently I'm working on a project for scanning credit card and text extraction from cards. So first of all I decided to preprocess my images with some filters like thresholding, dilation and some other stuff. But it was not successfully for OCR of every credit cards. So I learned a lot and I found a solution like this for number plate recognition that is very similar to my project.
In the first step I want to generate a random dataset like my cards to locate card number region, and for every card that I've generated I cropped two images that one of them has numbers and another has not. I generated 2000 images for every cards.

so I have some images like this:

 (does not have numbers)
 (has numbers)

And after generating my dataset I used this model with tensorflow to train my network.

    model = models.Sequential()
    model.add(layers.Conv2D(8, (5, 5), padding='same', activation='relu', input_shape=(30, 300, 3)))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Conv2D(16, (5, 5), padding='same', activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Conv2D(32, (5, 5), padding='same', activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Flatten())
    model.add(layers.Dense(512, activation='relu'))
    model.add(layers.Dense(2, activation='softmax'))

Here is my plot for 5 epochs.

I almost get 99.5% of accuracy and It seems to be wrong, I think I have kind of overfitting in my data. Does it work correctly or my model is overfitted ? And how can I generate dataset for this purpose ?

I am assuming the question you are asking is how to prevent over-fitting on the maximum accuracy. Your graph does show that your model over-fits.

There is a couple of different methods to prevent over-fitting from happening. You can specify training to stop after a certain amount of epochs. In your case it seems to be 2 or 3 epochs. Take care as a new initialization might require more or less epochs to reach optimal accuracy thus it is required run your model a couple of times to determine the correct amount of epochs.

You can also specify the accuracy you want your model to reach before you stop training, this can be dangerous as a local-minima could be found below your expected accuracy thus resulting in your training to run indefinitely. Also your model could have become more accurate then your expected accuracy but stopped prematurely due to the model reaching your stopping-condition.

You can combine the two. Terminate training once it reach your expected accuracy otherwise terminate it if it reaches a certain amount of epochs. This way you prevent your training from happening indefinitely.

In Tensorflow you would want to build your own custom training seen here

The method you are interested is

epoch_accuracy = tfe.metrics.Accuracy()

In their example they get the accuracy after each epoch, you would want to create your own batches and apply it there.

If you really want to dive deeper, you can implement methods for the model to detect when it starts over-fitting such as looking at the standard deviation of the accuracy across a certain measure (usually epochs but your model starts over-fitting after 5 so its not a good measure in your case, maybe do standard deviation over a certain amount of batches trained) if the accuracy leaves the standard deviation space then it might be good idea to stop training. The problem with this technique is that your model has to over-fit a bit to know it needs to stop. The other problem is that your model might seem it is over-fitting but was just a dip in accuracy and was stopped prematurely. (There is methods around this but can't recall them)

Another technique is to add drop out. This introduce noise and pushes your model out of a local-minima. Also forces nodes that was ignored to be used and optimized. This surprisingly works well but is not a surefire way to prevent over-fitting. This technique is built into Tensorflow and don't need custom training. You can find more reading about Tensorflow dropout here, also they have more techniques on how to overcome over-fitting but their solutions is to change the model and your model seems to be fine.

tf.keras.layers.Dropout(0.2),

These are just some techniques I listed but there is a lot more out there. Unfortunately there is no concrete method to prevent it but steps can be taken to stop well before over-fitting occurs. The first two techniques are mostly used together.


Is it a good idea to overfit on a small part of your data for faster model convergence? I working on a classification problem that needs to detect patterns on a time serie. Basically, there's a catch-all class that means "no pattern detected", the other are for the specific patterns. The data is imbalanced (ratio 1/10 at least), but I adapted the class weights.

I'm able to overfit successfully on a few days of data, but when I train on 2 years of data, the model seems stuck on class1 "no pattern detected" for a veeeery long time. I've tried several  learning rates, but it doesn't make the convergence happen significatively faster.

Is it a better starting point for my training to use the overfitting model's weight as a starting point? Could this allow the model to converge faster?

The first thing which you first have to understand is that does your trained model is working efficiently with both the training and testing data. If yes then its not overfitting. There is only one case where over-fitting doesn't matter and that is when your testing data is same as training data and dealing with is very important or it could be disastrous. Therefore, to avoid overfitting you can try several methods and they are followings:

Optimization Algorithms like sgd, Adam, RMSprop, Adagard, are few gradient descents.
Loss function like Hinge loss could be useful.
Parameter initialization greatly influence the optimization process.

Overall, Large datasets are better for building ML models but there are some methods there is an interesting article that i would recommend you to read: Article Link

I Hope this Helps!!!


Where to publish replications of machine learning papers? If I replicated the results of a machine learning paper, is there a website that I can publish my code to that collects informal replication attempts?

https://paperswithcode.com/ allows anyone to publish implementations, but I am looking for something similar that allows anyone to publish implementations along with replicated experimental results. Does such a website exist?

OpenReview (https://openreview.net/) may be what you're looking for. It's both a platform for ML conferences to organize their reviewing process as well as an arXiv-like preprint repository.

One major machine learning conference (ICLR) has been holding "reproducibility challenges," in which participants can submit replication reports of existing papers, but that probably only applies to papers submitted to that conference.

There is a general preprints section, which might be your best bet.

Disclaimer: I used to be on the OpenReview development team


What are the societal risks associated with AI? What are the actual risks to society associated with the widespread use of AI? Outside of the use of AI in a military context.

I am not talking about accidental risks or unintentional behaviour - eg, a driver-less car accidentally crashing.

And I am not talking about any transitional effects when we see the use of AI being widespread and popular.  For instance I have heard that the widespread use of AI will make many existing jobs redundant, putting many people out of work.  However this is true of any major leap forward in technology (for example the motor car killed off the stable/farrier industries).  The leaps forward in technology almost always end up creating more jobs than were lost in the long run.

I am interested in long term risks and adverse effects stemming directly from the widespread use of AI in a non-military sense.  Has anybody speculated on the social or psychological impacts that AI will produce once it has become popular?

The biggest risk is algorithmic bias. As more and more decision-making processes are taken on by AI systems, there will be an abdication of responsibility to the computer; people in charge will simply claim the computer did it, and they cannot change it.

The real problem is that training data for machine learning often contains bias, which is usually ignored or not recognised. There was a story on BBC Radio about someone whose passport photo was rejected by an algorithm because he supposedly had his mouth open. However, he belonged to an ethnic group which has larger lips than Caucasian whites, but the machine could not cope with that.

There is a whole raft of examples where similar things happen: if you belong to a minority group, machine learning can lead to you being excluded, just because the algorithms will have been trained on training data that was too restricted.

Update: Here is a link to a BBC News story about the example I mentioned.


How can I use 1-channel images as input to a CNN? I need to develop a convolutional neural network whose inputs are 1-channel images, but I dont know how to do it, given that most libraries use 3 channel images. Should I convert my images to RGB? Is there any way to implement a CNN that receive as input 1-channel images?

The libraries should allow you to specify the number of input channels of the convolutional layer, so no one should prevent you from passing 1-channel images as input to a CNN. For example, in PyTorch, you can specify the number of input channels of the Conv2d object.

If your library does not provide such feature, you could convert your 1-channel images to 3-channel images, where e.g. all 3 channels of each image are equal to the only channel of the corresponding original 1-channel image.


Can we achieve what a CNN can do with just a normal neural network? When I was learning about neural networks, I saw that a complex neural network can understand the MNIST dataset and a simple convolution network can also understand the same. So I would like to know if we can achieve a CNN's functionality with just using a simple neural network without the convolution layer and if we can then how to convert a CNN into an ANN.

The convolutional aspect of a CNN comes purely from the connections between layers. Instead of a fully-connected network, which can be difficult to train and tends to overfit more, the convolutional network utilizes hierarchical patterns in the data to limit the number of connections - a local edge detection feature in an image analysis network, for example, only needs input from a small number of local pixels, not the entire image. But in principle, you could assign weights to a fully-connected network to perfectly mimic a convolutional one - you just set the weights of the unneeded connections to zero. Because a general ANN has all the connections present in a CNN plus more, it can do anything a CNN can do plus more, although the training can be more difficult.


Neural network cost function implementation I am implementing neural network to train hand written digits. Following is the cost function,

In log(1-(h(x)), if h(x) is 1, then it would result in log(1-1) (i.e) log(0).
So im getting math error.

Im initializing the weights randomly between 10-60. Im not sure where i have to change and what i have to change! Thanks for your help.

So, firstly, for h _{Theta}(x) to be 1, the weighted sum of x (after you dot product it with Theta) would have to be literally infinity, if you're using the sigmoid function. Doesn't happen in practice, even with the rounding computers do, as we don't use big numbers to initialize our Theta matrices.

Intuitively, that'd mean you're basically more certain than one can possibly be in this universe that the label of this example should be '1'.

So if (1 - h_{Theta}(x)) = 0, 'y' is certainly 1, and so 1-y will be zero.

Secondly, as the comments note, the convention is to drop the entire right-hand-side term when y^(i) is 1. This will not cause problems when programming, due to the first point I made above.


Is there an alternative to RNNs that doesn't require knowing input history? In an RNN to train it, you need to roll it out, and enter in the history of inputs and the history of expected outcomes.

This doesn't seem like a realistic picture of the brain since this would require, for example, for the brain to store a perfect history of every sense that comes in to it for many time-steps.

So is there an alternative to RNNs that doesn't require this history? Perhaps storing differences or something? Or storing some accumulator?

Perhaps there is a way to calculate with RNNs that doesn't require keeping hold of this history?

You don't necessarily need to roll out the inputs to an RNN; doing so makes it easier to optimize computation (if the sequence length is the same each batch), but it's not a necessity. Furthermore, RNNs (and, incidentally, the brain) doesn't necessarily remember the input history as is; rather, the history is encoded via the RNN's cell state (or states, in the case of LSTMs and other RNN cell architectures with multiple states). Neural Turing Machines (NMTs) and Differential neural computers expand on that concept by also using a larger "memory" storage (in the form of a matrix).


What's the right way of building a deep Q-network? I'm new to RL and to deep q-learning and I have a simple question about the architecture of the neural network to use in an environment with a continous state space a discrete action space.

I tought that the action $a_t$ should have been included as an input of the neural network, togheter with the state. It also made sense to me as when you have to compute the argmax or the max w.r.t. $a_t$ it was like a "standard" function. Then I've seen some examples of networks that had as inputs only $s_t$ and that had as many outputs as the number of possible actions. I quite understand the logic behind this (replicate the q-values pairs of action-state) but is it  really the correct way? If so, how do you compute the $argmax$ or the $max$? Do I have to associate to each output an action?

Do I have to associate to each output an action?

You are absolutely correct! In DQN, each output node of the neural network will be associated with one of your possible actions (assuming a finite discrete action space). After passing an input through the network, the value of each output node is the estimated q-value of the corresponding action. One benefit of this architecture is that you only need to pass the input through the neural network once to compute the q-value of each action, which is constant in the number of actions. If you were to include an action as an input to the neural network along with an observation, then you would need to pass an input for each action, which scales linearly in the number of actions. This is mentioned in paragraph 2 of Section 4.1 in the original DQN paper (https://arxiv.org/abs/1312.5602)

  Is it really the correct way? If so, how do you compute the argmax or the max?

It's one possible way that is used in many popular algorithms such as DQN. To find the argmax, you simply take the action corresponding to the output node with highest q-value after passing an input through the network.


Do self-driving cars resort to randomness to make decisions? I recently heard someone make a statement that when you're designing a self-driving car, you're not building a car but really a computerized driver, so you're trying to model a human mind -- at least the part of the human mind that can drive.

Since humans are unpredictable, or rather since their actions depend on so many factors some of which are going to remain unexplained for a long time, how would a self-driving car reflect that, if they do?

A dose of unpredictability could have its uses. If, say, two self-driving cars are in a stuck in a right of way deadlock, it could be good to inject some randomness instead of maybe seeing the same action applied at the same time if the cars run the same system.

But, on the other hand, we know that non-deterministic isn't friends with software development, especially in testing. How would engineers be able to control it and reason about it?

Driving Priorities

When considering the kind of modeling needed to create reliable and safe autonomous vehicles, the following driving safety and efficacy criteria should be considered, listed in priority with the most important first.

The safety of those inside the vehicle and outside the vehicle
Reduction of wear on passengers
The safety of property
The arrival at the given destination
Reduction of wear on the vehicle
Thrift in fuel resources
Fairness to other vehicles
The thrift in time

These are ordered in a way that makes civic and global sense, but they are not the priorities exhibited by human drivers.

Copy Humans or Reevaluate and Design from Scratch?

Whoever said that the goal of autonomous car design is to model the portions of a human mind that can drive should not be designing autonomous cars for actual manufacture.  It is well known that most humans, although they may have heard of the following safety tips, cannot bring them into consciousness with sufficient speed to benefit from them in actual driving arrangements.

When the tires slip sideways, steer into the skid.
When a forward skid starts, pump the breaks.
If someone is headed tangentially into your car's rear, immediately accelerate and then break.
On an on ramp, accelerate to match the speed of the cars in the lane into which you merge, unless there is no space to merge.
If you see a patch of ice, steer straight and neither accelerate nor decelerate once you reach it.

Many collisions between locomotives and cars are because a red light causes a line in multiple lanes across the tracks.  Frequently, a person will move onto the railroad tracks to gain one car's length on the other cars.  When others move to make undoing that choice problematic, a serious risk emerges.

As absurd as this behavior is to anyone watching, many deaths occur as a fast traveling 2,000 ton locomotive hits what feels like a dust speck to the train passengers.

Predictability and Adaptability

Humans are unpredictable, as the question indicates, but although adaptability may be unpredictable, unpredictability may not be adaptive.  It is adaptability that is needed, and it is needed in five main ways.

Adaptive in the moment to surprises
Adaptive through general driving experience
Adaptive to the specific car
Adaptive to passenger expression
Adaptive to particular map regions

In addition, driving a car is

Highly mechanical,
Visual,
Auditory,
Plan oriented
Geographical, and
Preemptive in surprise situations.

Modelling Driving Complexities

This requires a model or models comprise of several kinds of objects.

Maps
The vehicle
The passenger intentions
Other vehicle
Other obstructions
Pedestrians
Animals
Crossings
Traffic signals
Road signs
Road side

Neither Mystery nor Indeterminance

Although these models are cognitively approximated in the human brain, how well they are modeled and how effective those models are at reaching something close to a reasonable balance of the above priorities varies from driver to driver, and varies from trip to trip for the same driver.

However, as complex as driving is, it is not mysterious.  Each of the above models are easy to consider at a high level in terms of how they interact and what mechanical and probabilistic properties they have.  Detailing these is an enormous task, and making the system work reliably is a significant engineering challenge, in addition to the training question.

Inevitability of Achievement

Regardless of the complexity, because of the economics involved and the fact that it is largely a problem of mechanics, probability, and pattern recognition, it will be done, and it will eventually be done well.

When it is, as unlikely as this sounds to the person who accepts our current culture as permanent, human driving may become illegal in this century in some jurisdictions.  Any traffic analyst can mount heaps of evidence that most humans are ill equipped to drive a machine that weighs a ton at common speeds.  The licensing of unprofessional drivers has only become widely accepted because of public insistence on transportation convenience and comfort and because the workforce economy requires it.

Autonomous cars may reflect the best of human capabilities, but they will likely far surpass them because, although the objects in the model are complex, they are largely predictable, with the notable exception of children playing.  AV technology will use the standard solution for this.  The entire scenario can be brought into slow motion to adapt for children playing simply by slowing way down.  AI components that specifically detect children and dogs are likely to emerge soon, if they do not already exist.

Randomness

Randomness is important in training.  For instance, a race car driver will deliberately create skids of various types to get used to how to control them. In machine learning we see some pseudo random perturbations introduced during training to ensure that the gradient descent process does not get caught in a local minimum but rather is more likely to find a global minimum (optimum).

Deadlock

The question is correct in stating that, "A dose of unpredictability could have its uses."  The deadlock scenario is an interesting one, but is unlikely to occur as standards develop.  When four drivers come to a stop sign at the same time, they really don't.  It only seems like they did.  The likelihood that none of them arrived more than a millisecond before the others is astronomically small.

People will not detect (or even be honest enough) to distinguish these small time differences, so it usually comes to who is most gracious to wave the others on, and there can be some deadlock there too, which can become comical, especially since all of them really wish to get moving.  Autonomous vehicles will extremely rarely encounter a deadlock that is not covered by the rule book the government licensing entity publishes, which can be programmed as driving rules into the system.

On those rare occasions, the vehicles could digitally draw lots, as suggested, which is one place where unpredictability is adaptive.  Doing skid experimentation like a race car driver on Main Street at midnight may be what some drunk teen might do, but that is a form of unpredictability that is not adaptive toward a sensible ordering of the priorities of driving.  Neither would be texting or trying to eat and drive.

Determinism

Regarding determinism, in the context of the uses discussed, pseudo-random number generation of particular distributions will suffice.

Deadlock release or
Training speed-ups and improved reliability when there are local minima that are not the global minimum during optimization,

Functional tests and unit testing technologies are not only able to handle the testing of components with pseudo-randomness, but they sometimes employ pseudo-randomness to provide better testing coverage.  The key to doing this well is understanding of probability and statistics, and some engineers and AI designers understand it well.

Element of Surprise

Where randomness is most important in AV technology is not in the decision making but in the surprises.  That is the bleeding edge of that engineering work today.  How can one drive safely when a completely new scenario appears in the audio or visual channels?  This is perhaps the place where the diversity of human thought may be best adept, but at highway speeds, it is usually too slow to react in the way we see in movie chase scenes.

Correlation Between Risk and Speed

This brings up an interesting interaction of risk factors.  It is assumed that higher speeds are more dangerous, the actual mechanics and probability are not that clear cut.  Low speeds produce temporally longer trips and higher traffic densities.  Some forms of accidents are less likely at higher speeds, specifically ones that are related mostly to either traffic density or happenstance.  Other forms are more likely at higher speeds, specifically ones that are related to reaction time and tire friction.

With autonomous vehicles, tire slippage may be more accurately modeled and reaction time may be orders of magnitude faster, so minimum speed limits may be more imposed and upper limits may increase once we get humans out of the driver's seats.


How to study the correlation between GAN's input vector and output image A generative adversarial network (GAN) takes a vector of numbers as input and generates an image, based on the input. Each element of the vector causes some feature of the image to change, but the mapping between input and output is not clear, as often happens in deep learning. What is the best way to study the correlation between the vector elements and the output image features? The first approach that comes to mind is to manually change every element and check the result, however I am not sure that this is the best solution.

Let me first provide a brief introduction first

GANs as VAEs are generative models which means they learn exactly what you described: to map a typically small dimensional vector/tensor into a higher dimensional one, which in your case is (interpreted as) an image.

These 2 generative models differ in the actual learning strategy which results typically in making GAN outperform VAE in realistic image generation (but it’s better not to get into the reasons for this here)

However in principle the Generator Network in GAN works in a similar way to the Decoder Network in VAE so as their output space is interpreted as an image, you can interpret their input space as a sort of semantic space hence the values you are setting define the semantic of the output image (in fact you observe the features change)

It is possible to say the generative model learns a  Semantic to Appearance Mapping hence it works the opposite way of the classifier model which learns an Appearance to Semantic Mapping

Unfortunately as it typically happens in NN this mapping are definitely hard to understand for humans, nevertheless this is an active area of research

So after this introduction, aimed at giving you an essential understanding of the problem, I can suggest you some papers to get into more details like this one

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

In case you found it not that clear, I’m planning to write a summary of this paper and publish it on my Medium so you could check it there (I’ll update this answer or add a comment to notify about it)


How does the MCTS tree look like? I have come across the Monte Carlo tree search (MCTS) algorithm, but I can't find what the tree should look like. For example, does it still represent a minimax process, i.e. player 1 from the root has its child nodes as probabilities of moves, then from those child nodes, the next move is player 2, etc.? Is this how the tree looks like, so when we backpropagate we update only the winning player nodes?

In a MCTS (Monte Carlo tree search) the resulting tree is expanded by two policies:

tree-policy (selects next and unknown child nodes by a exploration-exploitation tradeoff)
default policy (selects known child nodes till terminal state e.g. with random uniform moves).

The term terminal state is often considered to be the end a tree. If you consider designing an agent with MCTS that is able to play chess, then the terminal state would a win, lose or draw. One game like this is called a playout (sometimes also roll-out). The main concept of MCTS relies on repeating this playouts many times and updating the tree according to the scores. The tree is built up in an incremental and asymmetric manner.

Imagine designing a chess-playing agent. The tree, or rather how the tree is built up can be like this:

Root Node: Starting with an empty board
Child Nodes: Board configuration after each move (opponent and agent)
Selection of Child Nodes: If the opponent is a human, then the selection of child nodes is performed by his/her own strategy. The agent is selecting moves according to a policy.
Terminal State: Score of the game (win/lose/draw).

So after each move (opponent or agent) the current child node/root node is expanded by the resulting child nodes.
If you want to dive deeper into MCTS, then I would recommend to take a look at A Survey of Monte Carlo Tree Search Methods.


Should I ignore the actions RIGHTFIRE and LEFTFIRE in the SpaceInvaders environment? I'm trying to replicate the DeepMind DQN paper. I'm using OpenAI's Gym. I'm trying to get a decent score with Space Invaders (using SpaceInvaders-v4 environment). I checked the actions available with env.unwrapped.get_action_meanings(), and I get this:
['NOOP', 'FIRE', 'RIGHT', 'LEFT', 'RIGHTFIRE', 'LEFTFIRE']

Checking the number of actions with env.action_space.n gives me a number of 6 actions.
The RIGHTFIRE and LEFTFIRE actions, I suppose, aren't used, given that they seem to do the same as LEFT and RIGHT, am I right?
If so, restricting the action size to the 4 first actions would improve my learning?

I found that we shouldn't eliminate those actions, because RIGHTFIRE and LEFTFIRE do RIGHT + FIRE and LEFT + FIRE, respectively. So they are different actions. I post this just to clarify in case someone faces the same doubt.


What are the purposes of autoencoders? Autoencoders are neural networks that learn a compressed representation of the input in order to later reconstruct it, so they can be used for dimensionality reduction. They are composed of an encoder and a decoder (which can be separate neural networks). Dimensionality reduction can be useful in order to deal with or attenuate the issues related to the curse of dimensionality, where data becomes sparse and it is more difficult to obtain "statistical significance". So, autoencoders (and algorithms like PCA) can be used to deal with the curse of dimensionality.

Why do we care about dimensionality reduction specifically using autoencoders? Why can't we simply use PCA, if the purpose is dimensionality reduction?

Why do we need to decompress the latent representation of the input if we just want to perform dimensionality reduction, or why do we need the decoder part in an autoencoder? What are the use cases? In general, why do we need to compress the input to later decompress it? Wouldn't it be better to just use the original input (to start with)?

It is important to think about what sort of patterns in the data are being represented.

Suppose that you have a dataset of greyscale images, such that every image is a uniform intensity. As a human brain you'd realise that every element in this dataset can be described in terms of a single numeric parameter, which is that intensity value. This is something that PCA would work fine for, because each of the dimensions (we can think of each pixel as a different dimension) is perfectly linearly correlated.

Suppose instead that you have a dataset of black and white 128x128px bitmap images of centred circles. As a human brain you'd quickly realise that every element in this dataset can be fully described by a single numeric parameter, which is the radius of the circle. That is a very impressive level of reduction from 16384 binary dimensions, and perhaps more importantly it's a semantically meaningful property of the data. However, PCA probably won't be able to find that pattern.

Your question was "Why can't we simply use PCA, if the purpose is dimensionality reduction?" The simple answer is that PCA is the simplest tool for dimensionality reduction, but it can miss a lot of relationships that more powerful techniques such as autoencoders might find.


Can supervised learning be recast as reinforcement learning problem? Let's assume that there is a sequence of pairs $(x_i, y_i), (x_{i+1}, y_{i+1}), \dots$ of observations and corresponding labels. Let's also assume that the $x$ is considered as independent variable and $y$ is considered as the variable that depends on $x$. So, in supervised learning, one wants to learn the function $y=f(x)$.

Can reinforcement learning be used to learn $f$ (possibly, even learning the symbolic form of $f(x)$)?

Just some sketches how can it be done: $x_i$ can be considered as the environment and each $x_i$ defines some set of possible "actions" - possible symbolic form of $f(x)$ or possible numerical values of parameters for $f(x)$ (if the symbolic form is fized). And concrete selected action/functional form $f(x, a)$ (a - set of parameters) can be assigned reward from the loss function: how close the observation $(x_i, y_i)$ is to the value that can be inferred from $f(x)$.

Are there ideas or works of RL along the framework that I provided in the previous passage?

Any supervised learning (SL) problem can be cast as an equivalent reinforcement learning (RL) one.
Suppose you have the training dataset $\mathcal{D} = \{ (x_i, y_i \}_{i=1}^N$, where $x_i$ is an observation and $y_i$ the corresponding label. Then let $x_i$ be a state and let $f(x_i) = \hat{y}_i$, where $f$ is your (current) model, be an action. So, the predicted label of observation $x_i$ corresponds to the action taken in state $x_i$. The reward received after having taken action $f(x_i)$ in state $x_i$ can then be defined as the loss $|f(x_i) - y_i|$ (or any other suitable loss).
The minimization of this loss is then equivalent to the maximization of the (expected) reward. Therefore, in theory, you could use trajectories of the form $$T=\{(x_1, f(x_1), |f(x_1) - y_1|),  \dots, (x_N, f(x_N), |f(x_N) - y_N|)\}$$ to learn a value function $q$ (for example, with Q-learning) or a policy $\pi$, which then, given a new state $x_{\text{new}}$ (an observation) produces an action $f(x_{\text{new}})$ (the predicted label).
However, note that the learned policy might not be able to generalise to observations not present in the training dataset. Moreover, although it is possible to solve an SL problem as an RL problem, in practice, this may not be the most appropriate approach (i.e. it may be inefficient).
For more details, read the paper Reinforcement Learning and its Relationship to Supervised Learning (2004) by Barto and Dietterich, who give a good overview of supervised and reinforcement learning and their relationship. The paper Learning to predict by the methods of temporal differences (1988), by Richard Sutton, should also give you an overview of reinforcement learning from a supervised learning perspective. However, note that this does not mean that a reinforcement learning problem can be cast as an equivalent supervised learning one. See section 1.3.3 Converting Reinforcement Learning to Supervised Learning of the mentioned paper Reinforcement Learning and its Relationship to Supervised Learning for more details.
Reinforcement learning can thus be used for classification and regression tasks. See, for example, Reinforcement Learning for Visual Object Detection (2016) by Mathe et al.


How to (theorically) build a neural network with input of size 0? Say I want to train a NN that generates outputs of some sort (say, even numbers).
Note that the network does not classify outputs, but, rather GENERATES the output.

I want let it run forward and generate some number, then either give it a positive reward of 1 for an even number, and a reward of -1 for an odd number, to make i output only even numbers over time.

What would be an architecture for such a NN?

I am getting caught in the part where here is actually no input, and I can't really start with a hidden layer, can I?

I am quite confused and would appreciate guidance

Whether a neural network has learned anything or not, it is a function that maps some input to an output. Training is the process of tweaking the weights so that the output is something that we want. Thus there is always in input of some sorts.

The problem you have presented, of generating even numbers, is much like a Generative Adversarial Network (GAN). In a GAN there are 2 networks: a Generator that tries to generate a sample from a target distribution and a Discriminator that tries to tell real samples from fake samples. The classic analogy being a criminal making counterfeit money and a copy trying to tell what is real money or not.

The generator input is usually a random number (or a matrix of random numbers). The generator then learns to transform a particular random input to a particular point in the target space.

So to answer your question, no there can't be a neural network with 0 inputs as there must always be an input of some kind. Even if the network was to generate a sequence instead of one instance, it would still need something to start with.

For your example, there would have to be some input for the  network to start with. A really simple NN that could solve your problem might look like:

_input = [RandomInteger()]
neuralNetworkWeights = [2]
result = _input * neuralNetworkWeights
result is always even


Would a new human-like general artificial intelligence be more similar, in terms of eduction, to a toddler or an adult human? The naive concept of a general AI, or strong AI, or artificial general intelligence, is some kind of software that can answer questions like

  What is the volume of a cube that is 1 m wide?

or even

  Why are there only two political parties in the US?

The second question requires external knowledge and high-level reasoning. For example that US means USA in the context, the constitution and that having two parties is caused by the mathematical properties of the election system.

But I would expect that newborn human child is intelligent in the sense of intelligence that is used in general artificial intelligence, but toddlers can not answer these questions.

That is not because an infant is not intelligent, but because it is not educated, I think.

What is the apparent level of education of an artificial intelligence that could be called human-like?

A human-level AI should be able to learn and behave in the same way that humans learn and behave, otherwise, we shouldn't be calling it a human-level AI. So, either if it starts with more or less knowledge of a baby human, it should be able to learn similarly to a human and acquire more knowledge with experience.

  What is the apparent level of education of an artificial intelligence that could be called human-like?

To answer you question more directly, the level of education of a human-like AI (even newly created ones) can potentially be variable (i.e. different human-like AIs could potentially have different levels of education), but the human-like AI will necessarily need to be able to increase its level of education and, in general, knowledge (because humans can also do this).

Moreover, note that AGI is not necessarily restricted to human-level AI. So, other AGIs may not follow the same principles of humans.


Q learning tic tac toe I have a tic-tac-toe with a Q-learning algorithm, and the AI plays against the same algorithm (but they don't share the same Q matrix). But after 200,000 games, I still beat the AI very easily and it's rather dumb.
My selection is made by epsilon greedy policy.

What could cause the AI not to learn?

[EDIT]
Here is how I do it (pseudo code):

for(int i = 0; i < 200000; ++i){
    //Game is restarted here
    ticTacToe.play();
}

And in my ticTacToe I have a simple loop :

while(!isFinished()){
    swapPlaying(); //Change the players' turn
    Position toPlay = playing.whereToMove();

    applyPosition(toPlay);
    playing.update(toPlay);
}

//Here I just update my players whether they won, draw or lost.

In my players, I select the move with epsilon-greedy implemented sa below :

Moves moves = getMoves(); // Return every move available
Qvalues qValues = getQValues(moves); // return only qvalues of interest
//also create the state and add it to the Q-matrix if not already in.

if(!optimal) {
     updateEpsilon(); //I update epsilon with simple linear function epsilon = 1/k, with k being the number of games played.
     double r = (double) rand() / RAND_MAX; // Random between 0 and 1
     if(r < epsilon) { //Exploration
         return randomMove(moves); // Selection of a random move among every move available.
     }
     else {
         return moveWithMaxQValue(qValues);
     }
} else { // If I'm not in the training part anymore
     return moveWithMaxQValue(qValues);
  }

And I update with the following :

double reward = getReward() // Return 1 if game won, -1 if game lost, 0 otherwise
double thisQ, maxQ, newQ;
Grid prevGrid = Grid(*grid); //I have a shared_ptr on the grid for simplicity
prevGrid.removeAt(position) // We remove the action executed before

string state = stateToString(prevGrid);
thisQ = qTable[state][action];
mawQ = maxQValues();

newQ = thisQ + alpha * (reward + gamma*maxQ - thisQ);
qTable[state][action] = newQ;

As mentioned above, both AI have the same algorithm, but they are two distinct instances so they don't have the same Q-matrix.
I read somewhere on Stack Overflow that I should take in account the movement of the opposite player, but I update a state after player move and opponent move so I don't think it's necessary.

I couldn't add a comment because of my low reputation but you can check this. It is about the state space. https://math.stackexchange.com/questions/485752/tictactoe-state-space-choose-calculation