Is it possible to flip the features and labels after training a model? The goal of this program is to predict a game outcome given a game-reference-id, which is a serial number like so:

id,totalGreen,totalBlue,totalRed,totalYellow,sumNumberOnGreen,sumNumberOnBlue,sumNumberOnRed,sumNumberOnYellow,gameReferenceId,createdAt,updatedAt
1,1,3,2,0,33,27,41,0,1963886,2020-08-07 20:27:49,2020-08-07 20:27:49
2,1,4,1,0,36,110,31,0,1963887,2020-08-07 20:28:37,2020-08-07 20:28:37
3,1,3,2,0,6,33,83,0,1963888,2020-08-07 20:29:27,2020-08-07 20:29:27
4,2,2,2,0,45,58,44,0,1963889,2020-08-07 20:30:17,2020-08-07 20:30:17
5,0,2,4,0,0,55,82,0,1963890,2020-08-07 20:31:07,2020-08-07 20:31:07
6,2,4,0,0,36,116,0,0,1963891,2020-08-07 20:31:57,2020-08-07 20:31:57
7,3,2,1,0,93,16,40,0,1963892,2020-08-07 20:32:47,2020-08-07 20:32:47

Here's the link for a full training dataset.
After training the model, it becomes difficult to use the model to predict the game output, since the game-reference-id is the only independent column, while others are random.
Is there a way to flip the features with the labels during prediction?

The body of your post seems to be asking a completely separate question than the title of your post, so I will answer both:
"Body: How do I complete the goal of this program?"
Your dataset does not have the dependent variable, which is the outcome of the game (win/loss/draw). What I am assuming is that you have a way of looking up the outcome of the game from either the "id" field or the "gameReferenceId" field.
So you would have to augment the dataset with a new column, "gameOutcome", which has values (win/loss/draw), by looking up the outcome of each game (each row) and adding that to your dataset.
Once you have this, you have the 12 independent variables (the 12 columns already there), and the 1 dependent variable (the "gameOutcome"), and the prediction task should be straightforward from there.
"Title: Given a label, how do I predict features?"
(note: this section will not help your program)
What you are looking for are generative models. Generative models can generate instances given the label and/or some random seed. This is a completely different model than discriminative models, which given the instance predicts the label (this is the one you have).
The simplest generative model is a Naive Bayes model. While Naive Bayes is normally used as a discriminative model (to classify a label), it has enough information to generate instances given a label as well. Here are some tips on how to turn Naive Bayes into a generative model.
If you are looking for generative deep neural network models, this blog by OpenAI has a nice overview explaining the three modern approaches (generative adversarial networks, variational autoencoders, and autoregressive models). The blog explains at a high level what the strengths and weaknesses are of each approach, and links to some open source projects for you to try it out as well.


Why isn't my decision tree classifier able to solve the XOR problem properly? I was trying to solve an XOR problem, and the dataset seems like the one in the image.

I plotted the tree and got this result:

As I understand, the tree should have depth 2 and four leaves. The first comparison is annoying, because it is close to the right x border (0.887). I've tried other parameterizations, but the same result persists.

I used the code below:

from sklearn.tree import DecisionTreeClassifier

clf = DecisionTreeClassifier(criterion='gini')
clf = clf.fit(X, y)

fn=['V1','V2']

fig, axes = plt.subplots(nrows = 1,ncols = 1,figsize = (3,3), dpi=300)

tree.plot_tree(clf, feature_names = fn, class_names=['1', '2'], filled = True);

I would be grateful if anyone can help me to clarify this issue.

I can reproduce this problem for an even more easily separable dataset:

The ideal tree for it should be as follows:

However, when I run DecisionTreeClassifier with the maximal depth = 2 in scikit-learn many times, it splits the dataset randomly and never gets it right.
This is an example of 4 different runs:

The problem is that scikit-learn has only two measures of the quality of a split: gini, and entropy. Both of them estimate mutual information between the target and only one predictor.
However, in XOR problem, mutual information of each predictor with the target is zero. You can read more about it here: link from which you can see that this problem exists not only for XOR but for any task where interaction between features is important.
In order to solve it, the tree should be built based neither on the Gini impurity, nor on the information gain but on measures that estimate how the target depends on multiple features, e.g. multivariate mutual information,  distance correlation, etc which might solve simple problems like XOR but might fail in the case of real tasks. It is easy to find simple cases when they fail (just try them for a regression for simple non-linear functions of a few variables). There is no such a measure that would estimate the dependence of a target on multiple interacting predictors very well and would work for all problems.
EDIT to answer Asher's comment: I did several runs for max_depth=3. It is better than for max_depth=2 but still misses the correct classification from time to time. Taking max_depth=4 almost always gets XOR correctly with the occasional misses. Below are pictures of some runs for max_depth=3 and max_depth=4.

However, the trees for max_depth=3 and max_depth=4 become ugly. They are ugly not only because they are bigger than the ideal tree shown above but they totally obscure the XOR function. For example, can you decipher an XOR from this tree?

It is probably possible with some pruning technique but still, an extra work.


In reinforcement learning, does the optimal value correspond to performing the best action in a given state? I am confused about the definition of the optimal value ($V^*$) and optimal action-value (Q*) in reinforcement learning, so I need some clarification, because some blogs I read on Medium and GitHub are inconsistent with the literature.
Originally, I thought the optimal action value, $Q^*$, represents you performing the action that maximizes your current reward, and then acting optimally thereafter.
And the optimal value, $V^*$, being the average $Q$ values in that state.  Meaning that if you're in this state, the average "goodness" is this.
For example:
If I am in a toy store and I can buy a pencil, yo-yo, or Lego.
Q(toy store, pencil) = -10
Q(toy store, yo-yo) = 5
Q(toy store, Lego) = 50

And therefore my $Q^* = 50$
But my $V^*$ in this case is:
V* = -10 + 5 + 50 / 3 = 15

Representing no matter what action I take, the average future projected reward is $15$.
And for the advantage of learning, my baseline would be $15$.  So anything less than $0$ is worse than average and anything above $0$ is better than average.
However, now I am reading about how $V^*$ actually assumes the optimal action in a given state, meaning $V^*$ would be 50 in the above case.
I am wondering which definition is correct.

I am wondering which definition is correct.

The asterisk * in both the definitions stands for "optimal" in the sense of "value when following the optimal policy"
So this one is correct:

$V^*$ actually assumes the optimal action in a given state, meaning $V^*$ would be $50$ in the above case

However, you have got the definition of Q slightly wrong.
I think this is because you are omitting the parameters.
The state value function uses the state as a parameter, $V_{\pi}(s)$, it returns the value of being in state $s$ and following a fixed policy $\pi$. The * is used to denote following an optimal policy.
The action value function has two parameters - a state and an action that is possible in that state, $Q_{\pi}(s, a)$, it returns the value of being in state $s$, taking action $a$ (regardless of whether it is the best action or not) and following the policy $\pi$ after that point.
Your assertion in the question:

And therefore my $Q^* = 50$

is wrong, or rather not meaningful, as you have not stated the parameters. You already list all the possible values of Q with the parameters. You could say $\text{max}_a Q(\text{toy store}, a) = 50$, or to choose the best action $\pi(\text{toy store}) = \text{argmax}_a Q(\text{toy store}, a)
= \text{Lego}$


Can we combat against deepfakes? I came across 'Amber'(https://ambervideo.co/) where they are claiming that they have trained their AI to find patterns emerging due to artificially created videos which are invisible to naked eye.

I am wondering that the people who are creating deepfakes can as well their AI's to remove these imperfections and so the problem reduces to 'cat-mouse' game where having more resources(to train their AI) is more crucial.

I do not work in AI and vision and so I may be missing some trivial points in the area. I would really appreciate if detailed explanation or relevant resources are given.

Edit: Most of the people who do manipulate the media news or create fake news could afford more resources than an average citizen. So, is the future is really going to be dark where only few strong have even more control on the society than today?

I mean even though there are fake photos created by photo shop, most of the good photo-shopped photos do take a long time to make. But if AIs can be trained to do that then it is more about having large resources. Are there related works which give hope to know real from fakes?

P.S.: I realize that after the edit, the question also went tangential to the topic-tags here. Please let me if there are relevant tags.

I think this game will go pretty crazy, because, at some point, the generator AI will be able to generate absolutely perfect images. Actually, no, just perfect enough that no AI can be sure whether they are real or fake.

So, I think the AI war will go onto more than the image, the detector AI will probably evolve to analyze whether this video is logically plausible, for example, by tracking the celebrities' position to prove that it is impossible that he/she was, for example, let's put it this way, being unloyal to his/her partner.

I mean, currently, AI can tell whether an image is fake or not better than human because it has seen about a million times more samples than us, but if we know who the person in the image is and we are as stalky as the AI I just described, we can probably work out that this image is implausible.

Of course, there will be counter measurements to that. But, at that point, we might as well just let the AI rule the world, given that it will have become this smart (lol).

But, seriously, if it's smart enough to think this far ahead in this 'real world' problem, then strong AI is nigh.


Similarity score between 2 words using Pre-trained BERT using Pytorch I'm trying to compare Glove, Fasttext, Bert on the basis of similarity between 2 words using Pre-trained Models. Glove and Fasttext had pre-trained models that could easily be used with gensim word2vec in python.

Does BERT have any such models?
Is it possible to check the similarity between two words using BERT?

1. Does BERT have any such models?
If you mean the pretrained model, then the answer is YES.
BERT has Cased and Uncased models and also models for other language (BERT-Base, Multilingual Cased and BERT-Base, Multilingual Uncased, trained in 104 languages). You can check those models here.
2. Is it possible to check similarity between two words using BERT?
Refer to this Google Colab Notebook, it is not appopriate with BERT Embeddings for Word-Level similarity comparisons.
However, doing sentence embedding similarity comparisons are still valid with BERT.
That's it, hope it helps you :)


Can I use an autoencoder with high latent representational space? I am trying to use a neural network to predict the next state output given the current state and action pairs. Both input and outputs are continuous variables. Due to the high dimensionality of each input, ( ~50 dimensional input ) and 48 dimensional output, I am not able to achieve an achieve a satisfiable enough accuracy.

I am thinking of using an auto-encoder to learn a latent representation of the state. Would a latent representation from an auto-encoder help to improve the prediction accuracy ? and can the latent representation have a higher dimensional space compared to the original state ?

i feel , i can answer this question, based the web source that i found and i read. that can use an autoencoder with hight latent representational space. example use LSTM autoencoder, LTSM autoencoder used for sequence or time series data. on the source, they use CNN autoencoder to denoise some syntetic noised data they have generated. but they have asked what is the meaning of this latent representation space? for what they have done before. on that answer. they said"your input data is noisy sinewave data. your are not supposed to use Convolutional autoencoder for sequence data".

https://stackoverflow.com/questions/59438488/cnn-autoencoder-latent-space-representation-meaning

addition other source for answer this question
https://towardsdatascience.com/understanding-latent-space-in-machine-learning-de5a7c687d8d
https://towardsdatascience.com/deep-inside-autoencoders-7e41f319999f

hopefully help answer :))


How would you intuitively but rigorously explain what the VC dimension is? The VC dimension is a very important concept in computational/statistical learning theory. However, the first time you read its definition, you may not immediately understand what it really represents or means, as it involves other concepts, such as shattering, hypothesis class, learning and sets. For example, let's take a look at the definition given by Shai Shalev-Shwartz and Shai Ben-David (p. 70)

DEFINITION $6.5$ (VC-dimension) The VC-dimension of a hypothesis class $\mathcal{H}$, denoted $\operatorname{VCdim}(\mathcal{H})$, is the maximal size of a set $C \subset \mathcal{X}$ that can be shattered by $\mathcal{H}$. If $\mathcal{H}$ can shatter sets of arbitrarily large size we say that $\mathcal{H}$ has infinite VC-dimension.

Without knowing what a hypothesis class is, or what the specific $C$, $X$ and $H$ in this definition are, it's difficult to understand this definition. Even if you are familiar with what a hypothesis class is (i.e. a set of sets, i.e. our set of functions/hypotheses/models, e.g. the set of all possible neural networks with a specific topology) and you know that $C$ and $X$ are sets of input points, it should still not be clear what the VC dimesion really is or represents.
So, how would you intuitively and rigorously explain the exact definition of the VC dimension?
Note that I am not asking for answers like

The VC dimension represents the complexity (or expressive power, richness, or flexibility) of your model/hypothesis class.

Of course, this is easy to memorize, but it's quite vague. So, I am not looking for vague/general answers. I am looking for answers that rigorously but intuitively describe the mathematical definition of the VC dimension. For example, you could provide an illustration that shows what the VC dimension is, and, in your example (e.g. the XOR problem cannot be solved by a set of lines), you can describe what $H$, $C$, and $X$ are, and how they relate to the typical concepts you will find an introductory course to machine learning, but you should not forget to describe the concept of shattering. If you have other ideas of how to illustrate this concept memnomically, feel free to provide an answer.

Shattered set. First we need a concept of a shattered set. I'll work from a shattered set example in Wikipedia adjusting it to your notation.
The statement that $\mathcal{H}$ shatters $C$ means that for every subset $A \subset C$ there is a set $B\in\mathcal{H}$ such that $B$ "separates" $A$ from $C \backslash A$. Writing this formally:
$$\text{shatters}(\mathcal{H},C) = \forall A \subset C\; \exists B\in\mathcal{H}\;.\;A = B\,\cap A $$
As an example consider the set $C$ of four points on $\mathcal{X} = \mathbb{R}^2$:
$$C = \left\{(0,0); (0,1); (1,0); (1,1)\right\}$$
And the classification class $\mathcal{H}$ being all possible 2D discs on $\mathcal{X} =\mathbb{R}^2$. (Notice that people use the world "class" here, because we are dealing with "set-of-sets" stuff and that might get tricky). Note that a disk can be represented as the set of all points inside the circle.
Now, it turns out that this $\mathcal{H}$ doesn't shatter $C$. The counterexample would be the subset $A$ of "diagonal" points:
$$A = \left\{(0,0); (1,1)\right\} \subset C$$
There is no 2D disk $B\in\mathcal{H}$ (in the context of learning theory, an element/set of the hypothesis class is a hypothesis, which can also be viewed as a function) that satisfies $A = B\,\cap A$.  Intuitively, this means that you cannot use a 2D disk to classify your pair of points $A$ from the rest of the set $C$.
The VC dimension of $\mathcal{H}$ is the maximal cardinality $d$ of the set $C$ that it can shatter. For $d=3$, we can provide tree points $C'$ for which we can easily find all discs that cover all eight possible subsets of $C'$:
$$C' = \left\{(0,0); (0,1); (1,0)\right\}$$
Above we've shown that for a particular set $C$ of $d=4$ points $\text{shatters}(\mathcal{H}, C)$ is false. But we need to prove that it is false for all 4-point sets. To prove this, we  consider a convex hull of an arbitrary set of four points $C = \left\{a;b;c;d\right\}$. In general position, the convex hull is either a triangle or a quad:

In the case of a triangle, we choose the outermost three points as a counterexample set $A$. So, with this (labeling) configuration, you cannot find a disk that covers (i.e. classifies) the outermost three points correctly while excluding the point inside the triangle. In the case of a quad, we choose the pair on the longest diagonal. If any three points lay on a single line, then we choose the pair of outermost points.
This sketches a proof that no $d=4$ point set can be shattered by $\mathcal{H}$, but we've shown that there is a $d=3$ set $C'$ that can be. Concluding that $VCdim(\mathcal{H}) = 3$
Another example is considered on the next page (pg. 71) of the book you've referenced. It again considers 2D plane  $\mathcal{X} =\mathbb{R}^2$ and the classification class $\mathcal{H}$ is all possible axis-aligned rectangles. Authors show a configuration $C$ of four points that can be shattered by $\mathcal{H}$ on the left of figure 6.1. And then provide a proof that no $d=5$ points can be shattered by axis-aligned rectangles. Concluding that VC dimension of their  $\mathcal{H}$ is four.
Hope these examples help. (BTW, note that, it seems that Deep Learning has quite bad VC dimension but it still works somehow - which is rather puzzling).


What is the best variant of darknet to use? pjreddie's official darknet version (link from official website here) has been forked several times. In particular I've come accross AlexeyAB's fork through this tutorial. I assume the tutorial's author used AlexeyAB's fork because he wanted to use it on a Windows machine, which pjreddie's darknet cannot do AFAIK.

I am not really concerned about that (I am a linux user), but I am very interested about the half precision option (CUDNN_HALF) that AlexeyAB's darknet has, whereas pjreddie's darknet does not. Of course I've checked that this option was handled by the graphic card (RTX2080) we use at my office.

Nevertheless, I wonder: how stable/robust is that fork? Of course I want high-performing software, but I also want a certain level of stability! On the other hand, the latest commit on pjreddie's darknet is back from September 2018 (ie 1 year old), whereas AxeleyAB's darknet is active…

More broadly, there seems to be a lot of darknet forks: which ones to prefer?

What does the neural network community think?

I used Trieu's Darkflow and it was trained well up to around 30 to 50 epoch with maximum 30 GB dataset. At the same time, it was quite often crashed after these epochs.

Half precision...I don't remember Darkflow have that.

One way to estimate each fork reputation is by number of Star and Issues, for example Alexey's Darknet 5,000 and 2,500 and Darkflow 5,000, 539 respectively.

Reference: OS Ubuntu 16.04


What are the main ideas behind NER? Named entity recognition (NER), also known as entity chunking/extraction, is a popular technique used in information extraction to identify and segment the named entities and classify or categorize them under various predefined classes.

Briefly, how does NER work? What are the main ideas behind it? And which algorithms are used to perform NER?

There are different algorithms, each with their advantages and disadvantages.

Gazetteers: these have lists of the entities to be recognised, eg list of countries, cities, people, companies, whatever is required. They typically use a fuzzy matching algorithm to capture cases where the entity is not written in exactly the same way as in the list. For example, USA or U.S.A., United States, United States of America, US of A, etc. Advantage: generally good precision, ie can identify known entities. Disadvantage: can only find known entities
Contextual Clues: here you have patterns that you find in the text, eg [PERSON], chairman of [COMPANY]. In this case, sentences like Jeff Bezos, chairman of Amazon, will match, even if you have never come across either Bezos or Amazon. Advantage: can find entities you didn't know about. Disadvantages: could end up with false positives, might be quite labour-intensive to come up with patterns; patterns depend on the domain (newspapers vs textbooks vs novels etc_
Structural description: this is basically a 'grammar' describing what your entities look like, eg (in some kind of pseudo-regex): title? [first_name|initial] [middle_name|initial]? surname would match "Mr J. R. Ewing" or "Bob Smith". Similar descriptions could match typical company names; you'd probably still need lists of possible surnames or firstnames. Advantages: some flexibility, and potentially good precision. Disadvantages: patterns need to be developed and maintained.

Ideally you would want to combine all three approaches for a hybrid one to get the advantages of recognising unknown entities while keeping excess false positives in check.

There might also be other machine-learning approaches, but I'm not too familiar with those. The main problem is that they are hard to fine-tune or work out why they do what they do.

UPDATE: A good starting point would be to use a gazetteer-based approach to annotate some training data, and use that to identify contextual patterns. You can then use that data to train a machine learning approach (see OmG's answer on CRF) to broaden the approach; and then you add newly recognised entities to your list.

Ideally you would want to have a gazette as your main database to avoid false positives, and use machine-learning or contextual patterns to just capture previously unseen entities.


What is the type of problem requiring to rate images on a scale? I'm new to the topic, but I've used some off the shelf knowledge about computer vision for classifying images.

For example, you can easily generate labels that can determine whether or not e.g. a cloud is in the image. However, what is the general type of problem called where you want to assign a value, or rate the image on a scale - in this example, the degree of cloudiness in the image?

What are useful algorithms or techniques for addressing this type of problem?

The main distinction between tasks is 'classification' vs 'regression'. In classification you would try to identify the presence of a cloud or not in an image, if you want to predict the level of 'cloudness' with continuous values you are then performing a regression task.

I'm not aware about state-of-the models specific for images, but you can potentially use whatever architecture you desire to perform regression, CNN, RNN, the only thing to pay attention to is the loss function you will use. There are specific functions for classifications (which usually use the argmax function to turn probabilities into labels) and for regression (the most used is the Mean Squared Error, the model try to approximate the continuous values directly).

For a quick overview of loss functions I suggest this easy tutorial 5 Regression Loss Functions. Hope it might be of use.


What is the proof that the branch and bound algorithm always finds optimal path in a graph? I've been studying Branch and Bound's graph algorithm and I hear it always finds the optimal path because it uses previously found solutions to find others

However, I haven't been able to find a proof of why it finds the optimal path. (In fact, most sites kind of do a bad job generalizing the algorithm itself.)

What is the proof that this algorithm always finds the optimal path in the case of a graph with 1 or more goal nodes?

Branch and Bound is similar to an exhaustive search, except it incorporates a method for computing lower bounds on branches. If the lower bound on a given branch is greater than the upper bound on the problem (i.e. the current best solution encountered), that branch can be discarded since it will never produce an optimal solution.

Hence, since you explore all options except those you know will produce values less optimal than your current best value, you are guaranteed to encounter the global optimum.

Note this is a generic algorithm, and you will need to reference a specific implementation if you want proof of why it satisfies these criteria.


When should I use Reinforcement Learning vs PID Control? When designing solutions to problems such as the Lunar Lander on OpenAIGym, Reinforcement Learning is a tempting means of giving the agent adequate action control so as to successfully land.

But what are the instances in which control system algorithms, such as PID controllers, would do just an adequate job as, if not better than, Reinforcement Learning?

Questions such as this one do a great job at addressing the theory of this question, but do little to address the practical component.

As an Artificial Intelligence engineer, what elements of a problem domain should suggest to me that a PID controller is insufficient to solve a problem, and a Reinforcement Learning algorithm should instead be used (or vice versa)?

I think the comments are basically on the right track.

PID controllers are useful for finding optimal policies in continuous dynamical systems, and often these domains are also used as benchmarks for RL, precisely because there is an easily derived optimal policy. However, in practice, you'd obviously prefer a PID controller for any domain in which you can easily design one: the controller's behaviors are well understood, while RL solutions are often difficult to interpret.

Where RL shines is in tasks where we know what good behaviour looks like (i.e., we know the reward function), and we know what sensor inputs look like (i.e. we can completely and accurately describe a given state numerically), but we have little or no idea what we actually want the agent to do to achieve those rewards.

Here's a good example:

If I wanted to make an agent to maneuver a plane from in front of an enemy plane with known movement patterns to behind it, using the least amount of fuel, I'd much prefer to use a PID controller.
If I wanted to make an agent to control a plane and shoot down an enemy plane with enough fuel left to land, but without a formal description of how the enemy plane might attack (perhaps a human expert will pilot it in simulations against our agent), I'd much prefer RL.


Nonbinary and binary values in input tensor In my input tensor, I would like to use both integer values as well as booleans. For example, if there is a spelling difference between 2 texts, I want to set the value to true, and otherwise false. In the same tensor, I would like to assign a value to, for example, the maximum number of consecutive messages, which will be an integer.

Am I allowed to use 0's and 1's for the booleans together with integers, or will it have any negative impact on the working of the network? The ANN wont see any difference between the binary and nonbinary values, but is it a problem?

If the two inputs for the binary values are held to the domain set $\{0, 1\}$ during training, testing, and use, it will not break the network functionality, although the question is a good one. Why use so many bits to hold one?

Theory of Holors: A Generalization of Tensors, by Parry Moon and Domina Spencer (ISBNs 978-0521019002 and 0521019001), propose holors precisely because of this kind of limitation on the natural heterogeneity of numerical structure. Gibbs, Ricci, Einstein, and others used holors in their mathematical expressions but without the name holor. Holors are are conceptually related to objects in ontology and classes in object oriented design, but they are numbers, so they fit into mathematical expressions as scalars, vectors, matrices, vector fields, and other tensors do.

This is the basis for Coplien's operator overload in C++ and one of the reasons the language is too mutable to be mastered by average applications programmers. Support for what Moon and Spenser called holors is also the basis of decoupled type safety in some of the early LISP object oriented frameworks.

The current programming paradigm we seen in Java, Python, and other popular languages was set in FORTRAN, which was more attainable for average programmers, so we are stuck with homogeneous structure, even in AI libraries and GPUs. Therefore, once one number requires an IEEE 64 float, bytes and bits will use all 64.

In artificial networks, this only matters in the very first multiplication with a parameter.


What does deep learning offer with respect to standard machine learning? I've been reading a lot about DL. I can understand to an extent how it works, in theory at least, and how it's technically different from conventional ML. But what I'm looking for is more of a "conceptual" meaning.

Let's say you're designing a self-learning system, why would you choose DL? What are the main performance advantages that DL offers? Is it the accuracy, speed, power efficiency, a mix of all of them?

Deep learning allows you to solve complex problems without necessarily being able to specify the important "features" or key input variables for the model in advance.

To give an example, a problem that may be easily tackled without deep learning could be predicting the frequency and claim amounts of insurance vehicle claims,  given historical claim data that may include various attributes of the policy holder and their vehicle.  In this example, the "features" to be specified in the model are the known attributes of policy holder and vehicle.  The model will then attempt to utilise these features to make predictions.

On the other hand,  facial recognition is a problem more suited to deep learning algorithms.  This is because it is difficult to manually identify what combinations of pixels may be important features to include in a conventional machine learning model.  A multi-layered neural network however has the potential to identify/create the important features itself, which may include for example eyes, nose and mouth, and then utilise these features to recognise faces and other objects.


Do correlations matter when building neural networks? I am new to working with neural networks. However, I have built some linear regression models in the past. My question is, is it worth looking for features with a correlation to my target variable as I would normally do in a linear regression or is it better to feed the neural network with all the data I have?
Assuming that the data I have is all related to my target variable of course. I am working with this dataset and building a neural network regressor for it.
https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DL0101EN/labs/data/concrete_data.csv
Here is a snippet of the data. The target variable is the concrete strength rate given a certain combination of materials for that concrete sample.

I greatly appreciate any tips and explanations. I excuse me if this is too noob of a question but unfortunately I did not find any info about it on google. Thanks again!

If there is some correlation between features, that is what the network will ideally find out on its own and learn to utilize. So, in general, don't take correlated samples or features out of the training loop only because they look correlated. After all, they could convey a lot of valuable information.
When it comes to correlation between data samples during training, this correlation is commonly broken up by training a network on randomly selected mini-batches of training data samples. So, you randomly sample e.g. 16 or 32 (or so) training examples based on which you apply a single update of the weights using some Stochastic Gradient Descent variant. Since the members of a mini-batch are sampled at random, chances for finding highly correlated training samples in some mini-batch shall be sufficiently minimized in order not to negatively affect the training outcome.
Having said that, if you are concerned about overfitting of your model or weights that would overly weight just a small subset of all available input features, you could try applying regularization techniques like L1 (encouraging sparse representations) or L2 (encouraging low weights in general) regularization or dropout.
In your particular case, since the main concern is an excessive contribution of only a small set of input features, L2 shall yield better results (avoiding excessively large weights that would be required to excessively much weight just a small number of features).
Besides that: Commonly, you split your training dataset into 3 parts:

Data used for fitting the model (actual training data)
Data used for assessing the training progress & possibly for determining when to apply early stopping (validation data)
Test data used to assess the performance of the system after all training & intermediate testing is done

The final evaluation on the test dataset shall reveal then the generalization ability of your trained model to novel data.
So, with regularization in place during training and relatively low error rates on the validation and test datasets, you are pretty much save even without checking for correlated data beforehand. Only when you really struggle decreasing the validation loss, it might be worth to further inspect what exactly is going wrong in terms of correlations and such.


What is the definition of the "cost" function in the SVM's objective function? In a course that I am attending, the cost function of a support vector machine is given by
$$J(\theta)=\sum_{i=1}^{m} y^{(i)} \operatorname{cost}_{1}\left(\theta^{T} x^{(i)}\right)+\left(1-y^{(i)}\right) \operatorname{cost}_{0}\left(\theta^{T} x^{(i)}\right)+\frac{\lambda}{2} \sum_{j=1}^{n} \Theta_{j}^{2}$$
where $\operatorname{cost}_{1}$ and $\operatorname{cost}_{0}$ look like this (in Magenta):

What are the values of the functions $\operatorname{cost}_{1}$ and $\operatorname{cost}_{0}$?
For example, if using logistic regression the values of $\operatorname{cost}_{1}$ and $\operatorname{cost}_{0}$ would be $-\log* \operatorname{sigmoid}(-z)$ and $-\log*(1-\operatorname{sigmoid}(-z))$.

That is the hinge loss, a type of loss most notably used for SVM classification. The hinge loss is typically defined as:
$$
\ell(y)=\max (0,1-t \cdot y),
$$
which, in your use case, is something like this:
$$
\operatorname{cost}\left(h_{\theta}(x), y\right)=\left\{\begin{array}{ll}
\max \left(0,1-\theta^{T} x\right) & \text { if } y=1 \\
\max \left(0,1+\theta^{T} x\right) & \text { if } y=0
\end{array}\right.
$$
Check this article on it.


What is the correct way to read and analyse images in machine learning? I am trying to understand the best practice to read and analyze images. If your image has 10,000 pixels, your input layers will have 10,000 inputs?

It sounds that my neural network will have too many inputs if I do it that way. Is that a problem? What is the recommended way of feeding an image through a neural network?

If you are using a fully connected network (aka an MLP) and images with one channel (grey scale) and 100 x 100 = 10,000 pixels, then yes, MLP would have 10k inputs and 10k x N 1 trainable weights in the first layer (as noted by Neil Slater). If you have a color image with 3 channels, e.g. RGB, then you can expect 3 times as many weights because there are 3 times as many values used to represent the image.

A convolutional neural network is a common architecture for analyzing images. For a 100x100 (10k) pixel image, the input layer might have 3x3x1x32 = 288 weights (for 1 channel) or 3x3x3x32 = 864 weights in the first layer, much less than 10k x N 1 from a fully connected network. This would transform your image into a 98x98x32 size image. The main point is that you would have 3x3 weights per input channel per output channel at each layer, instead of 10k weights per input channel per output channel. CNNs also give you some invariance properties that are usually nice in machine learning with images.

For images in general, having a lot of weights is normal. ImageNet (linked above) has 60 million weights to train. Typically special hardware, like a GPU is used to handle this many weights. If you are using just a CPU, your model may not train well in any reasonable amount of time, i.e. years.


Getting Negative Information Gain (using Entropy as a measure of disorder) I'm creating a Decision Tree and at the very root level itself, I'm getting negative Information Gain.

As per my knowledge, Information Gain is always > 0....

Any explanation....please....

Please look at the node below.....

    [69+,42-]
     /   \
    /     \
[56+,33-]  [11+,11-]

IG = H([69+, 42-]) - H([56+,33-],[11+,11-])

   = {(-69/111 * lg(69/111) -42/111 * lg(42/111))
      - 89/111 * ( -56/89*lg(56/89) -33/89*lg(33/89))
      - 22/111 * ( -11/22*lg(11/22) -11/22*lg(11/22))}

   = -0.004

It turns out that for every feature, the IG < 0.

What should I do to decide the feature at the root node?

Thanks

It is not possible to have a negative IG, your outcome is negative because of a computational mistake: initially you have 69 positive instances, and after creating 2 children the sum drops to 56+11=67. Set is therefore less pure -> IG is negative.


Why are inhibitory connections often used in virtual neural networks when they don't seem to exist in real life neural networks? Something I like about neural network AI is that we already have a blueprint for it - we know in great detail how different types of neurons work, and we don't have to invent the AI, we can just replicate what we already know works (neural networks) in a simplified way and train it.

So what's confusing me right now is why many popular neural models I've seen when studying neural networks include both inhibitory and stimulating connections. In real neural networks, from my understanding, there is no negative signal being transferred, rather the signals sent between neurons are comparable to values between 0.1 and 1. There's no mechanic for an inverse (inhibiting) signal to be sent.

For example, in this network (seen overlaying a snake being simulated), the red lines represent inhibiting connections, and the blue lines represent stimulating neurons:

Is this just an inconsequential detail of neural network design, where there's really no significant difference between a range of 0 to 1 and a range of -1 to 1? Or is there a reason that connections in our simulated neural networks benefit from being able to a express a range from -1 to 1 rather than just 0 to 1?

I think you are confused. The reason why some neural neural networks have neurons with an output in range (-1, 1) all depends on the activation function used. Some networks even have neurons with an output range of (-Infinity, +Infinity) ( aka the identity function ).

I advise you to take a look at this list: activation functions w/ ranges.

Nowadays a lot of neural networks have a mixture of different activation functions. E.g. combining ReLU with Sigmoid or TanH.

This neural network library even shows that for most simple problems, neural networks prefer a variety of activation functions.

The red lines just mean that that connection has a low value in comparison with other connections.

There is no rule of thumb for activation functions, some problems benefit from a certain function, some don't.


How would one disambiguate between two meanings of the same word in a sentence? The boy lifted the bat and hit the ball.

In the above sentence, the noun "bat" means the wooden stick. It does not mean bat, the flying mammal, which is also a noun. Using NLP libraries to find the noun version of the definition would still be ambiguous.
How would one go about writing an algorithm that gets the exact definition, given a word, and the sentence it is used in?
I was thinking you could use word2vec, then use autoextend https://arxiv.org/pdf/1507.01127.pdf to differentiate between 2 different lexemes e.g. bat (animal) and bat (wooden stick).
Then the closest cosine distance between the dictionary definition and any of the words of the sentence might indicate the correct definition.
Does this sound correct?

I'd suggest BERT for this. It is essentially a word-embedding model that uses at local context to determine the appropriate embedding for each word. This means it would assign "bat" a different embedding in a sentence containing "hit the ball" vs. a sentence containing "flies and eats bugs". On top of that, Google has released a number of pre-trained versions of BERT, which can be used directly without additional training (depending on your task of course). BERT as a service is great if you just want embeddings. The Python transformers library makes it exceedingly simple to incorporate BERT into your task-specific model.


Can an AI distinguish between good and bad according to people living in a restricted geographical area? Would people go far with Artificial Intelligence and machine learning to the point where machines could learn during a long period of time to distinguish what's 'good' from 'bad' according to people living in a restricted geographical area, and then the machines take control and turn what was learnt into a set of 'rules' and 'laws' (think of it as an effective machine of 'politics') that match the majority of the people's view of issues.

That should be accepted by everyone, since a contract set at the beginning says: "Everyone is ok".

Even if you could do that (which I believe is a long way off), what would be the point?

If I understand you correctly, you want an AI system to learn through observation of human beings what their 'rules of interaction' are. It sees a person killing someone else, and then that person is punished by the community, so the AI learns that killing people is not right. However, that is already codified in laws... So what it would pick up are social behaviours, which hopefully be things like "be nice to other people", "don't do anybody any harm", "be honest and truthful", etc.

The first question then is how an AI could assess events[*] as to whether they are 'good' or 'bad'. If everybody lies and steals, that would be learnt as normal behaviour, and the AI would not be able to pick up that most people would see this as 'bad'[**]. Causal relations are also hard to grasp. Somebody steals something. Then later someone else buys him a drink. So stealing stuff means other people will buy you drinks? These are really hard problems to solve. You need to know about people's motivations, and the multitude of 'threads' of interaction happening at the same time, even in a very limited area.

So, recognising events and causal links, plus a moral evaluation of them is pretty difficult. I don't think we will get there anytime soon. Unsupervised learning of behavioural is also pretty difficult, as you only have unlabelled observations and no real criteria to classify them. Plus, many actions are morally ambiguous. Killing people is generally seen as bad. What about the officers who tried to assassinate Hitler in 1944? Life is complex, and our artificial models are not anywhere near that.

So even if you were able to do all that, what would you end up with? An AI system that has picked up a lot of unwritten rules about human behaviour, and then postulates that as laws? So everyone has to behave the same way? I just don't see the point of that, even as a thought experiment.

[*] leaving aside here the question how you determine what an 'event' is in the first place
[**] Please note that even if people steal things, they steal can view theft as a bad thing.


What is the advantage of using more than one environment with the advantage actor-critic? make_env = lambda: ptan.common.wrappers.wrap_dqn(gym.make("PongNoFrameskip-v4"))
 envs = [make_env() for _ in range(NUM_ENVS)]

Here is a code you can look at.

The two above lines of code create multiple environments for the game of Atari Pong with the A2C algorithm.

I understand why it is very useful to have multiple agents working on different instances of the same environment as it is presented in A3C (i.e. an asynchronous version of A2C). However, in the above code, it has a single agent working on different instances of the same environment.

What is the advantage of using more than one environment with a single agent?

UPDATE

class GymEnvVec:
    def __init__(self, name, n_envs, seed):
        self.envs = [gym.make(name) for i in range(n_envs)]
        [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]

    def reset(self):
        return [env.reset() for env in self.envs]

    def step(self, actions):
        return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))

What is the advantage of using more than one environment with a single agent?

There are two main advantages to this approach:

The dataset for training is closer to the independent, identically distributed (i.i.d.) ideal, important for theoretical and practical reasons when training a neural network. Samples taken from a single trajectory are not independent, but instead are correlated due to rules of the environment - so using a single trajectory is furthest from i.i.d. This is a similar motivation to use of experience replay tables for DQN variants of Q-learning. However, experience replay is inherently off-policy, so not a good fit to A2C or A3C that need samples taken when acting under the current policy.
Collecting experience is often a major bottleneck in training RL agents. Being able to do so in parallel in a distributed environment can significantly speed up the training process.


Is my fine-tuned model learning anything at all? I am practicing with Resnet50 fine-tuning for a binary classification task. Here is my code snippet.
base_model = ResNet50(weights='imagenet', include_top=False)
x = base_model.output
x = keras.layers.GlobalAveragePooling2D(name='avg_pool')(x)
x = Dropout(0.8)(x)
model_prediction = keras.layers.Dense(1, activation='sigmoid', name='predictions')(x)
model = keras.models.Model(inputs=base_model.input, outputs=model_prediction)
opt = SGD(lr = 0.01, momentum = 0.9, nesterov = False)

model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])  #

train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2, horizontal_flip=False)

test_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
        './project_01/train',
        target_size=(input_size, input_size),
        batch_size=batch_size,
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        './project_01/val',
        target_size=(input_size, input_size),
        batch_size=batch_size,
        class_mode='binary')

hist = model.fit_generator(
        train_generator,
        steps_per_epoch= 1523 // batch_size, # 759 + 764 NON = 1523
        epochs=epochs,
        validation_data=validation_generator,
        validation_steps= 269 // batch_size)  # 134 + 135NON = 269

I plotted a figure of the model after training for 50 epochs:

You may have noticed that train_acc and val_acc have highly fluctuated, and train_acc merely reaches 52%, which means that network isn't learning, let alone over-fitting the data.
As for the losses, I haven't got any insights.
Before training starts, network outputs:
Found 1523 images belonging to 2 classes.
Found 269 images belonging to 2 classes.

Is my fine-tuned model learning anything at all?
I'd appreciate if someone can guide me to solve this issue.

It's difficult to say without knowing what your data looks like but from the numbers it seems too less and the images might be too similar to one another or very different. In any case, I'd have checked using other networks like Inception and decreasing learning rate even further (say 0.0001) to not mess with the Imagenet weights if your data is not very different from Imagenet classes.


OpenAI Spinning Up: Breakout-v0 example I haven't been able to find any assistance / examples which could help me implement OpenAI's Spinning Up resource to solve Atari's Breakout-v0 game in the OpenAI gym.

I simply want to know why the following command doesn't run, and instead produces an error that I can't find any help on:

python -m spinup.run ppo --env Breakout-v0 --exp_name simpletest

...and then the error:

ValueError: Shape must be rank 2 but is rank 4 for 'pi/multinomial/Multinomial'
 (op: 'Multinomial') with input shapes: [?,210,160,4], [].

I understand the shape dynamics, and have written several (albeit quite unoptimized!) reinforcement learning neural nets in Python, but I was looking forward to using OpenAI's Spinning Up environment to use something more sophisticated and optimized.

Thank you so much for any help on the seeminly noobish question!

So for anyone struggling to understand the OpenAI's Spinning Up educational resource, I'll provide the answer to my question here.

Firstly, it's important to understand that the algorithms expect a 2-dimensional input shape, in rudimentary terms a shape of Box(int), which isn't the case with the default Breakout-v0 game environment, which supplies inputs in the shape Box(210, 160, 3), which is the game screen's height, width, and RGB colour space.

The Breakout-ram-v0 game environment, however, DOES provide an input of the appropriate shape, consequently Box(128,). So switching to this environment solves the initial problem I was having. To train any of the on-policy algorithms (PPO in my case), the command should be:

python -m spinup.run ppo --env Breakout-ram-v0 --exp_name simpletest

To take this a step further, how would you train an off-policy algorithm (such as DDPQ)? Well, the off-policy algorithms expect a continuous action space, which can be seen in the differences in the source code here between MountainCar-v0, and MountainCarContinuous-v0:

self.action_space = spaces.Discrete(3)

...versus:

self.action_space = spaces.Box(low=self.min_action, high=self.max_action,
    shape=(1,), dtype=np.float32)

So, to sum up, how would you train something like the Lunar Lander game environment with the DDPQ algorithm? Well, you'd need the continuous action space version, and by providing it with 2 dense layers with 192 nodes each, I was able to achieve the following results with just 200 epochs using the following command:

python -m spinup.run ddpg --env LunarLanderContinuous-v2 --exp_name ddpq-test --hid [192,192] --epochs 200

Pretty cool...anyways, I hope this helps someone sometime, and I would thoroughly recommend looking at the source code for the various OpenAI game environments, as well as their table of game environments for a quick reference to the observation and action spaces. Good luck!


Can Reinforcement Learning be used to generate sequences? So basically can we use reinforcement learning for seq2seq tasks? Albeit being a good/ bad choice, How can the approach be taken?

One renowned example for the specified case is SeqGAN

Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by directly performing gradient policy update. The RL reward signal comes from the GAN discriminator judged on a complete sequence, and is passed back to the intermediate state-action steps using Monte Carlo search. Extensive experiments on synthetic data and real-world tasks demonstrate significant improvements over strong baselines.


How does one prove comprehension in machines? Say we have a machine and we give it a task to do (vision task, language task, game, etc.), how can one prove that a machine actually know's what's going on/happening in that specific task?

To narrow it down, some examples:

Conversation - How would one prove that a machine actually knows what it's talking about or comprehending what is being said? The Turing test is a good start, but never actually addressed actual comprehension.

Vision: How could someone prove or test that a machine actually knows what it's seeing? Object detection is a start, but I'd say it's very inconclusive that a machine understands at any level what it is actually seeing.

How do we prove comprehension in machines?

This is one of the most important issues in the philosophy of artificial intelligence.

The most famous philosophical argument that attempts to address this issue is the Chinese Room argument published by the philosopher John Searle in 1980.

The argument is quite simple. Suppose that you are inside a room and you need to communicate (in a written form) with people outside the room in a certain language that you do not understand (in the particular example given by Searle, Chinese), but you are given the rules to manipulate the characters of this language (for a given input, you have the rules to produce the correct output). If you follow these rules, to the people outside the room, it will seem as if you understand this language, but you don't.

To be more concrete, when I say "apple", you understand that it refers to a specific fruit because you have eaten apples and you have a model of the world. That's understanding, according to Searle.

The most famous mathematical model of computers, the Turing machine, is essentially a system that manipulates symbols, so the Chinese Room argument directly applies to computers.

Many replies or counterarguments to the CR argument have been discussed, such as

the system reply (the symbol manipulator is only a part of the larger system).
the robot reply (the symbol manipulator does not understand the meaning of the symbols because it has not experienced the associated real-world objects, so it suggests that understanding requires a body with sensors and controllers)
the brain simulator reply (the symbol manipulator can actually simulate the activity in the brain of a person that understands the unknown language)

So, can we prove that machines really understand? Even before Searle, Turing had already asked the question "Can machines think?". To prove this, you need a rigorous definition of understanding and thinking that people agree on. However, many people do not want to agree on a definition of intelligence and understanding (hence the many counterarguments to the CR argument). So, if you want to prove that machines understand, you need to provide a proof with respect to a specific definition of understanding. For example, if you think that understanding is just a side effect of symbol manipulation, you can easily prove that machines understand many concepts (it just follows from the definition of a Turing machine). However, even if understanding was just a side effect (what does a side effect actually mean in this case?) of symbol manipulation, would a machine be able to understand the same concepts and in the same way that humans understand? It's harder to answer this question because we really do not know if humans only manipulate symbols in our brains.


Reinforcement Learning with more actions than states I have read a lot about RL recently. As far as I understood, most RL applications have much more states than there are actions to choose from.

I am thinking about using RL for a problem where I have got a lot of actions to choose from, but only very few states.

To give a handy example:
The algo should render (for whatever reason) a sentence with three words. I always want to have a sentence with three words, but I have many words to choose from. After choosing the words, I get some sort of reward.

Are RL algorithms an efficient way to solve this?

I am thinking about using policy gradients with an ε-greedy algorithm to explore a lot of the possible actions before exploiting the knowledge gained.

As far as I understood, most RL applications have much more states than there are actions to choose from.

Yes, this is quite common, but in no way required by the underlying theory of Markov Decision Processes (MDPs). The most extreme version of the opposite thing - with one state (or effectively no state, as state is not relevant)  - are k-armed bandit problems, where an agent tries to find a single best long-term action in general from a selection of actions. These problems typically would not use RL algorithms such as Q-learning or policy-gradients. However, that is partly because they are described with different goals in mind (e.g. minimising "regret" or simply gaining as much reward as possible during the learning process), and RL algorithms will work to solve them, albeit less efficiently than algorithms designed to work on bandit problems directly.

  I am thinking about using RL for a problem where I have got a lot of actions to choose from, but only very few states.

That should work, provided your problem is still a MDP. That means for instance that the state evolves according to rules depending on which action was taken in which starting state. If the state evolution is instead arbitrary or random, then you may have a contextual bandit problem instead.

There is an important difference here between:

a large number of entirely different actions, each with different results, which need enumeration and have to be explored separately

and

a large action space due to measurable values which are part of an action, such as how much force to apply in a motor

The former will require lots of exploration, since any specific combination of action and state could be the ideal. With the latter, you can use that fact that numerical values that are similar will often give similar results, which will make generalisation via function approximation (e.g. neural networks) work efficiently in your favour.

  The algo should render (for whatever reason) a sentence with three words. I always want to have a sentence with three words, but I have many words to choose from. After choosing the words, I get some sort of reward

This seems more like the first bullet-point above, although that may depend if a natural language model could be applied for example. E.g. if "This is good" and "This is great" would produce similar rewards in a specific state, then there is maybe some benefit to generalisation, although I am not quite sure where you would fit this knowledge - possibly in a generator for a sentence vector as the "raw" action and then have a LSTM-based language model produce the actual action from that vector, similar to seq2seq translation models.

  Are RL algorithms an efficient way to solve this?

Yes, but whether or not it is the most efficient will depend on other factors, such as:

Whether the environment is stochastic or deterministic with regard to both reward and state progression.
Whether state progression is key to obtaining the best rewards in the long term (e.g. there is some "goal" or "best" state that can only be reached by a certain route).
What the actual size of the MDP is $|\mathcal{S}| \times |\mathcal{A}|$. Small MDPs can be fully enumerated, allowing you to estimate action values in a simple table. If you have 10 states and 1,000,000 discrete actions, with no real pattern of actions mapping to results, then a big 10 million entry table will actually be reasonably efficient.

Competitive algorithms to RL here might be global search and optimisation ones, such as genetic algorithms. The more arbitrary and deterministic your environment is, the more likely it is that an exhaustive search will find your optimal policy faster than RL.

  I am thinking about using policy gradients with an ε-greedy algorithm to explore a lot of the possible actions before exploiting the knowledge gained.

This should be fine. Exploration here is definitely important, but finding the sweet spot for the right amount of it will be hard, and depend on other traits of the environment.

You may want to use something like upper confidence bound action selection or simply optimistic initial values, in order to ensure exploration does not miss certain actions. An epsilon greedy approach will miss a certain fraction of actions over time, and the expectation of that fraction grows smaller progressively more slowly, so it may be possible to miss an important action for a long time if you rely on being able to randomly select it.

  To give a handy example: The algo should render (for whatever reason) a sentence with three words. I always want to have a sentence with three words, but I have many words to choose from. After choosing the words, I get some sort of reward.

I would consider modelling this as sequences of 3 actions, each of which chooses a word, with the state being a start token (whatever the state you already have) plus the sentence so far, and on every 3 words the environment is consulted to reset the state to the next start token and to gather a reward (rewards for interim steps would be zero).

Doing this immediately makes the state space much larger than the action space, as your state includes history of up to two actions. If you had 10 different start states, and 100 word choices, then your state space would be 101,010 and action space 100.

This will fit available designs of RL algorithms, and allow for learning some internal language modelling if it is relevant. It will reduce your need to model sentence construction outside of the agent. Most importantly, if "good" or "bad" sentences tend to start or end with certain words, and you use function approximation, then the algorithm may discover combinations more efficiently than iterating over all sentences as if they were completely independent.


What are the most common methods to enable neural networks to adapt to changing environments? For real applications, concept drifts often exist, i.e., the relationship between the input and output changes overtime. Thus, we need our AI or machine learning system to quickly adapt to the environment.

What are the most common methods to enable neural networks to quickly adapt to the changing environment for supervised learning? Could somebody provide a link to a good review article?

For the vast majority of cases where you have a dynamic(and assumed non-linear) relationship between your input and output, you would not use modified architecture. You would simply retrain on the new data.

In some cases, based on domain knowledge or intuition, one might put a "weight" on the new data to increase or decrease its importance relative to previous data.

There are some attempts(mostly by those studying one-shot learning) to create NNs that quickly fit to new data effectively with only a few samples. However, most of these are not ready for anything resembling real-world problems(particularly on tabular data).


Could a neural network detect primes? I am not looking for an efficient way to find primes (which of course is a solved problem). This is more of a "what if" question.
So, in theory, could you train a neural network to predict whether or not a given number $n$ is composite or prime? How would such a network be laid out?

Early success on prime number testing via artificial networks is presented in A Compositional Neural-network Solution to Prime-number Testing, László Egri, Thomas R. Shultz, 2006.
The knowledge-based cascade-correlation (KBCC) network approach showed the most promise, although the practicality of this approach is eclipsed by other prime detection algorithms that usually begin by checking the least significant bit, immediately reducing the search by half, and then searching based other theorems and heuristics up to $floor(\sqrt{x})$.  However the work was continued with Knowledge Based Learning with KBCC, Shultz et. al. 2006
There are actually multiple sub-questions in this question.  First, let's write a more formal version of the question: "Can an artificial network of some type converge during training to a behavior that will accurately test whether the input ranging from $0$ to $2^n-1$, where $n$ is the number of bits in the integer representation, represents a prime number?"

Can it by simply memorizing the primes over the range of integers?
Can it by learning to factor and apply the definition of a prime?
Can it by learning a known algorithm?
Can it by developing a novel algorithm of its own during training?

The direct answer is yes, and it has already been done according to 1. above, but it was done by over-fitting, not learning a prime number detection method.  We know the human brain contains a neural network that can accomplish 2., 3., and 4., so if artificial networks are developed to the degree most think they can be, then the answer is yes for those.  There exists no counter-proof to exclude any of them from the range of possibilities as of this answer's writing.
It is not surprising that work has been done to train artificial networks on prime number testing because of the importance of primes in discrete mathematics, its application to cryptography, and, more specifically, to cryptanalysis.  We can identify the importance of digital network detection of prime numbers in the research and development of intelligent digital security in works like A First Study of the Neural Network Approach in the RSA Cryptosystem, G.c. Meletius et. al., 2002.  The tie of cryptography to the security of our respective nations is also the reason why not all of the current research in this area will be public.  Those of us that may have the clearance and exposure can only speak of what is not classified.
On the civilian end, ongoing work in what is called novelty detection is an important direction of research.  Those like Markos Markou and Sameer Singh are approaching novelty detection from the signal processing side, and it is obvious to those that understand that artificial networks are essentially digital signal processors that have multi-point self tuning capabilities can see how their work applies directly to this question.  Markou and Singh write, "There are a multitude of applications where novelty detection is extremely important including signal processing, computer vision, pattern recognition, data mining, and robotics."
On the cognitive mathematics side, the development of a mathematics of surprise, such as Learning with Surprise: Theory and Applications (thesis), Mohammadjavad Faraji, 2016 may further what Ergi and Shultz began.


How does an activation function's derivative measure error rate in a neural network? A blog post called "Text Classification using Neural Networks" states that the derivative of the output of a sigmoid function is used to measure error rates.

What is the rationale for this?

I thought the derivative of a sigmoid function output is just the slope of the sigmoid line at a specific point.

Meaning it's steepest when sigmoid output is 0.5 (occuring when the sigmoid function input is 0).

Why does a sigmoid function input of 0 imply error (if i understand correctly)?

Source: https://machinelearnings.co/text-classification-using-neural-networks-f5cd7b8765c6

  We use a sigmoid function to normalize values and its derivative to
  measure the error rate. Iterating and adjusting until our error rate
  is acceptably low.

def sigmoid(x):
    output = 1/(1+np.exp(-x))
    return output

def sigmoid_output_to_derivative(output):
    return output*(1-output)

def train(...)
    ...
    layer_2_error = y - layer_2
    layer_2_delta = layer_2_error * sigmoid_output_to_derivative(layer_2)
    ...

UPDATE

Apologies. I don't think I was clear (I've updated the title)

I understand we don't need to use sigmoid as the activation funtion (we could use relu, tanh or softmax).

My question is about using the derivative to measure the error rate (full quotation from article above in yellow) -> what does the derivative of the activation function have to do with measuring/fixing the "error rate"?

This derivative is used when calculating the error of your machine learning algorithm during gradient based minimization methods.
Read below for more info.

When performing supervised classification (with X, Y data vectors of inputs and outcome data to train with) you begin with the error function

E(X, Y; θ)= ∑i (ƒ(xi; θ)-yi)2

for total error over all data instances i, where f is your neural network, linear regression,...method of interest and θ is the set of weights. The goal here is to find weights that minimize your error when predicting training data (y) (which ideally generalizes to new data as well). To be explicit, ƒ(xi; θ); outputs value of interest which should be yi. And E measures how far off it is in prediction.

So to train your classifier, you optimize E with something like gradient descent. Thus when ∂E/∂θ = 0 (for a particular θ), that means you hit a local minimum for the error function, or a point where the error in the current state of the predictor is low, meaning it is (hopefully) a good predictor.

Note the ƒ here is not the same as an activation function, as a neural network is defined differently than in linear regression, etc. and must perform a special kind of gradient descent called backpropagation.

So when you take ∂E/∂θ, what does it equal for a neural net? You should note the activation functions derivative is involved which is how it’s used to measure error so to say.


Should one use an "other" category in image classification? In image classification, there are sometimes images that do not fit in any category.
For example, if I build a CNN in Keras to classify Dogs and Cats, does it help (in terms of training time and performance) to create an "other" (or "unclassified") category in which images of houses, people, birds, etc., are classified? Is there any research paper that discusses this?
A similar question was asked before here, but, unfortunately, it has no answer.

It is not advisable because if you use an "other" class, you are just increasing problems for your network. Since "other" means not dog and not cat, then, what common feature does it have? Most of the time the "other" images won't have many features in common. If they do, then go ahead and make an "other" class.
There is a better way: if the probabilities for both cat and dog are less than a threshold (you need to decide that, take, 0.5), then, you can say it is an "other" object.


What are the typical sizes of practical/commercial artificial neural networks? I'm interested in artificial neural networks (ANN) and I wonder how big ANNs in practical use are, for example, Tesla Autopilot, Google Translate, and others.
The only thing I found about Tesla is this one:

"A full build of Autopilot neural networks involves 48 networks that
take 70,000 GPU hours to train. Together, they output 1,000 distinct
tensors (predictions) at each timestep."

It seems like most companies don't publish clear information about their ANN sizes. I really can't find anything detailed on this subject.
Is there any information about the size of big practical/commercial ANNs that include something like the amount of neurons/connections/layers etc.?
I'm looking for a few examples in this scale with more precise information on the size of the neural networks.

NLP Domain
You can easily find such open-source neural networks in NLP applications that have been published by Companies like Google. For example, in BERT models, you can see the BERT-Base has the following specifications:

BERT-Base, Multilingual Cased: 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters

You can find more data about other versions of BERT in the same link.
Another example is GPT models, like GPT-3:

All GPT-3 models use the same attention-based architecture as their GPT-2 predecessor. The smallest GPT-3 model (125M) has 12 attention layers, each with 12x 64-dimension heads. The largest GPT-3 model (175B) uses 96 attention layers, each with 96x 128-dimension heads.

Image Procssing Domain
Another useful domain for your expectation is image processing tasks such as image classification. Pretrained models such as VGG, ResNet, and Inception. These are mostly used for image classification tasks in different companies, and you can find their specification many where. For example for VGG-16, we can see the followings:

Speech Processing
Another practical domain is Auto-Speech Recognition or ASR in short. One of the renowned models in this context is DeepSpeech(2) by Baido Research center. For example, you can find some info like the number of its parameters and its structure in this github link.
Summing up
Note that one regular metric to measure the size of neural networks is "the number of parameters" of the network that is required to be learned in the training phase. Hence, you can compare the size of models even between cross domains by knowing their number of parameters (instead of going to more details about the number of hidden layers and their types). Although, sometimes the length (number of layers) and height (number of neurons in each layer) of the network are very important in the matter of performance and capability of the network.


What are the implications of the "No Free Lunch" theorem for machine learning? The No Free Lunch (NFL) theorem states (see the paper Coevolutionary Free Lunches by David H. Wolpert and William G. Macready)

  any two algorithms are equivalent when their performance is averaged across all possible problems

Is the "No Free Lunch" theorem really true? What does it actually mean? A nice example (in ML context) illustrating this assertion would be nice.

I have seen some algorithms which behave very poorly, and I have a hard time believing that they actually follow the above-stated theorem, so I am trying to understand whether my interpretation of this theorem is correct or not. Or is it just another ornamental theorem like Cybenko's Universal Approximation theorem?

This is a really common reaction after first encountering the No Free Lunch theorems (NFLs). The one for machine learning is especially unintuitive, because it flies in the face of everything that's discussed in the ML community. That said, the theorem is true, but what it means is open to some debate.

To restate the theorem for people who don't know it, the NFL theorem for machine learning is really a special case of the NFL theorem for local search and optimization. The local search version is easier to understand. The theorem makes the following, somewhat radical claim:

  Averaged across all possible optimization problems, the average solution quality found by any local search algorithm you choose to use is exactly the same as the average solution quality of a local "search" algorithm that just generates possible solutions by sampling uniformly at random from the space of all solutions.

Another formulation, when people want an even stronger reaction, is to say that if you want to find the best solution to a problem, it's just as good to try things that seem to be making your solution iteratively worse as it is to try things that seem to be making your solution iteratively better. On average, both these approaches are equally good.

Okay, so why is this true? Well, the key is in the details. Wolpert has sometimes described the theorem as a specialization of Hume's work on the problem of induction. The basic statement of the problem of induction is: we have no logical basis for assuming that the future will be like the past. Logically, there's no reason that the laws of physics couldn't all just radically change tomorrow. From a purely logical perspective, it's totally reasonable that the future can be different from the past in any number of ways. Hume's problem is that, in general the future is like the past in a lot of ways. He tried to formulate a philosophical (logical) argument that this needed to be so, but basically failed.

The No Free Lunch theorems say the same thing. If you don't know what your search space looks like, then if you iteratively refine your guess at what a good solution looks like, in response to the observations you've made in the past about what good solutions look like (i.e. learning from data), then it's just as likely that operation you make helps as it is that it hurts. That's why the "averaged over all possible optimization problems" part is key. For any optimization problem where hill climbing is a good strategy after $k$ moves, we can make one that is identical, except that the kth hill climbing move leads to an awful solution. The actual proof is more subtle than that, but that's the basic idea.

A very brief lay summary might be:

  A machine learning algorithim can only be made to work better on some kinds of problems by being made to work worse on another kind of problem.

So what does this mean in a practical sense? It means that you need to have some apriori reason for thinking that your algorithm will be effective on a particular problem. Exactly what a good reason looks like is the subject of vigorous debate within the ML community. This is very closely related to the bias/variance tradeoff.

Some common responses are:

When you're looking at a new optimization problem, although it could have any random kind of structure, the problems we actually encounter in the real world are a lot more regular, and certain common themes are present, like the the fact that moving "uphill" (minimizing error) iteratively tends to lead to good solutions. Basically, this school of thought says NFL is an ornamental theorem: most ML algorithms work better on "the kind of problems we see in real life", by working worse on "the kind of problems we don't see in real life".
When you're looking at a new optimization problem in [insert your favourite application domain], although it could have any random kind of structure,  problems tend to look like [whatever you think], which makes [your favorite algorithm] a lot more effective than random guessing.
Wolpert & McCready themselves published an interesting result showing that there actually are specialized optimization processes, based on co-evolution, that are consistently better than random guessing.

Regardless, it's indisputable that some algorithms are better than others, in certain sub-domains (we can see this empirically). NFL tells us that to be better there, they need to be worse somewhere else. The question up for debate is whether the "somewhere else" is real problems, or purely artificial ones.


Do individual dimensions in vector space have meaning? Word2vec assigns an N-dimensional vector to given words (which can be considered a form of dimensionality reduction).

It turns out that, at least with a number of canonical examples, vector arithmetic seems to work intuitively. For example "king + woman - man = queen".

These terms are all N-dimensional vectors. Now, suppose, for simplicity, that $N=3$, $\text{king} = [0, 1, 2], \text{woman} = [1, 1, 0], \text{man} = [2, 2, 2], \text{queen} = [-1, 0, 0]$, then the expression above can be written as $[0, 1, 2] +  [1, 1, 0] - [2, 2, 2] = [-1, 0, 0]$.

In this (contrived) example, the last dimension (king/man=2, queen/woman=0) suggests a semantic concept of gender. Aside from semantics, a given dimension could "mean" a part of speech, first letter, or really any feature or set of features that the algorithm might have latched onto. However, any perceived "meaning" of a single dimension might well just be a simple coincidence.

If we picked out only a single dimension, does that dimension itself convey some predictable or determinable information? Or is this purely a "random" artefact of the algorithm, with only the full N-dimensional vector distances mattering?

Do individual dimensions in vector space have meaning?

IIRC, some dimensions are interpretable, but in general this is not the case. Also it is debatable as to wether it is actually learning the actual representation or just an approximation of it. But in any case its not very reliable outside from some edge cases.

  If we picked out only a single dimension, does that dimension itself convey some predictable or determinable information?

Yes, but as to what that information entails in terms of "meaning" is lesser clear. You could say that if in a certain dimension the distance between two vectors is 0, then you have am estimate of the real distance that is better than guessing.


Why is the status of artificial life software so under-developed? I'm interested in self replicating artificial life (with many agents), so after reviewing the literature with the excellent Kinematic Self-Replicating Machines I started looking for software implementations. I understand that the field is still in the early stages and mainly academic, but the status of artificial life software looks rather poor in 2019.

On wikipedia there is this list of software simulators.
Going trough the list only ApeSDK, Avida, DigiHive, DOSE, Polyword have been updated in 2019. I did not find a public repo for Biogenesis. ApeSDK, DigiHive and DOSE are single author programs.

All in all I don't see a single very active project with a large community around (I would be happy to have missed something). And this is more surprising considering the big momentum of AI and the proliferation of many ready to use AI tools and libraries.

Why is the status artificial life software so under-developed, when this field looks promising both from a commercial (see manufacturing, mining or space exploration applications) and academic (ecology, biology, human brain and more) perspective? Did the field underdelivered on expectations in past years and got less funding? Did the field hit a theoretical or computational roadblock?

I don’t know for certain, but I can make a guess. This is just my opinion, some others may disagree.
The field of ALife has four branches that I’m aware of:
Self-Organizing/self assembly behavior. This is the application you refer to, another context it’s useful is swarm control (for drone swarms, for example). While this is technically ALife, as far as I’m aware it’s not really where most of the emphasis is. Swarm control and self assembly are seen as “different” problems, as machines that can work together and also build more of themselves is interesting (and potentially dangerous), but is missing out on the diversity, the open-endedness that life on earth has. Much of ALife research is focused on trying to formally define this open-endedness and coming up with systems that achieve that. Self assembly and swarm control are interesting and difficult problems, just different. This leads to the other three sides of ALife research:
Coming up with environments, and running tests on them. This is a constant game of coming up with a definition that seems to capture open-endedness, then coming up with ALife sims that meet that criteria but fall short of our expectations. So new definitions are made and we repeat. Geb is a classic example: Geb has passed pretty much every test so far, but it’s fairly uninspiring to watch. Most of those programs you reference chose a particular ALife paradigm, but that paradigm may not be the right one, and is often disappointing. Because we still haven’t found something that really “looks like life”, new paradigms and programs are constantly being created and abandoned when they fail to work (Or perhaps some would have already worked, but the computing time is too much). That’s what you’re seeing. Without any unifying theory or sim that is really convincing, I suspect it’ll stay this way for a while. And because:

we still haven’t made much progress since Karl Sims in the 90s, or since Geb (this point is debatable)
these sorts of sims don’t really have much commercial use aside from games

the direction of making new simulators seems to be lacking funding and research interest, as far as I can tell. Commercial sim games seem to push the boundary these days.
Fortunately there’s a sub field of cellular automata life that’s pretty interesting, its software is slightly more developed due to the overlap with cellular automata and ease of implementation, and research seems to be progressing there at an okay rate.
Realistically, there seem to be two things people want: novel behavior, and novel bodies. My two cents is that these are separate problems, and achieving both is more expensive than just achieving one. But most of these sims end up not balancing development happening in both of these factors (doing this is very difficult), so one factor develops much further than the other, and this disconnect is disappointing to the sim creator. For example, Geb does behavioural diversity really well, while Karl Sims does body diversity well. Sensitivity to small details like mutation rate or genetic encoding also can be quite frustrating. Fortunately, eventually we’ll sorta get behavioural diversity for free in any sim once RL/AI is really understood well.
The third piece of ALife research I’m aware of is the theoretical side, which right now mostly isn’t really far enough along to warrant practical implementation. One big branch of this is the learning theory side, represented by Valiant’s Evolvability theory and followups. Essentially this talks about what functions are possible to evolve, and using stuff like PAC Learning theory they are able to prove some things. Some of these models are more natural than others, but it’s an interesting perpendicular approach to coming up with sims and seeing if they do what we want. Maybe eventually these two approaches will meet in the middle at some point, but they haven’t yet.
The fourth piece is Artificial Chemistry. I recommend this paper as a somewhat dated overview. While this is technically a field of ALife, and is centered around understanding a chemical system that has the necessary emergent properties, it has broken off into applications that may have industrial relevance. For example, robust self repairing and self assembling electronic systems, DNA computing (DNA is capable of simulating arbitrary chemical reaction networks which are capable of arbitrary computing), and artificial hormone systems for automatic task assignment. This has some software developed, but much of that software isn’t really considered ALife anymore since it has branched off into its own domain.


Expected behavior of adversarial attacks on deep NN? I am trying adversarial attack (AA) for a simple CNNs. Instead of the clean image, my simple CNN is trained with attacked images as suggested by some papers. As the training goes on, I am not sure if the training is well going or something is wrong.
Here is what I observed:
When the epsilon value is large, the classification performance of the model from the adversarial training is low. I understand if the attacked image is given to the model, then the performance is poor. Although the model is from the adversarial training, because the epsilon is large, the model is poorly perform. However, when an clean image is given, the performance of the model is still low. Performance on the clean images are higher than the performance of the attacked images, but not as high as the baseline model without adversarial training.
So, I wonder if the adversarial training also degrades the performance of the model on the clean images. When I read papers, I only see the results on the adversarial Images, not clean images. If you have any experience, it will be very helpful to check if my training code is working well or not.
When the epsilon is very large, the accuracy of the model on clean image is around 15%. The model without the adversarial training is around 81%.
Some details.
I use PGD attack with 5-iterations and epsilon is one of eps = [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.03, 0.05, 0.07]. Step size is eps/3. Only one epsilon is selected and the adversarial training is conducted. So there are 8 different models trained with different epsilons.  I use natural image Dataset.

This seems to be a known problem, and intuitively seems reasonable. You might be interested in the paper Adversarial Training Can Hurt Generalization.
The authors suggest that this might be because training on the perturbed data requires the model to learn more robust features, which means more samples are required to obtain performance comparable to a model that is not adversarially trained.
You could try collecting or generating additional samples to see if this leads to an improvement. They also mention that in their experiments on the MNIST dataset, using Xavier initialisation led to a significant benefit, so you could experiment with that too.


In Deep Q-learning, are the target update frequency and the batch training frequency related? In a Deep Q-learning algorithm, we perform a batch training every train_freq and we update the parameters of the target network every target_update_freq. Are train_freq and target_update_freq necessary related, e.g., one should be always greater than the other, or must they be independently optimized depending on the problem?
EDIT Changed the name of batch_freq to train_freq.

It is fairly common in DQN to train a minibatch after every observation received after the replay memory has enough data (how much is enough is yet another parameter). This is not necessary, and it is fine to collect more data between training steps, the alogrithm is still DQN. The value higher than 1 for train_freq here might be related to use of prioritised replay memory sampling - I have no real experience with that.
The update to target network generally needs to occur less frequently than training steps, it is intended to stabilise results numerically, so that over or under estimates of value functions do not result in runaway feedback.
The parameters choices will interact each other, most hyperparameters in machine learning do so unfortunately. Which makes searching for ideal values fiddly and time-consuming.
In this case it is safe to say that train_freq is expected to be much lower than target_update_freq, probably by at least an order of magnitude, and more usually 2 or 3 orders of magnitude. However, that's not quite the same as saying there is a strong relationship between choices for those two hyperparameters. The value of batch_size is also relevant here, as it shows the rate that memory is being used (and re-used) by the training process.
The library you are using has these defaults:
    batch_size::Int64 = 32
    train_freq::Int64 = 4
    target_update_freq::Int64 = 500

They seem like sane starting points. You are relatively free to change them as if they were independent, as there is no simple rule like "target_update_freq should be 125 times train_freq". As a very rough guide, you can expect that high values of train_freq, low values of batch_size and low values of target_update_freq are likely to cause instability in the learning process, whilst going too far in the opposite direction may slow learning down. You might be able to set train_freq to 1, but I am not completely certain about that either in combination with the prioritised replay memory sampling which seems to be the default in the library you are using.


What are the exact meaning of "lower-order structure" and "higher-order structure" in this paper? I recently read a paper on community detection in networks. In the paper EdMot: An Edge Enhancement Approach for Motif-aware Community Detection, the authors consider the "lower-order structure" of the network at the level of individual nodes and edges. And they mentioning about some "higher-order structure" method. The point is, what is the exact meaning (definition) of lower- and higher-order structure in a network?

Low order/low level information refers to the most granular level of information. This is the most informative in terms of volume of information, but it can often be difficult to conceptualise for humans.

High order/high level information refers to abstractions of the low level information to more intuitive but less easy to describe technically concepts.

An example would be images of faces. The low level information might be the raw $x, y, z$ values of the pixels: their position and colour value. Some high level information might be the direction the face in the images is facing, from what direction the lighting in the image is coming etc.

In the paper cited, the example they use is low level information (node and edge values); high level information (motifs).


Adversarial Q Learning should use the same Q Table? I'm creating a RF Q-Learning agent for a two player fully-observable board game and wondered, if I was to train the Q Table using adversarial training, should I let both 'players' use, and update, the same Q Table? Or would this lead to issues?

should I let both 'players' use, and update, the same Q Table?

Yes this works well for zero-sum games, when player 1 wants to maximise a result (often just +1 for "player 1 wins") and player 2 wants to minimise a result (score -1 for "player 2 wins"). That alters algorithms such as Q-learning because the greedy choice switches beween min and max functions over action values - player 1's TD target becomes $R_{t+1} + \gamma \text{min}_{a'} [Q(S_{t+1}, a')]$ because the greedy choice in the next state is taken by player 2.

Alternatively, if the states never overlap between players, then you could learn a Q function that always returns the future expected return according the current player. This can be forced if necessary by making part of the state record whose turn it is. With this you need a way to convert between player 1 and player 2 scores in order to use Q learning updates. For zero-sum games, then the Q value from a player 2 state is negative of player 1's value for that same state, and vice versa, so the TD target changes for both players to $R_{t+1} - \gamma \text{max}_{a'} [Q(S_{t+1}, a')]$

The first option can result in slightly less complexity in the learned function, that might be an issue if you are using function approximation and learning a Q function e.g. using neural networks instead of a Q table. That may result in faster learning and generalisation, although it will depend on details of the game.

  Or would this lead to issues?

No major issues. I am performing this kind of training - a single Q function estimating a global score which P1 maximises and P2 minimises - for the Kaggle Connect X competition, and it works well.

I can think of a couple of minor things:

You may still want to have the ability for each player to be using a different version of the table or learned Q function. This would allow you to have different versions of your agent (e.g. at different stages of learning) compete, to evaluate different agents against each other. To do this, you have to write code that allows for multiple tables or functions in any case.
You need to keep track of how both players express and achieve their opposing goals when using the table, as you can already see by the modified TD targets above. This becomes more important when adding look-ahead planning, which is a common addition and can significantly improve the performance of an agent - during look-ahead you must switch between player views on what the best action choice is. It is possible to make off-by-one errors in some part of the code but not others and have the agent learn inconsistently.


Should I be trying to create a generic or specific (to particular game) reinforcement learning agent? I'm creating an RL application for the game Connect Four.

In general, should I be aiming to create an application that's more generic, which would 'learn' different games, or specific to a particular game (e.g. Connect Four, by assigning greater rewards to certain token positions in the C4 grid)?

Does the difference between the two approaches just come down to adapting their respective reward functions to reward specific achievements or positions (in a board game setting), or something else?

If what you mean by a generic reinforcement-learning application is an application that can learn any game (or some games), then you can't do it. Why? Because the goal of each game isn't the same, so you have to adapt the rewards depending on the game. If you just want to make an AI for Connect Four, I suggest you to make a specific RL application for that game.

I want to mention another thing: you shouldn't give a reward based on token's position, because it's hard to know what token's position is the best. Instead, just assign a big reward to the winner. That way, you're generalizing your algorithm, and you're avoiding your AI to focus on the wrong goal.

You have to be careful, you don't know for sure what token's position is the best. Try to give a reward ONLY when a player wins the game, because you can know for sure that winning is the best thing to do. It may sound idiot said like that, but that way, you're ensuring you AI to learn by itself what token's position is the best.


Is it still called linear separation with a layer of more than 1 neuron A single neuron will be able to do linear separation. For example, XOR simulator network:

x1 --- n1.1
   \  /    \
    \/      \
             n2.1
    /\      /
   /  \    /
x2 --- n1.2

Where x1, x2 are the 2 inputs, n1.1 and n1.2 are the 2 neurons in hidden layer, and n2.1 is the output neuron.

The output neuron n2.1 does a linear separation. How about the 2 neurons in hidden layer?

Is it still called linear separation (at 2 nodes and join the 2 separation lines)? or polynomial separation of degree 2?

I'm confused about how it's called because there are curvy lines in this wiki article: https://en.wikipedia.org/wiki/Overfitting

What you have depicted is a nonlinear classificator. Although each stage does a linear separation, the sequential composition of linear separations is nonlinear. The nonlinearity of the neuron is key in this regard, as otherwise it would all be equivalent to a matrix multiplication, which is linear. You were right about the degree, although it's rarely called like that. People usually describe just the number of layers, and I guess the main reason is that it's not directly equivalent to such polynomials, as that depends on other factors (e.g. the activation function).