Spam Detection using Recurrent Neural Networks I am working on this code for spam detection using recurrent neural networks.

Question 1. I am wondering whether this field (using RNNs for email spam detection) worths more researches or it is a closed research field.

Question 2. What is the oldest published paper in this field?

Quesiton 3. What are the pros and cons of using RNNs for email spam detection over other classification methods?

Initially, spam detection relied on simple rule based techniques to sort out spam. However following Paul Graham's famed article 'A Plan for Spam'  the Naive Bayes approach became very popular to the point that it became regarded as the baseline for dealing with spam.

However following breakthroughs in deep learning, researchers have now turned their focus to neural networks to help them deal with the perenial problem of spam emails. Google recently reported that by introducing NN's to g-mail's spam filters. It took them from 99.5% to over 99.9% accuracy, suggesting that neural networks especially when used in conjunction with Bayesian classification may be effective for enhancing spam filters. You can refer to the link below to read about Google's success story https://www.wired.com/2015/07/google-says-ai-catches-99-9-percent-gmail-spam/

Developing a spam filter using neural networks is basically a classification problem. You need to follow the steps below to develop such a system. (Nikhil B 2016)

Collect a dataset of spam and legitimate email messages. Label these datasets. You can find email and spam datasets here http://csmining.org/index.php/spam-email-datasets-.html
Process these messages with feature extraction and vectorising techniques i.e. tf-idf vectorizer, word2vec, bag-of-words e.t.c.
Once you have vectorised the dataset succesfully, apply a supervised learning NN algorithm i.e. radial basis network, multi-layer perceptron (MLP) or backpropagation.
Train your labelled data-set on the neural network. Once training is complete you can use cross validation to calculate the precision of your trained model using the test dataset.

Some of the advantages of using NN's for spam detection over other methods include.

Neural networks have a higher accuracy of identifying spam as demonstrated by google.
They have a lower false positive rate compared to other methods such as rule based techniques.
Their main disadvantage is that they require specialised computing hardware to deploy.

Some old influential papers published in the field include.

Machine Learning Techniques in Spam Filtering (2004) http://ats.cs.ut.ee/u/kt/hw/spam/spam.pdf

Detecting Spam Blogs: A Machine Learning Approach (2006) https://www.aaai.org/Papers/AAAI/2006/AAAI06-212.pdf

A review of machine learning approaches to Spam filtering (2009) https://www.sciencedirect.com/science/article/pii/S095741740900181X


What kind of neural network can be trained to recognise patterns? Is there a type of neural network that can be fed patterns to train itself on to complete new patterns that it has not seen before?
What I'm trying to do is train a neural network to transform an image into another image. The image may be slightly different each time (denoted with different lines in the shapes) but a human would get the idea of how the new images should look. I'd like to make a network that can learn how to learn what comes next and then predict the rest of the sequence from the first part of a new sequence.
Taking the picture below as an example. The neural network would be fed the patterns in grey and learn how to predict the next ones in the sequence. Then the user would put the blue shapes into the network and hope to get the green ones out.
Is there a neural network that could perform this type of function of completing a pattern based on only a small number of examples to start the pattern based on the other patterns it has seen?

EDIT: Corrected image and added more context

The idea is simple, but it requires some time to develop.
Assumption: I am assuming in your problem the final model will have seen all possible shapes.
What your algorithm needs is a convolutional NN to understand each shape by extracting features, but you just need to be very careful with pooling.
Then what you need is a recurrent NN. In the example you showed (the image of shapes) we have bigrams (sequence of 2) which means we have the first shape as an input and the second shape is a target. In this case normal RNN should work.
But if you have sequence of many shapes, for example 10 shapes and let's say the final shape, 10th, is the target; And also if the sequence is in the way that 10th shape could be more depend on initial shapes (e.g. 1st or 2nd shape) then what you also need to consider beside RNN is long-short-term-memory (LSTM).
I cannot think of a simpler solution than this.


What do I need to study for machine learning? Starting from last year, I have been studying various subjects in order to understand some of the most important thesis of machine learning like

  S. Hochreiter, & J. Schmidhuber. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780.

However, due to the fact that I don't have any mathematical backgrounds, I started to learn subjects like

Calculus
Multivariate Calculus
Mathematical Anaylsis
Linear Algebra
Differential Equations
Real Anaylsis (Measure theory)
Elementary Probability and Statistics
Mathematical Statistics

Right now, I can't say I have done studying those subjects rigorously, but I know what the subjects above want to deal with. The thing is that I don't know what I have to do at this point. There are many subjects that machine learning uses to solve many problems out there and I don't know how to utilize them correctly.

For example, reinforcement learning is now one of the most popular topic that hundreds of thousands of researchers are now doing their research to make a breakthrough of curse of dimensionality. But, as a future employee who's gonna work in IT companies, the task on the desk wouldn't be something I expected to do.

Is it important to have my own expertise to work in the fields? If so, what kinds of subjects do I have to study right now?

For your convenience, I want to know more about Markov process and Markov decision process.

As a master's degree student in Artificial Intelligence, I strongly advise you to study some basics in Machine Learning.

To do that, you can get a good book (Machine Learning, Tom Mitchell, McGraw Hill, 1997) for the theory and practice by yourself trying some Kaggle competitions.

I suggested the book of Mitchell because he is an expert in the field, and lot of Machine Learning courses uses his book. You can also follow his videolectures online

On Kaggle, you can find many useful tutorials (named as Notebooks) to get started working with the available datasets.
Some tutorials about Titanic Challenge here


Which is best: evaluation of states or probability of moves? If you have a game and you are training an AI there seems to be two ways to do it.

First you take the game-state and a possible move and evaluate whether this move would be good or bad:

(1) GAME_STATE + POSSIBLE_MOVE  --> Good or bad?

The second is to take the game state and get probabilities of every conceivable move:

(2) GAME_STATE ---> Probabilities for each move

It seems that both models are used. I.e. in language and RNN might use (2) to find the probabilities for each next word or letter. But AlphaZero might use (1). Noting also that in a game like chess GAME_STATE + POSSIBLE_MOVE = NEW_GAME_STATE. Whereas in some games you might not know the result of your move.

Which do you think is the best method? Which is the best way to do AI? Or some combination of the two?

Which do you think is the best method?

As with most machine learning, each approach has its strengths and weaknesses, and other than a little bit of intuition:

Policy-based methods are strong in large or continuous action spaces, and/or where there is a simple relationship between state and optimal action. E.g. controlling a robotic arm with continuous action space.
Value-based methods are strong where there is a simple relationship between state and value under an optimal policy. E.g. in a maze game.

It may not always be clear which is best, in which case experimentation is required. If using neural networks, there will then be a large number of hyper-parameters on each approach, so it may be hard to come to a strict conclusion about which is better. Although you can include "easy to find a working model" or "robust for a range of hyper-parameter values" as benefits of any type of model if you wish - these are important practical benefits of any approach, developers rarely want to do 100s of experiments to tune a learning rate parameter for example.

  Which is the best way to do AI? Or some combination of the two?

Actor-Critic, as seen in Asynchronous Advantage Actor Critic. A3C and A2C (deterministic variant of A3C) is producing current state-of-the-art results in video games. This is a combination of both approaches, where the agent maintains two related models - one, the Actor, generates a policy directly by looking at the state, and the second, the Critic, tracks the estimated value of each state. Often, these two models share some parameters - e.g. using neural networks, the initial layers may be the same for both.


What is machine learning? What is the definition of machine learning? What are the advantages of machine learning?

What is machine learning?
Machine learning (ML) has been defined by multiple people in similar (or related) ways.
Tom Mitchell, in his book Machine Learning (1997), defines an ML algorithm/program (or machine learner) as follows.

A computer program is said to learn from experience $E$ with respect to some class of tasks $T$ and performance measure $P$, if its performance at tasks in $T$, as measured by $P$, improves with experience $E$.

This is a quite reasonable definition, given that it describes algorithms such as gradient descent, Q-learning, etc.
In his book Machine Learning: A Probabilistic Perspective (2012), Kevin P. Murphy defines the machine learning field/area as follows.

a set of methods that can automatically detect patterns in data, and then use the uncovered patterns to predict future data, or to perform other kinds of decision making under uncertainty (such as planning how to collect more data!)

Without referring to algorithms or the field, Shai Shalev-Shwartz and Shai Ben-David define machine learning as follows

The term machine learning refers to the automated detection of meaningful patterns in data.

In all these definitions, the core concept is data or experience. So, any algorithm that automatically detects patterns in data (of any form, such as textual, numerical, or categorical) to solve some task/problem (which often involves more data) is a (machine) learning algorithm.
The tricky part of this definition, which often causes a lot of misconceptions about what ML is or can do, is probably automatically: this does not mean that the learning algorithm is completely autonomous or independent from the human, given that the human, in most cases, still needs to define a performance measure (and other parameters, including the learning algorithm itself) that guides the learning algorithm towards a set of solutions to the problem being solved.
As a field, ML could be defined as the study and application of ML algorithms (as defined by Mitchell's definition).
Sub-categories
Murphy and many others often divide machine learning into three main sub-categories

supervised learning (or predictive), where the goal is to learn a mapping from inputs $\textbf{x}$ to outputs $y$, given a labeled set of input-output pairs

unsupervised learning (or descriptive), where the goal is to find "interesting patterns" in the data

reinforcement learning, which is useful for learning how to act or behave when given an occasional reward or punishment signals

However, there are many other possible sub-categories (or taxonomies) of machine learning techniques, such as

deep learning (i.e. the use of neural networks to approximate functions and related learning algorithms, such as gradient descent) or
probabilistic machine learning (machine learning techniques that provide uncertainty estimation)
weakly supervised learning (i.e. SL where labeling information may not be completely accurate)
online learning (i.e. learning from a single data point at a time rather than from a dataset of multiple data points)

These sub-categories can also be combined. For example, deep learning can be performed online or offline.
Related fields
There is also a related field known as computational (or statistical) learning theory, which concerned with the theory of learning (from a computational and statistical point of view). So, in this field, we are interested in questions like "How many samples do we need to approximately compute this function with a certain error?".
Of course, given that machine learning is a set of algorithms and techniques that are data- or experience-driven, one may wonder what the difference between machine learning and statistics is. In fact, in many cases, they are very similar and ML adopts many statistical concepts, and you may even read on the web that machine learning is just glorified statistics. ML and statistics often tackle the same problem, but from a different perspective or with slightly different approaches (and the terminology may slightly change from one field to the other). If you are interested in a more detailed explanation of their difference, you could read Statistics versus machine learning (2018) by Danilo Bzdok et al.
What is machine learning good for?
ML can potentially be used to (at least partially) automate tasks that involve data and pattern recognition, which were previously performed only by humans (e.g. translation from one human language, such as English, to another, such as Italian). However, machine learning cannot automate all tasks: for example, it cannot infer causal relations from the data (which often must be done by humans), unless you include causal inference as part of machine learning. If you are interested in causal inference, you could take a look at the paper Causal Inference by Judea Pearl (Turing Award for his work in causal inference!).


Is the minimax algorithm model-based? Trying to get my head around model-free and model-based algorithms in RL. In my research, I've seen the search trees created via the minimax algorithm. I presume these trees can only be created with a model-based agent that knows the full environment/rules of the game (if it's a game)? If not, could you explain to me why?

Minimax is a planning algorithm, and all planning algorithms need access to a model of the environment in order to look ahead or simulate possible future states and results.

Technically this does not need to be 100% accurate or complete. It could even be a learned model. However, in the case of applying minimax to classic two player games, such as chess or Connect 4, then usually the game rules are used to create perfect predictions.

This difference between planning and learning is not quite the same as model-free vs model-based RL, but the ideas do overlap considerably. You can for instance consider the experience replay approach used in DQN as a form of "background planning" where the model used is the memory of previous events, whilst the core Q-learning algorithm used inside DQN is normally considered model-free.


How to work on different models for a given problem? I am working on the MNIST data on my own. The idea is to use different values for the number of hidden layers, number of nodes in a given layer, etc. How do you organize these things while you are working on creating a model for a problem? DO you do everything in one code file or you use different code files for choosing the best?

it seems to me that you are talking about hyperparameter tuning and effect of hyperparameters on the network in general. If you are working with tensorflow, I recommend you to look into tensorboard.

Hands-on TensorBoard can be a good starting point.


How to parse conjunctions in natural language processing? Is there an accepted way in NLP to parse conjunctions (and/or) in a sentence?

By following the example below, how would I parse

  I drink orange juice if its the weekend or if it's late and I'm tired.

into

  it's the weekend

and

  it's late

and

  I'm tired

?

Implying an action will be taken when one of the above elements at the 1st level of depth is true.

I know when I hear the sentence that it means "its the weekend" OR ("it's late" AND "I'm tired"), but how could this be determined computationally?

Can an existing python/other library do this?

This seems not easy for NLP. I doubt that state-of-the-art NLP tools can reliably determine the correct hierarchical structure of independent clauses. Examples below.

The Berkeley parser gets it basically right in the sense that it can put its late and I'm tired on parallel, and they together on parallel with the weekend. But still not perfect (the weekend should be in the same subtree with It's rather than if its late and I'm tired)

The Stanford parser, which is available in Python (NLTK), incorrectly parsed I\'m tired to the same level of I drink orange juice.


What are the variables used in a Gaussian radial basis kernel in the context of SVMs? If I have the Gaussian kernel

$$
k(x, x') = \operatorname{exp}\left( -\| x - x' \|^2 / 2\sigma^2 \right)
$$

What is $x$ and $x'$ in the context of training an SVM?

$\mathbf{x} \in \mathbb{R}^p$ and $\mathbf{x}' \in \mathbb{R}^p$ are two inputs (or feature vectors).

In the context of classification with an SVM, you are given a dataset $D = \{(\mathbf{x}_i, y_i) \}_{i=1}^N$, where $\mathbf{x}_i \in \mathbb{R}^p$ is an input (or point) and $y_i$ the corresponding label. The goal is to find a hyperplane that classifies the points $\mathbf{x}_i$. The hyperplane actually corresponds to a binary classifier that splits the plane into two, so the assumption is that there are two labels. However, these points $ \mathbf{x}_i$ may not be linearly separable in $\mathbb{R}^p$, i.e. there may not be a hyperplane (in 2d, i.e. when $p=2$, a hyperplane is a line) that separates them. The kernel trick, i.e. the use of kernels (such as the Gaussian radial basis), allows an SVM to perform non-linear classification by transforming the inputs to a space where they are linearly separable.


What is the definition of rationality? I'm having a little trouble with the definition of rationality, which goes something like:

  An agent is rational if it maximizes its performance measure given its current knowledge.

I've read that a simple reflex agent will not act rationally in a lot of environments. For example, a simple reflex agent can't act rationally when driving a car, as it needs previous perceptions to make correct decisions.

However, if it does its best with the information it's got, wouldn't that be rational behaviour, as the definition contains "given its current knowledge"? Or is it more like: "given the knowledge it could have had at this point if it had stored all the knowledge it has ever received"?

Another question about the definition of rationality: Is a chess engine rational as it picks the best move given the time it's allowed to use, or is it not rational as it doesn't actually (always) find the best solution (would need more time to do so)?

When we use the term rationality in AI, it tends to conform to the game theory/decision theory definition of rational agent.

In a solved or tractable game, an agent can have perfect rationality.  If the game is intractable, rationality is necessarily bounded. (Here, "game" can be taken to mean any problem.)

There is also the issue of imperfect information and incomplete information.

Rationality isn't restricted to objectively optimal decisions but includes subjectively optimal decisions, where the the optimality can only be presumed.  (That's why defection is the optimal strategy in 1-shot Prisoner's Dilemma, where the agents don't know the decisionmaking process of the the competitor.)

Rationality here conforms to Russell & Norvig's definition, where it is related to performance in an environment.

What may be rational in one environment may not be rational in a different environment.  Additionally, what may be locally rational for a simple reflex agent will not appear rational from the perspective of an agent with more knowledge, or a learning agent.

Iterated Dilemmas, where there is communication in the form of prior choices, may provide an analogy.  An agent that always defects, even where the competitor has shown willingness to cooperate, may not be regarded as rational because  defecting vs. a cooperative agent does not maximize utility.  A simple reflex agent wouldn't have the capacity to alter its strategy.

However, rationality used in the most general sense might allow that, to the agent making the decision, if the decision is based on achieving an objective, and the decision is reached utilizing the information available to that agent, the decision is may be regarded as rational, regardless of actual optimality.


Eligibility vector for softmax policy with policy gradients There is this nice result for policy gradients that the gradient of some performance measure, $\nabla v_{\pi_{\theta}}(s_0)$ (here, in the episodic case for the starting state $s_0$ and policy $\pi$, parametrised by some weights $\theta$) is equal to the expectation gradient of the logarithm of the policy, i.e.

$$\nabla v_{\pi_{\theta}}(s_0)=\mathbb{E}\Big{[}\sum_{t=0}^{T-1}\nabla_\theta\log(\pi_{\theta}(a_t|s_t)]\cdot G_t\Big{]},$$

where $G_t$ is the discounted future reward from state $s_t$ onward and $s_T$ the final state of some trajectory $(s_0, a_0, s_1, a_1, ..., s_{T-1}, a_{T-1}, s_T)$.

Now, when using a softmax policy, $\nabla_\theta\log(\pi_{\theta}(a_t|s_t)$ can be written as

$$\nabla_\theta\log(\pi_{\theta}(a_t|s_t))=\phi(s_t,a_t)-\mathbb{E}[\phi(s_t,\cdot)],$$

where $\phi(s,a)$ is some input vector of a state-action tuple.

However: what exactly is this vector? A typical input with policy gradients (for example in a neural network) is a feature vector for the state and the output a vector with dimensions equal to the number of actions, e.g. $(14, 15, 11, 17)^T$ for four possible actions. The softmax-function now scales these outputs, which results in the logits $(.042, .114, .002, .842)^T$ in this example.

What I would usually do in neural networks is take some input vector, for example something that describes if there are borders in a grid world, e.g. $\phi(s)=(1, 0, 0, 1)^T$, and multiply that with my weight matrix $\theta$ (and add biases b), i.e. $\theta\phi(s)+b$. So, continuing above example, $1\cdot \theta_{1,1} + 0\cdot \theta_{1,2} + 0\cdot \theta_{1,3} + 1\cdot \theta_{1,4} = 14$ and $1\cdot \theta_{2,1} + 0\cdot \theta_{2,2} + 0\cdot \theta_{2,3} + 1\cdot \theta_{2,4} = 15$.

But what is $\phi(s,a)$ here? And how would I compute $\nabla_\theta\log(\pi_{\theta}(a|s))=\phi(s,a)-\mathbb{E}[\phi(s,\cdot)]$?

Calculation of gradient
\begin{align}
\nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\
&= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a')
\end{align}
is only valid for linear function approximation with action preferences of form
\begin{equation}
h(s, a, \theta) = \theta^T \phi(s,a)
\end{equation}
and softmax policy
\begin{equation}
\pi(a|s) = \frac{e^{h(s,a,\theta)}}{\sum_{a'} e^{h(s,a',\theta)}}
\end{equation}
The gradient would be calculated as it is written. For example, if your current state is $s = (1, 1)$ and in that state you have actions $a_0 = 0$ and $a_1 = 1$ and probabilities for those actions are $\pi(a_0|s) = 0.7$, $\pi(a_1|s) = 0.3$ then gradient for action $a_0$ would be
\begin{equation}
\nabla_{\theta} \log(\pi_{\theta}(a_0|s)) = (1, 1, 0)^T - (0.7 \cdot (1, 1, 0)^T + 0.3\cdot (1,1,1)^T) = (0, 0, -0.3)^T
\end{equation}
Feature vector $\phi$ can be basically anything you want. For example you could stack state feature and action (like I did in small example), you could use polynomials, radial basis functions, tile coding, etc.

If you're using multilayered neural network you would have to propagate gradients through all layers, usually done with backpropagation algorithm. Easiest way is to use automatic differentiation software (e.g. Tensorflow) which can do that for you so you don't have to write your implementation. All you have to do is define your objective function that you want to optimize
\begin{equation}
J_\theta = \sum_t \log(\pi(a_t|s_t, \theta)) G_t
\end{equation}
and software will automatically calculate gradient $\nabla J_{\theta}$ and update weights.


Price difference predictions curve almost vanished With a team, we are studying how it is possible to predict the price movement with high-frequency. Instead of predicting the price directly, we have decided to try predicting price difference as well as the features. In other words, at time t+1, we predict the price difference and the features for time t+2. We use the predicted features from time t+1 to predict the price at time t+2.

We got very excited, because we thought getting good results with the following graph

We got problems in production and we wasn't known the problem till we plot the price difference.

Here is the content of the config file

{
    "data": {
        "sequence_length":30,
        "train_test_split": 0.85,
        "normalise": false,
        "num_steps": 5
    },
    "training": {
        "epochs":200,
        "batch_size": 64
    },
    "model": {
        "loss": "mse",
        "optimizer": "adam",
        "layers": [
            {
                "type": "lstm",
                "neurons": 51,
                "input_timesteps": 30,
                "input_dim": 101,
                "return_seq": true,
                "activation": "relu"
            },
            {
                "type": "dropout",
                "rate": 0.1
            },
            {
                "type": "lstm",
                "neurons": 51,
                "activation": "relu",
                "return_seq": false
            },
            {
                "type": "dropout",
                "rate": 0.1
            },
            {
                "type": "dense",
                "neurons": 101,
                "activation": "relu"
            },
            {
                "type": "dense",
                "neurons": 101,
                "activation": "linear"
            }
        ]
    }
}

Prices don't change very fast. Therefore, the next price is almost always very close to the last price. In other words, P_{t+1} - P_{t} is very often close to zero or zero directly. If there is too many zeros then the network will only recognize the zeros. The model has picked up on that.

I guess the model learned almost nothing except the very simple relationship that the next price is close to the last price. There is not necessarily anything wrong with the model. Predicting stock prices should be a very hard problem.

So a straightforward improvement should be of taking the features as a whole instead of their difference.

I want to keep working with price difference instead of the price in itself because we are making the series potential more stationary.

What might be a good solution to deal with the repetitive zeros related to our "price difference" problem? Does applying the log-return is a better idea than applying price differences?

Does a zero inflated estimators is a good idea? First predict whether it's gonna be a zero. If not predict the value. https://gist.github.com/fonnesbeck/874808 ?

I have encountered similar problem while trying to predict forex prices. I understand it this way:

  The data based on which you try to model price differences are so poor that the lowest error is achieved by zeroing the predicted values

In other words, due to poor "correlation" in data, zero values are the best solution.

My advice would be to look for more correlated data to be used.


What's the difference between hyperbolic tangent and sigmoid neurons? Two common activation functions used in deep learning are the hyperbolic tangent function and the sigmoid activation function. I understand that the hyperbolic tangent is just a rescaling and translation of the sigmoid function:

$\tanh(z) = 2\sigma(z) - 1$.

Is there a significant difference between these two activation functions, and in particular, when is one preferable to the other?

I realize that in some cases (like when estimating probabilities) outputs in the range of $[0,1]$ are more convenient than outputs that range from $[-1,1]$. I want to know if there are differences other than convenience which distinguish the two activation functions.

Sigmoid > Hyperbolic tangent:

As you mentioned, the application of Sigmoid might be more convenient than hyperbolic tangent in the cases that we need a probability value at the output (as @matthew-graves says, we can fix this with a simple mapping/calibration step). In other layers, this makes no sense.

Hyperbolic tangent > Sigmoid:

Hyperbolic tangent has a property called "approximates identity near the origin" which means $\tanh(0) = 0$, $\tanh'(0) = 1$, and $\tanh'(z)$ is continuous around $z=0$ (as opposed to  $\sigma(0)=0.5$ and $\sigma'(0)=0.25$). This feature (which also exists in many other activation functions such as identity, arctan, and sinusoid) lets the network learn efficiently even when its weights are initialized with small values. In other cases (eg Sigmoid and ReLU) these small initial values can be problematic.

Further Reading:

Random Walk Initialization for Training Very Deep Feedforward Networks


What happens when I mix activation functions? There are several activation functions, such as ReLU, sigmoid or $\tanh$. What happens when I mix activation functions?

I recently found that Google has developed Swish activation function which is (x*sigmoid). By altering activation function can it increase accuracy on small neural network problem such as XOR problem?

The general answer to the behavior of combining common activation functions is that the laws of calculus must be applied, specifically differential calculus, the results must be obtained through experiment to be sure of the qualities of the assembled function, and the additional complexity is likely to increase computation time. The exception to such increase will be when the computational burden of the combination is small compared to the convergence advantages the combination provides.
This appears to be true of Swish, the name given to the activation function defined as
$$f(x) = x \, \mathbb{S}(\beta x) \; \text{,}$$
where $f()$ is the Swish activation function and $\mathbb{S}$ is the sigmoid function. Note that Swish is not strictly a combination of activation functions. It is formed through the addition of a hyper-parameter $\beta$ inside the sigmoid function and a multiplication of the input to the sigmoid function result.
It does not appear to be developed by Google. The originally anonymously submitted paper (for double blind review as a ICLR 2018 paper), Searching for Activation Functions, was authored by Prajit Ramachandran, Barret Zoph, and Quoc V. Le around 2017. This is their claim.

Our experiments show that the best discovered activation function, ... Swish, ... tends to work better than ReLU on deeper models across a number of challenging datasets.

Any change in activation function to any one layer will, except in the astronomically rare case, impact accuracy, reliability, and computational efficiency. Whether the change is significant cannot be generalized. That's why new ideas are tested against data sets traditionally used to gauge usefulness1.
Combining activation functions to form new activation functions is not common. For instance, AlexNet does not combine them.2. It is, however, very common to use different activation functions in different layers of a single, effective network design.

Footnotes
[1] Whether these traditions create a bias is another question. Those who follow the theory of use case analysis pioneered by Swedish computer scientist Ivar Hjalmar Jacobson or 6 Sigma ideas would say that these tests are unit test, not functional tests against real world use cases, and they have a point.
[2] To correct any misconceptions that may arise from another answer, AlexNet, the name given to the approach outlined in ImageNet Classification with Deep Convolutional Neural Networks (2012) by Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton from the University of Toronto, does not involve combining activation functions to form new ones. They write this.

The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels.
...
The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer. The internal layers are pure ReLU and the output layer is Softmax.

There are also convolution kernels and pooling layers in the AlexNet approach's series of layers used by them, and the design has entered common use since their winning of the ImageNet competition in 2012. Other approaches have won subsequent competitions.


What are the main advantages of using declarative programming languages for building AI? What specific advantages of declarative languages make them more applicable to AI than imperative languages?

What can declarative languages do easily that other languages styles find difficult for this kind of problem?

There's no objective reason to state that declarative languages are better suited for AI development. However, there's indeed a bias towards them in practice. Although most functional languages are impure (that is, they allow side effects), and such can't count as "declarative", a few languages are purely functional (that is, they don't allow side effects), most prominently Haskell. Purity is key here. In Haskell, even I/O is pure.

The key difference between imperative languages and (purely) functional languages is in the way they describe the program. An imperative program describes how to do stuff, that is, algorithms. It specifies the specific instructions that the machine must carry on in order to perform the computation. OTOH, purely functional languages describe what is to be computed, that is, the relationship between the input and the output. In mathematics, "function" is just a fancy name for a relationship between an input and an output.

Again, in the mathematical sense, the only variability is that of the function's arguments. That is, the function's output depends solely on its input (arguments). This is known as referential transparency. Referential transparency states that:

$$
\forall f \in \varphi, \forall x \in \delta_f, fx = fx
$$

Where $\varphi$ is the set of all functions, and $\delta_f$ is $f$'s domain. For the typical imperative language's definition of "function", the above doesn't hold. For instance, C's getchar() does not always return the same value.

Let's say we want to calculate the set of the ten least prime numbers whose least significant digit is $3$. First, in mathematical notation:

$$
\mathbb{S} = \mathbb{G}_{10} = \{ x \mid x \in \{ p \mid p \in \mathbb{N} \setminus \{0, 1 \} \text{ and } (\forall q \in \mathbb{N} \setminus \{0, 1, p \}, p \text{ mod } q = 0) \},  x \text{ mod } 10 = 3 \}
$$

Where G(n, s) is the set of the lesser nth elements from s. In mathematics, you don't worry about how is the set $\mathbb{S}$ supposed to be computed, but rather about $\mathbb{S}$'s definition itself.

Now, in Python (in imperative style):

def is_prime(n):
  for x in range(2, n):
    if n % x == 0:
      return False

  return True

def foo():
  s = set()
  n = 2
  while len(s) < 10:
    if is_prime(n) and n % 10 == 3:
       s.append(n)

  return set(s)

In Python, we care about (and are responsible for) the algorithm being used to compute the set. We specify, pretty much in recipe-style, how to build the set from scratch. If there's an algorithm that may be better suited for checking whether a number is prime or odd, but we don't use it, it's our fault, not Python's.

Finally, Haskell steps in:

import qualified Data.Set as Set

isPrime :: Integer -> Bool
isPrime n = ( == 1 ) . length . filter ( == 0 ) . map ( n `mod` ) $ [ 2 .. n ]

s = Set.fromList . take 10 . filter ( ( == 3 ) . ( `mod` 10 ) ) . filter isPrime $ [1..]

Haskell's version is a lot more like the mathematical model than Python is. We define the isPrime function in terms of the constraints that a prime number must obey, not by describing a step-by-step algorithm to do such a check. Moreover, s (the set we have been defining so far) is defined in terms of the constraints its members must obey, rather than in terms of an algorithm to compute s itself. The compiler, more often than not The Glorious Glasgow Haskell Compilation System (a.k.a GHC), is the one responsible for generating an algorithm. GHC's optimizer is known to be one of the strongest in the world, not because its the best compiler of 'em all, but because Haskell's nature allows for this.

Haskell (and other functional languages), in summary, have several features that make it look, taste, and behave like pure math:

Referential transparency and purity.
Haskell is lazy.
A very strong type system, with such exotic (but very useful!) stuff as recursive (and algebraic) data types.

So, the bottom line is that AI researchers often prefer functional languages over imperative languages (or, more assertively, pure over impure languages), because they are attempting to define artificial intelligence itself, by means of functions (relationship between an input and an output). At the end, we do this because we have no real algorithm for human-level intelligence to raise from a handful of transistors. Also, there's been a historical bias towards these kind of languages, starting with John McCarthy, Lisp's creator and a pioneer in early AI research.


Is it suitable to find inverse of last layer's activation function and apply it on the target output? I have a neural network with the following structure:

I am expecting specific outputs from the neural network which are the target values for my training. Let's say the target values are 0.8 for the upper output node and -0.3 for the lower output node.

The activations function used for the first 2 layers are ReLu or LeakyReLu while the last layer uses atan as an activation function.

For back propogation, instead of adjusting values to make the network's output approach 0.8,-0.3. is it suitable if I use the inverse function for atan -which is tan itself- to get "the ideal input to the output layer multiplied by weights and adjusted by biases".

The tan of 0.8 and -0.3 is 0.01396 and -0.00524 approximately.

My algorithm would then adjust weights and biases of the network so that the "pre-activated output" of the output layer -which is basically (sum(output_layer_weight*output_layer's inputs)+output_layer_biases)- approaches 0.01396 and -0.00524.

Is this suitable

The main difference your change would have is to allow you to apply a loss function to a different part of the network. This may affect training.

If you keep the same loss function (e.g. MSE), but apply it to the pre-transformed values, then you will have changed the objective of the network, perhaps significantly. Whether or not this is a good thing depends on how much you needed the original loss function. However, the fact that it would result in a different training target is usually going to be a bad thing if your original training target was correct. This will also be true if you pick a new arbitrary loss function that seems to fit the pre-transform representation better.

If you engineer a "correct" loss function such that the objective of the network remains unchanged, then the behaviour of the network will not change much - probably not at all. However, in some cases this can lead to more stable and/or faster training - it is often used for classifiers to avoid need to use exponentiation, see tf.nn.softmax_cross_entropy_with_logits in TensorFlow, which does exactly this.


Which AI algorithm to use to identify a subject from many unknown factors Say I have a set of data generated by someone. It could be either bytes from a photo, or readings from bio-sensors, and I have a huge amount of said information, from many people or subjects. Which AI algorithms could be used to learn that a set of data belongs to a subject. I would have the information map that a huge set of data belongs to  Bob, and another belongs to Alice to train the system.

You have a set of different types of data available for each of your subjects, and given one set you'd like to classify which subject it belongs to. This looks like a supervised classificiation problem.

The most popular classifiers for supervised learning are neural networks. Now given the heterogenous nature of your data types, a simple approach would be to use separate classifiers for each type of data. For example, a convolution neural net for the image data, and a simple feed forward net for the biosensor data. Another thing you try is a multi channel approach, where towards the input side you have multiple channels for the different types of data, and the final few layers are fully connected.

The image has only CNNs for the multi channel part but you could have one channel as a simple feed forward net while another one as having conv layers.

Also, if you wish to classify the data as belonging to a subject on the basis of just one of the data types from the set, then you should have separate classifiers for all types. In that case it might be worthwhile to look into classifiers other than neural nets like multi class logistic regression which might be simpler to work with for a particular data type.


What is the goal of a constraint solver? What is the goal of a constraint solver? How are constraints propagated in a constraint satisfaction search?

Any references are also appreciated.

You want Chapter 6 of Russell & Norvig's AI: A Modern Approach, for a starting place.

The goal of a constraint solver is to find an assignment of values to variables such that every variable is assigned a value, but none of a list of constraints are violated.

An example problem that a constraint solver can solve is the problem of assigning students to groups for a project. The variables are the names of the students. Each variable has a set of possible values it could take on that correspond to the names of the groups. The constraints prohibit assigning certain students to the same group (perhaps they hate each other), or require certain students to be assigned to the same group (perhaps they like each other). More complex constraints are also possible (e.g. at least 2 of these 5 students need to be in the same group).

Constraint propagation is the idea that several constraints might logically imply a stronger constraint. As a simple example, suppose that student $S$ must be assigned to group $1$. That's one constraint. Suppose further that student $T$ must be assigned to the same group as $S$. That's another constraint. Together, however, they imply that $T$ must be assigned to group $1$.

Most constraint solvers operate by iteratively trying to assign values to variables and then backtracking when a constraint is violated. When assignments are made, different constraint propagation methods make different decisions about how much computational effort to spend deriving the logical consequences of constraints, as opposed to simply trying more assignments of values to variables until the consequences become apparent (one of the existing constraints gets violated).

The simplest form of constraint propagation is called forward checking. After each speculative assignment of a value to a variable, forward checking looks at all the constraints the variable is involved in. It then derives the logical consequences of the assignments. For example, continuing with the student problem from above, if $S$ was assigned value $1$, then forward checking would notice the constraint $S=T$, and would, effectively, add the constraint $T=1$ (it actually does something a little different, but it has this effect). As another example, if the assignment $T=2$ were made, forward checking would notice that $S=T$, and would add the constraint $S=2$. It would also detect the contradiction that $S=1$ and $S=2$ are both constraints, and would declare the assignment $T=2$ invalid.

More advanced constraint propagation methods can reason about these kinds of logical consequences before assignment are made. For example, they could derive that $T=1$ before the program starts. Wikipedia has a good summary of these, starting with arc consistency.


Finding anomaly detection by pattern matching in a set of continous data I have series of sensors (around 4k) and each sensor will measure the amplitudes at each point.Suppose I train the neural network with sufficent set of 4k values (N * 4k shape). The machine will find a pattern in the series of values.If the values stray away from the pattern (that is anomaly)  it can detect the point and will be able to say that anomaly is in the 'X'th  sensor.Is this possible.If so what kind of neural network should I use?

Without knowing the kind of data and the process generating it it's hard to give a definite answer.
In general, I would attempt a network that has as inputs the actual sensor readings, and outputs the expected readings. You train this network by presenting data with errors added as inputs, and correct readings as outputs.  It should learn to guess the correct (non-anomalous) readings from a set of actual readings, and you can find the sensors with anomalies by taking the difference between actual and predicted readings.
Depending on the kind of anomalies you expect (misreadings such as fluke zero or max values, or actually anomalous states of the system being measured) your training data should be set up to have samples of such anomalies.
If there are temporal correlations (for example, temperature readings that change slowly) using a RNN might be helpful but its dimensioning heavily depends on the nature of your measured system.
External factors influencing the system (state of a heating mechanism, time of day/year, etc.) could be added at inputs for better predictions.
At the end of the day, trial and error is your friend. Start with a simple network and see how well it behaves. Go for more complex networks when you see where the limitations of the simple solution are.


Existing programs that find out words with same meanings I'm wondering if these 2 specific programs already exist and if not how hard would it be to write them:

A program that would figure out (by only "reading" large amounts of texts in human language 1 and 2) which words in second language have the same meaning as a word in first language.  You would give for input texts in both languages and for output you would get for every word in first language a list of words in second language that are most similar to it with a probability that they mean the same thing.
A program that would figure out which words have the most similar meaning by analyzing large amounts of texts in one human language.

I'm planning on writing these two programs and it would be nice if I could get existing programs that do this so that I could compare results of my program to those of existing programs.

Let us simplify case 1 of the question: assume two files, first one with numbers written in indo-arabic numeral system (i.e. 123, 9, 186754, ...), second one in roman numeral system (XX, LCVI, IX, ...). How do you match pairs of symbols with same meaning ?

Without external information or assumptions, you can not. You could made the hypothesis that probability of one specific number is the same in both examples, and base your pairing on it. But then you need to find two input files that fulfills this condition.


How can I use Generative Adversarial Networks to solve the imbalanced class problem? Problem setting
We have to do a binary classification of data given a training dataset $D$, where most items belong to class $A$ and some items belong to class $B$, so the classes are heavily imbalanced.
Approach
We wanted to use a GAN to produce more samples of class $B$, so that our final classification model has a nearly balanced set to train.
Problem
Let's say that the data from both classes $A$ and $B$ are very similar. Given that we want to produce synthetic data with class $B$ with the GAN, we feed real $B$ samples that we have into the discriminator alongside with generated samples. However, $A$ and $B$ are similar. It could happen that the generator produces an item $x$, that would naturally belong to class $A$. But since the discriminator has never seen class-$A$ items before and both classes are very close, the discriminator could say that this item $x$ is part of the original data that was fed into the discriminator. So, the generator successfully fooled the discriminator in believing that an item $x$ is part of the original data of class $B$, while $x$ is actually part of class $A$.
If the GAN keeps producing items like this, the produced data is useless, since it would add heavy noise to the original data, if combined.
At the same time, let's say before we start training the generator, we show the discriminator our classes $A$ and $B$ samples while giving information, that the class-$A$ items are not part of class $B$ (through backprop). The discriminator would learn to reject class-$A$ items that are fed to it. But wouldn't this mean that the discriminator has just become the classification model we wanted to build in the first place to distinguish between class $A$ and class $B$?
Do you know any solution to the above-stated problem or can you refer to some paper/other posts on this?

In my experience, GANs work really well for the scenario of semi-supervised learning, where you don't necessarily have labels for all your class $B$ data, but you do have a balanced dataset. In my (limited) experience, you do have to have a balanced (in numbers) set of $A$ and $B$ objects, even though you are not sure of the labels.
And yes, GANs can overfit to outliers as well, especially in the absence of a lot of examples, so be cautious.
Currently, the version that works best for me (in terms of GANs) is WGAN-GP or WGAN-LP in combination with Optimistic Mirror Descent Adam (here, Ncritic/Nactor = 1). Take a look at the paper by Adiwardana et al., especially Fig. 7 (astonishing!), for semi-supervised learning with a limited number of class labels.


Does it make sense to combine classifiers trained on the same dataset? I am working on a classification problem.
I have a dataset $S$ and I am training several prediction algorithms using S: Naive Bayes, SVM, classification trees.
Intuitively, I was planning to combine my models, and, for each data point in the test sample $S'$, take the majority vote as my prediction.
Does that make sense? I feel this is a very simplistic way to combine different models.

These are generally known as ensemble methods. Your method is essentially what Scikit-Learn's VotingClassifier does, which is perfectly reasonable and might give you better results. Of course, if you have an ensemble of classifiers and some of them perform quite poorly, the ensemble might not be able to beat the best classifier: you'll need to check this in your cross-validation. Be aware that any confidence or probability estimates may not be well-calibrated, so the predictions from the ensemble may not be particularly meaningful.
There are more elaborate ways of ensembling classifiers. Random forest classifiers, for example, are just ensembles of decision trees; a technique called bagging is also used here to improve performance.
The technique of using a second model to weight the predictions from an ensemble is known as stacked generalisation. It was introduced by Wolpert in 1991 [1], and you can find plenty of interesting examples of using the technique, e.g. on Kaggle.
[1] David H. Wolpert. Stacked generalization. Neural Networks 5.2 (1992), pp. 241-259.


Recent work on “Oracular” systems such as Eliza? I know Eliza is considered a Natural Language Processing application, but the application of NLP in this context is “Oracular”.

What I mean by Oracular is that the systems was designed to produce ambiguous output to facilitate the instinct of the user to read meaning into the answer.  (My experience with Eliza was as a child on a 64KB system and the program could fool the user for a little while based on sheer novelty, although the limitations were quickly revealed by repetition of output.  For kids, this actually became a game of tricking the program into saying funny things;)

This method has a long history in oracles, the most famous certainly being the early binary symbolic system of the I-Ching.  (Times being simpler in ancient days, the idea was that a workable amalgam of the universe could be constructed (2)+(4)+(8)+(64) symbols.  Each set of symbols is defined by the meanings of the set of the previous order and modified by sequence, which is the key for explaining a given symbol.) The output is ambiguous enough that it may be applied to any input, and rather than the system understanding the input or output, it requires the user to provide the analysis.  This may be said to be an engine for generating human insight about a problem. (Monte Carlo may even be utilized, although the sage, working to attain an understanding of each of the symbols, may use intuition to match input with output.)

The reason I ask is I believe this demonstrates a very ancient, algorithmic method of engaging the human mind without the requirement that the algorithm understand the input or output--merely that it produce output to which meaning can be ascribed.

(This almost certainly relates to the relative success of “pornbots” beating the “Turing test” in that the user is chemically induced to read meaning into a given output or string of outputs.)

Aspects of the grounding problem are what got me thinking about this.  Not sure if it's relevant that the broken and unbroken lines in the I-Ching represent on and off bits and can be extended to circuits as open and closed.

Tariq's comment hints at this, but this is still in some sense a very mainstream idea.

Check out the transcripts for this year's Loebner Prize. The winner (called Tutor) is again making use of Eliza-like deflections. Some of the other candidates try to use seduction (Columbia), or speaking in a flighty or insistent way (like 2017's Rose).

In all cases, the bots are relying on humans to read meaning into ambiguous or vague statements, which we seem to be hardwired to do. Kind of neat!


Why does my neural network to predict $x$ given $\sin(x)$ not generalize? I made a simple feedforward neural network (FFNN) to predict $x$ from $\sin(x)$. It failed. Does it mean the model has overfitted? Why doesn't it work?

set.seed(1234567890)
Var3 <- runif(500, 0, 20)
mydata3 <- data.frame(Sin=sin(Var3),Var=Var3)
set.seed(1234567890)
winit <- runif(5500, -1, 1)
#hidUnit <- c(9,1)
set.seed(1234567890)
nn3 <-neuralnet(formula = Var~Sin,data = mydata3,
                hidden =c(4,2,1),startweights =winit,
              learningrate = 0.01,act.fct = "tanh")

plot(mydata3, cex=2,main='Predicting x from Sin(x)',
     pch = 21,bg="darkgrey",
     ylab="X",xlab="Sin(X)")
points(mydata3[,1],predict(nn3,mydata3), col="darkred",
       cex=1,pch=21,bg="red")

legend("bottomleft", legend=c("true","predicted"), pch=c(21,21),
       col = c("darkgrey","red"),cex = 0.65,bty = "n")

The labels are not unique for the input domain [0,20]. Think about sin(x)=0, x could be 0, pi, 2*pi, 3*pi, ..., n*pi, all are correct from a mathematical point of view, but this is not reflected in your MSE loss. At this point your NN has to guess the correct label from your input data. Predicting the mean of your input data is the safest bet for the network.
In essence you're trying to build a arcsin function with a NN. If only consider x values in [-0.5*pi,0.5*pi], the labels are unique and your network should work.


How to improve the reward signal when the rewards are sparse? In cases where the reward is delayed, this can negatively impact a models ability to do proper credit assignment. In the case of a sparse reward, are there ways in which this can be negated?
In a chess example, there are certain moves that you can take that correlate strongly with winning the game (taking the opponent's queen) but typically agents only receive a reward at the end of the game, so as to not introduce bias. The downside is that training in this sparse reward environment requires lots of data and training episodes to converge to something good.
Are there existing ways to improve the agent's performance without introducing too much bias to the policy?

Andrew Y. Ng (yes, that famous guy!) et al. proved, in the seminal paper Policy invariance under reward transformations: Theory and application to reward shaping (ICML, 1999), which was then part of his PhD thesis, that potential-based reward shaping (PBRS) is the way to shape the natural/correct sparse reward function (RF) without changing the optimal policy, i.e. if you apply PBRS to your sparse RF, the optimal policy associated with the shaped, denser RF is equal to the optimal policy associated with the original unshaped and sparse RF. This means that PBRS creates an equivalence class of RFs associated with the same optimal policy, i.e. there are multiple RFs associated with the same optimal policy. So, PBRS is the first technique that you can use to deal with sparse reward functions.
To give you more details, let $R\left(s, a, s^{\prime}\right)$ be your original (possibly sparse) RF for your MDP $M$, then
$$R\left(s, a, s^{\prime}\right)+F\left(s, a, s^{\prime}\right)\tag{1}\label{1}$$
is the shaped, denser RF for a new MDP $M'$.
In PBRS, we then define $F$ (the shaping function) as follows
$$
F\left(s, a, s^{\prime}\right)=\gamma \Phi\left(s^{\prime}\right)-\Phi(s),
\tag{2}\label{2}
$$
where $\Phi: S \mapsto \mathbb{R}$ is a real-valued function that indicates the desirability of being in a specific state. So, if we define $F$ as defined in \ref{2}, we call it a potential-based shaping function.
The intuition of $F$ in \ref{2} is that it avoids the agent to "go in circles" to get more and more reward. To be more precise, let's say that there's a state $s^* = s_t$ that is desirable but not the goal state. If you shaped the reward function by adding a positive reward (e.g. 5) to the agent whenever it got to that state $s^*$, it could just go back and forth to that state in order to get the reward without ever reaching the goal state (i.e. reward hacking). That's, of course, not desirable. So, if your current state is $s^*$ and your previous state was $s_{t-1}$, and whenever you get to $s_{t-1}$ you just shaped the reward function by adding a zero reward, to avoid going back to $s_{t-1}$ (i.e. to avoid that the next state $s_{t+1} = s_{t-1}$, and then go back to $s^*$ again, $s_{t+2} = s^*$, i.e. going in circles), if you use \ref{2}, you will add to your original sparse reward function the following
$$
F\left(s, a, s^{\prime}\right)=\gamma 0 - 5 = -5,
\tag{3}\label{3}
$$
In other words, going back to $s_{t+1}$ (to later try to go back to $s^*$) is punished because this could again lead you to go back to $s^*$. So, if you define $F$ as in equation \ref{2}, you avoid reward hacking, which can arise if you shape your RF in an ad-hoc manner (i.e. "as it seems good to you").
One of the first papers that reported this "going in circles" behaviour was Learning to Drive a Bicycle using Reinforcement Learning and Shaping

We agree with Mataric [Mataric, 1994] that these heterogeneous reinforcement functions have to be designed with great care. In our first experiments we rewarded the agent for driving towards the goal but did not punish it for driving away from it. Consequently the agent drove in circles with a radius of 20–50 meters around the starting point. Such behavior was actually rewarded by the reinforcement function

You probably also want to watch this video.
Another approach to solving the sparse rewards problem is to learn a reward function from an expert/optimal policy or from demonstrations (inverse reinforcement learning) or to completely avoid using the reward function and simply learn the policy directly from demonstrations (imitation learning).
Note that I'm not saying that any of these solutions fully solves all your problems. They have advantages (e.g. if done correctly, PBRS can speed the learning process) but also disadvantages (e.g., in the case of IRL, if your expert demonstrations are scarce, then your learned RF may also not be good), which I will not discuss further here (also because I don't currently have much practical experience with none of these techniques).


Is AI living or non-living? I'm a bit confused about the definition of life. Can AI systems be called 'living'? Because they can do most of the things that we can. They can even communicate with one another.

They are not formed of what we call cells. But, you see, cells are just a collection of several chemical processes which is in turn non-living just like AI is formed of several lines of code.

You're unsure about the definition of life (which the other answers clarify) but also most people are unclear about the definition of AI. Do you mean an AI that can accomplish a routine task (such as the path finder in a GPS) or a General AI that is able to find a creative solution to any directive given to it (such an AI does not yet exist and may not ever exist) or do you mean a SENTIENT computer program? Here is a simple article introducing some different concepts refered to as AI

Some people believe that a sentient computer program would be entitled to human rights. Not technically 'alive' in the biological sense, but having self awareness, will, desires, etc. Others disagree and believe that the program is a mere simulation that artificially mimics the actions of a human with a human soul, and is no more human than a washing machine. This is a very deep philosophical and meta-physical debate. For example, in A.I. the movie the overall message is that an android can simulate the emotion of love in a way that is more loyal and sincere than any human.

What I find interesting about this purely theoretical debate is that in almost every instance of sci-fi media that deals with the theme, the AI exists inside of a human-like android. But technically, the shape of the robot should be irrelevant.


How can we teach a neural net to make arbitrary data associations? Let's say I have pairs of keys and values of the form $(x_1, y_1), \dots, (x_N, y_N)$. Then I give a neural net a key and a value, $(x_i, y_i)$. For example, $x_i$ could be $4$ and $y_i$ could be $3$, but this does not have to be the case.
Is there a way to teach the neural net to output the $y_i$ variable every time it receives the corresponding $x_i$?
By the way, how do our brains perform this function?

In a nutshell : Memorizing is not Learning
So, first let's just remind the classical use of a neural net, in Supervised Learning :

You have a set of $(x_{train}, y_{train}) \in X \times Y$ pairs, and you want to extract a general mapping law from $X$ to $Y$
You use a neural net function $f_{\theta} : x \rightarrow f_{\theta}(x)$, with $\theta$ the weights (parameters) of your net.
You optimise $f_{\theta}$ by minimizing the prediction error, represented by the loss function.

Can this solve your question ? Well, I don't think so. With this scheme, your neural net will learn an appropriate mapping from the set $X$ to the set $Y$, but this mapping is appropriate according to your loss function , not to your $(x_{train}, y_{train})$ pairs.
Imagine that a small part of the data is wrongly labelled. A properly trained net learns to extract relevant features and thus will predict correctly the label, not like you did. So the net doesn't memorize your pairs, it infers a general law from the data, and this law may not respect each $(x_{train}, y_{train})$.
So classical Supervised Deep Learning should not memorize $(x_{train}, y_{train})$  pairs.
However, you could memorize using a net with too many parameters :  it's Overfitting !

In this case, you set up the net with too many parameters. That gives too much degrees of freedom to your net, and the net will use these DoFs to exactly fit rightly each $(x_{train}, y_{train})$  pair you feed during training.
However, for an input $x$ that it never saw during training, $f_{\theta}(x)$ would have no meaning. That's why we say an overfitted net did not learn, and overfitting is feared by many DL practitioner.

But as long as you want only to memorize, and not to learn, a overfitted net may be the a solution. An other solution for memorization may be Expert Systems, I don't know them enough to explain them, but you may check that if you want.
What about the brain ?
The matter in answering this question is that we don't really know how does the brain work. I highly recommend this article discussing neural networks and the brain.
Some thoughts to start :

The brain has an incredibly huge amount of parameters, and has a great plasticity. In that sense, we could draw a parallel with overfitted neural networks :  so the brain could be also able to overfit, and thus to memorize by this mean.
Our brain is not a feed forward network at all, we can't delimitate any layer, just some rough zones where we know that some specific information is processed. This makes any parallel between neural nets and the brain difficult.
It's still unclear how our brain updates itself. There's no backpropagation for instance. Our overfitted networks also stem from the update processes (for instance, adding regularization to the loss helps avoiding underfitting), but we have no idea of how this works in the brain, so that's another hurdle to drawing parallels !
A more personal thought : the brain is able to both learn and memorize ("The exception that proves the rule" motto shows that I think), while learning and memorizing are antonyms for neural nets...


Looping over Sarsa algorithm for better Q values Let's say an RL trading system places trades based on pricing data.
Each episode represents 1 hour of trading, and there are 24 hours of data available. The Q table represents for a given state, what is the most action with the highest utility.
The state is a sequence of prices, and the action is either buy, hold, sell.
Instead of "Loop for each episode" as per the Sarsa algorithm :

I add an additional outer loop. Now instead of just looping for each episode we have:
for 1 to N:
   "Loop for each episode"

Manually set N or exit out of the loop on convergence.
Is this the correct approach? Iterating multiple times over the episodes will produce more valuable state-action pairs in the Q table, because e greedy is not deterministic and for each iteration may exploit an action to greater reward than other episode epochs.

Given that your state space is continuous, then I would recommend using Deep Q-Learning. As you say, running several episodes will definitely be beneficial so that the agent is able to explore the space more thoroughly.


What is a simple game for validation of MCTS? What is a simple turn-based game, that can be used to validate a Monte-Carlo Tree Search code and it's parameters?

Before applying it to problems where I do not have a possiblity to validate its moves for correctness, I would like to implement a test case, that makes sure that it behaves as expected, especially when there are some ambiguities between different implementations and papers.

I built a connect-four game in which to MCTS-AI play against each other and an iterated prisoners dilemma implementation, in which a MCTS-AI plays against common strategies like Tit-for-Tat, but I am still not sure if there is a real good interpretation if the MCTS-AI finds the best strategy.

Another alternative would be a tic-tac-toe game, but MCTS will exhaust the whole search space within little steps, so it is hard to tell how the implementation will perform on other problems.

In addition, expanding a full game tree does not tell you if any states before the full expansion are following the best MCTS strategy.

Example:
You can alternate in the expand step of player 1's tree between optimize for player 1 and optimize for player 2, assuming that player 2 will not play the best move for player 1, but the best move for himself. Not doing so would result in an optimistic game tree, that may work in some cases, but probably is not the best choice for many games, while it would be useful for cooperative games.

When the game tree is fully expanded, you can find the best move, even when the order of the expand steps was not optimal, so using a game that can be fully expanded is no good test to validate the in-between steps.

Is there a simple to implement game, that can be used for validation, in which you can reliably check for each move, if the AI did find the expected move?

A good choice might be smaller-scale games of Go, like a 9x9 board. This was the original application domain MCTS was designed for, and the original paper by Brugmann from 1993 details parameters that should lead to an agent that can play above beginner level in what is today a minuscule amount of computational time, in a scaled-down 9x9 grid.

Go is a good choice for a benchmark because most learning algorithms fail at it pretty badly. The fact that MCTS worked here was a major breakthrough at the time, and helped cement it as a technique for game playing. If your algorithm is not working properly, it is therefore unlikely that it can learn to play Go at the level described in Brugmann's paper.


How would one implement a multi-agent environment with asynchronous action and rewards per agent? In a single agent environment, the agent takes an action, then observes the next state and reward:

for ep in num_episodes:
    action = dqn.select_action(state)
    next_state, reward = env.step(action)

Implicitly, the for moving the simulation (env) forward is embedded inside the env.step() function.

Now in the multiagent scenario, agent 1 ($a_1$) has to make a decision at time $t_{1a}$, which will finish at time $t_{2a}$, and agent 2 ($a_2$) makes a decision at time $t_{1b} < t_{1a}$  which is finished at $t_{2b} > t_{2a}$.

If both of their actions would start and finish at the same time, then it could easily be implemented as:

for ep in num_episodes:
    action1, action2 = dqn.select_action([state1, state2])
    next_state_1, reward_1, next_state_2, reward_2 = env.step([action1, action2])

because the env can execute both in parallel, wait till they are done, and then return the next states and rewards. But in the scenario that I described previously, it is not clear how to implement this (at least to me). Here, we need to explicitly track time, a check at any timepoint to see if an agent needs to make a decision, Just to be concrete:

for ep in num_episodes:
    for t in total_time:
       action1 = dqn.select_action(state1)
       env.step(action1) # this step might take 5t to complete.
       as such, the step() function won't return the reward till 5 t later.
        #In the mean time, agent 2 comes and has to make a decision. its reward and next step won't be observed till 10 t later.

To summarize, how would one implement a multiagent environment with asynchronous action/rewards per agents?

The cleanest solution from a theoretical point of view is to switch over to a hierarchical framework, some framework that supports temporal abstraction. My favourite one is the options framework as formalised by Sutton, Precup and Singh.

The basic idea is that the things that you consider "actions" for your agents become "options", which are "large actions" that may take more than a single primitive time step. When an agent selects an option, it will go on "auto-pilot" and keep selecting primitive actions at the more primitive, fine-grained timescale as dictated by the last selected option, until that option has actually finished executing. In your case, you could:

implement the first "primitive action" of an option to immediately apply all effects to the state, and append a sequence of "no-op" actions afterwards to make sure the option actually has a longer duration than a single primitive timestep, OR
implement the very last primitive action of an option to actually apply changes to the state, and prepend a sequence of "no-op" actions in front of it to make the option take more time, OR
something in between (i.e. actually make partial changes to the state visible during the execution of the option).

Since all legal choices for agents in your scenario appear to be options, i.e. you do not allow agents to select primitive actions at the more fine-grained timescale, you would only have to implement "inter-option" learning in your RL algorithms; there would be no need for "intra-option" learning.

In practice, if you only have a small number of agents and have options that take relatively large amounts of time, you don't have to actually loop through all primitive time-steps. You could, for example, compute the primitive timestamps at which "events" should be executed in advance, and insert these events to be processed into an event-handling queue based on these timestamps. Then you can always just skip through to the next timestamp at which an event needs handling. With "events" I basically mean all timesteps at which something should happen, e.g. timesteps where an option ends and a new option should be selected by one or more agents. Inter-option Reinforcement Learning techniques are basically oblivious to the existence of a more fine-grained timescale, and they only need to operate at precisely these decision points where one option ends and another begins.


Is there any research on models that provide uncertainty estimation? Is there any research on machine learning models that provide uncertainty estimation?

If I train a denoising autoencoder on words and put through a noised word, I'd like it to return a certainty that it is correct given the distribution of data it has been trained on.

Answering these questions or metrics for uncertainty are both things I am curious about. Just general ways for models to just say "I'm not sure" when it receives something far outside the inputs it's been trained to approximate.

Yes, there is some research on this topic. It's often called Bayesian machine learning or Bayesian deep learning (but I don't think this is a good name because there are models that aren't really based on a direct application of Bayesian statistics). Some ML/DL models that provide some kind of uncertainty estimation are, for example, Monte Carlo Dropout (MC dropout) or Bayesian neural networks. In theory, these techniques look promising. In practice, I don't think they are the ultimate solution to the problem of uncertainty estimation in deep learning. In fact, e.g. in the case of Bayesian neural networks, they have some disadvantages, such as more parameters to tweak and save.


How to design a good evaluation function for a go-like game? I am trying to find a good evaluation function for a game with:

A 7x7 tile board

2 players, given an equal number(>=3 currently undetermined) of stones placed randomly on the tiles

A turn is consisted of a player moving a stone owned by that player, vertically or horizontally but not diagonally to a very next tile of itself

A player loses when out of moves: a player is out of moves when every stone that player owns, has its very next tiles, except not for diagonals necessarily, occupied either by the board edge or other stones

Right now, my evaluation function's return value increases:

if the total moves available to the player is increasing, and/or

average distance to the middle tile of the board is decreasing

Question: Is there a better strategy? How can I improve my evaluation function?

If you have the best combination of distance between the stones, you should choose the best move to win. In this case, you have to be close to where your opponent plays. It is best to do this by surrounding your opponent's stones. You should always put the first stone in middle or corner of the table.


How can I use machine learning to predict properties (such as the area) of simple polygons? Imagine a set of simple (non-self-intersecting) polygons given by the coordinate pairs of their vertices $[(x_1, y_1), (x_2, y_2), \dots,(x_n, y_n)]$. The polygons in the set have a different number of vertices.

How can I use machine learning to solve various supervised regression and classification problems for these polygons such as prediction of their areas, perimeters, coordinates of their centroids, whether a polygon is convex, whether its centroid is inside or outside, etc?

Most machine learning algorithms require inputs of the same size but my inputs have a different number of coordinates. This may probably be handled by recurrent neural networks. However, the coordinates of my input vectors can be circularly shifted without changing the meaning of the input. For example, $$[(x_1, y_1), (x_2, y_2),...,(x_n, y_n)]$$ and $$[(x_n, y_n), (x_1, y_1),...,(x_{n-1}, y_{n-1})]$$ represent the same polygon where a starting vertex is chosen differently.

Which machine learning algorithm is both invariant to a circular shifting of its input coordinates and can work with inputs of different sizes?

Intuitively, an algorithm could learn to split each polygon into non-overlapping triangles, calculate areas or perimeters of each triangle, and then aggregate these computations somewhere in the output layer. However, the labels (areas or perimeters) are given only for the whole polygons, not for the triangles. Also, the perimeter of the polygon is not the sum of the perimeters of the triangles. Is thinking about this problem in terms of triangles misleading?

Could you please provide references on machine learning algorithms that solve such tasks? Or any advice, how to approach this task? It does not have to be neural network and does not have to learn exact analytic formulas. Approximate results would be enough.

You can split each polygon into a collection of triangles and sum up the areas. Not really sure why you would bother with ML.

Anyway if you approximates these polygons as images you could maybe train a CNN. Look at the image classification networks which provide bounding boxes.


How to measure the reasoning capabilities of neural networks Which possibilities exist to evaluate the visual reasoning capabilities of neural networks in the field of image recognition?

Are there methods to measure the ability of machine reasoning?

Or something more specific: Is it possible to measure if a network understood the concept of a car / a cat / a human without using the classification accuracy.

What do you mean by "understood the concept of a car" ? A classification algorithm need not know that cars are used to transport humans or anything else it's used for. It only needs to know what combination of features we decide to label as a car.

Or maybe you meant we could use some other metric for understanding that is not classification accuracy?

This is a kind of Catch-22 situation.

When we want to classify a car we can reformulate the task to learning what set of a combination of features do us humans decide to call a car. What labels we want to assign a combination of features is arbitrary to a degree. It preferably has to be  consistent for inference but an NN can learn a dataset assigned completely random features to a 100% accuracy.

For instance, perhaps in an alternative universe we label the front-view of a car as 'B' and the rear view as 'A' and the notion/label of 'car' is nonexistent: even though we are ostensibly referencing the same object in physical space, we can assign it different labels.

It follows that to measure if a model has properly interpolated what combination of features we like to call car, we ourselves must be able list out such features; and if we were able to do that, we would not not need the process of learning/teaching algorithms in the first place!

A much more interesting subject is testing the extrapolative powers of neural networks- something much more proximate to what we canonically refer to as 'reasoning'.

It's quite hard to measure this in static environments such as the classification of images, but the ability to extrapolate is more apparent in reinforcement learning settings where the model requires us to train it for times order of magnitudes above the time for needed for humans.

The whole notion of pristine reasoning in machine learning breaks down when it comes to induction(see sufficient reason). Thus, we resort to probabilities that can be roughly translated to "this line of reasoning yielded the correct result 99.9% of the time, so it would make sense that we keep using it.

In fact, the idea of perfect-as-possible induction has been mathematically formulated except it isn't computable :( Solomonoff Induction


How do neural networks weigh multiple inputs/features of different dimensionality? I am confused about how neural networks weigh different features or inputs.
Consider this example. I have 3 features/inputs: an image, a dollar amount, and a rating.  However, since one feature is an image, I need to represent it with very high dimensionality, for example with $128 \times 128 = 16384$ pixel values.  (I am just using 'image' as an example, my question holds for any feature that needs high dimensional representation: word counts, one-hot encodings, etc.)
Will the $16384$ 'features' representing the image completely overwhelm the other 2 features that are the dollar amount and rating?  Ideally, I would think the network would consider each of the three true features relatively equally.  Would this issue naturally resolve itself in the training process? Would training become much more difficult of a task?

As stated in your example, the three features are: an image, a price, a rating. Now, you want to build a model that uses all of these features and the simplest way to do is to feed them directly into the neural network, but it's inefficient and fundamentally flawed, due to the following reasons:

In the first dense layer, the neural network will try to combine raw pixel values linearly with price and rating, which will produce features that are meaningless for inference.

It could perform well just by optimizing the cost function, but the model performance will be nowhere as good as it could perform with a good architecture.

So, the neural network doesn't care if the data is a raw pixel value, price, or rating: it would just optimize it to produce the desired output. That is why it is necessary to design a suitable architecture for the given problem.
Possible architecture for your given example :

Separate your raw features, i.e. pixel value, and high-level data, i.e. price and rating

Stack 2-3 dense layers for raw features (to find complex patterns in images)

Stack 1-2 dense layers for high-level features

Combine them together in a final dense layer

If you want to de-emphasize the importance of the image, just connect the first dense layer 16,384 to another layer having fewer connections, say 1024, and have more connections from the high-level data, say 2048.
So, again, here's the possible architecture

Raw features -> dense layer (16384) -> dense layer (1024)
High-level features -> dense layer (2048)
Combine 1 and 2 with another dense layer


How can reinforcement learning be applied when the goal location or environment is unknown? I am studying RL. I was thinking whether a new state value or the observation is provided by the environment before the agent actually implements the action.
Take the maze problem as an example. Each state consists of all the available cells information, provided by the environment. But what if the environment is unknown? For example, there is a maze with an unknown destination cell. The agent needs to find the destination cell. The state is 1 or 0, meaning the destination reached or not. But the environment, which is the maze, can only provide the state at cell $i$ which is 0 or 1 only when the agent reaches cell $i$.
Can this still be solved by RL? I am confused about the environment setup.

When reading about RL and RL agents/algorithms, you always need to keep in mind that, typically, the RL agent/algorithm is trying to maximize the reward (or something equivalent, such as minimizing the regret) in the long run (i.e also the reward that you may receive in the future): that's its (mathematical) goal. Whether that  also corresponds to the high-level goal (e.g. reaching some physical location in some world) that the (human) designer of the so-called reward function (i.e. the function that gives the reward to RL agent) had in mind is a different story.
To maximize the reward, the RL agent interacts with the environment by taking actions, receiving rewards, and moving to other states. Initially, the RL agent does not know which actions lead to more rewards, so it may take random actions (this is known as exploration). Once it starts to understand the dynamics of the environment, it may start to take only the actions that lead to high reward (this is known as exploitation).
To answer your question directly, the RL agent can indeed take actions without knowing the dynamics of the environment. However, initially, it may need to take some random actions (which may lead to low rewards), so that to get more rewards in the long run.


If the reward function of an environment depends on some initial conditions, should I create a separate environment for each condition? I would like some guidance on how to design an Environment for a Reinforcement Learning agent where the stopping conditions and rewards for the environment change based on an initial set of input parameters.
For example, let's say that a system generated alert triggers the instantiation of the RL environment, whereby the RL agent is launched to make decisions in the environment, based on the alert. The alert has two priorities "HIGH" and "LOW", when the priority is "HIGH" the stopping condition is a reward of "100" and when the priority is "LOW", the stopping condition is a reward of "1000".
In this scenario, is it preferable to create two separate environments based on the priority (input parameter) of the alert? Or is this a common requirement that should be designed into the environment/agent? If so, how? Note that I have simplified the scenario, so there could be multiple conditions (e.g., alert, system type, etc), but I am just trying to find a basic solution for the general case.

In this scenario, is it preferable to create two separate environments based on the priority (input parameter) of the alert?

It is difficult to make a hard rule here.
If the resulting environments can be cleanly sorted into a few different categories, and the ideal behaviour and/or the states visited are radically different in each scenario, then maybe a few different agents optimised for each scenario could work well.
A more general approach however, is to include the episode start data as part of the state that the agent observes on each time step. A single agent can then in theory learn the different behaviours required depending on the initial values, plus still generalise from anything shared between the multiple scenarios.

The alert has two priorities "HIGH" and "LOW", when the priority is "HIGH" the stopping condition is a reward of "100" and when the priority is "LOW", the stopping condition is a reward of "1000".

This may work against you. RL agents do not respond to the absolute values of rewards, other than how they compare to other rewards also available within the same episode (or continuing environment).
If there is only ever one issue to solve at a time, and no conflict between solving either of the "HIGH" or "LOW" priority problems (such as splittig resources or effort between them), the different reward system seems redundant. Solved is solved. You might rate the usefulness of an agent that solves the "LOW" priority issue well higher, but it seems to me that this describes what you should work on first, not the goals of the agent. To influence the goals of the agent, both rewards would need to be available within the same episode or continuing environment, requiring the agent to make a choice between them.


Are machine learning and self-learning really possible? Some AI's, such as some chess players, are extremely well coded and have defeated humans in several matches. But I think that they won simply because computers can make calculations way faster than humans can not because they learned from their opponents.

If you put an AI against itself, who will win? Will the game continue indefinitely or will the game eventually finish because the AI plays randomly?

So, are machine learning and self-learning really possible?

Yes of course they are possible, I have built some! You are correct that there is an aspect of randomness to the process of machine learning but it is more accurate to describe this as trial and error. Each successive try in a machine learning system is evaluated against a goal and if it is an improvement or is closer to the goal, then this try is stored and some aspect that made it successful is incorporated into similar trys for this type of input. Therefore, machines learn by trying all possible combinations of a problem, albeit with some clever human described short cuts or "heuristics" to make the task easier.


How to know if the hyperparameters of a neural network relate to each other? According this thread some hyperparameters are independent from each other while some are directly related.

One of the answers give an example where two hyperparameters affect each other.

  For example, if you're using stochastic gradient descent (that is, you train your model one example at a time), you probably do not want to update the parameters of your model too fast (that is, you probably do not want a high learning rate), given that a single training example is unlikely to be able to give the error signal that is able to update the parameters in the appropriate direction (that is, the global or even local optimum of the loss function).

How would someone creating a neural network know how the hyperparameters affect each other?

In other words, what are the heuristics for hyperparameter selection when trying to build a robust model?

This is one of the most difficult and unsolved problems in machine learning and deep learning!

There are many different ways to estimate the most appropriate hyper-parameters, such as grid search, random search, Bayesian optimization, meta-learning, reinforcement learning, and evolutionary algorithms (e.g. NEAT).

However, the problem is that most if not all these approaches are typically not very computationally efficient if your model isn't anything but very small. The number of possible configurations of the hyper-parameters is very large. If you don't have good computational resources (e.g. GPUs and powerful servers), you probably out of luck or require some days to get some insight.

In certain cases, it is obvious that certain hyper-parameters are dependent on each other, e.g. in the case of the batch size and the learning rate, because we have some decent understanding of gradient descent, but, in other cases, the situation isn't so nice.

As far as I know, there isn't a very good general rule of thumb or method to solve this issue (i.e. find the dependence of hyper-parameters on each other). Maybe, as our knowledge of our models (especially, neural networks) increases, we'll get some more insights and we'll develop more efficient approaches to understand the dependence of the hyper-parameters.

Nowadays, there's automatical machine learning (AutoML), which is a fancy name to denote services that provide hyper-parameter optimization (plus some other stuff).