Can entire neural networks be composed of only activation functions? Inverse Reinforcement Learning based on GAIL and GAN-Guided Cost Learning(GAN-GCL), uses a discriminator to classify between expert demos and policy generated samples.
Adversarial iRL, build upon GAN-GCL, has its discriminator $D_{\theta, \phi}$ as a function of a state-only reward approximator $f_{\theta, \phi}$.
$$
D_{\theta, \phi}\left(s, a, s^{\prime}\right)=\frac{\exp \left\{f_{\theta, \phi}\left(s, a, s^{\prime}\right)\right\}}{\exp \left\{f_{\theta, \phi}\left(s, a, s^{\prime}\right)\right\}+\pi(a \mid s)},
$$
where $f_{\theta,\phi}$ is expressed as:
$$f_{\theta,\phi} = g_{\theta} (s) + γh_φ (s\prime ) − h_φ (s).$$
The optimal $*g(s)$ tries to recover the optimal reward function $r^*(s)$. While the $h(s)$ tries to recover the optimal value funtion $V^*(s)$, which makes $f_{\theta,\phi}$ interpretable as the advantage.
My question comes from the network architecture used for $h(s)$ in the original paper.

... we use a 2-layer ReLU network for the shaping term h. For the policy, we use a two-layer (32 units) ReLU gaussian policy.

What is meant by the quoted text in bold, because my interpretation of that text, (shown below) doesn't seem viable
h = nn.Sequential([nn.ReLu(), nn.ReLu()])

The Pytorch docs define a fully connected ReLU network as:
torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)

Neural networks are called are made of neurons. Activation functions only help determine which of these neurons to fire up, meaning they have no learnable nodes themselves through which we can back-propagate gradients. A network with no learnable parameters is therefore not a neural net. So neural nets can't be composed of activation functions only.

What is meant by the quoted text in bold, because my interpretation of that text, (shown below) doesn't seem viable

Yes, what's given here is not a network that can approximate the said function $h(s)$. A two layer ReLU network would resemble:
   x = nn.ReLU(nn.Linear(d_IN, H))
   x = nn.ReLU(nn.Linear(H, H))
   out = nn.Linear(x, d_OUT)

Another way to see it is that a network must have an input and ouput layer, and optional hidden layers. It's not possible to use an activation funtion as an input layer, because then you'll have no way of configuring the number of features to represent your input data. In this context, a ReLU can't represent the features of the observation input s.
To show activation functions have no learnable nodes in them and that this interpretation

h = nn.Sequential([nn.ReLu(), nn.ReLu()])

is not what the authors are driving across, here a script that counts the number of parameters in a network.

import torch.nn as nn
import numpy as np

activation = nn.ReLU

def count_params(module):
    return np.sum([np.prod(x.shape) for x in module.parameters()])

one_linear = nn.Sequential(nn.Linear(32, 10), nn.Linear(10, 1))
linear_act = nn.Sequential(nn.Linear(32, 10), activation(), nn.Linear(10, 1))
act_only = nn.Sequential(activation(), activation())

t_lin = count_params(linear_act)
lin = count_params(one_linear)
act = count_params(act_only)

print(f'Linear only: {lin}, Linear + Activation: {t_lin},' +
        f'Activation only: {act}')

[Out]: Linear only: 341, Linear + Activation: 341, Activation only: 0.0
The activation-function-only module has zero learnable parameters. Likewise, an activation function adds no parameters to the fully connected layers.
Update: Links to implementations
To confirm this answer's interpretation is correct, here are links to GAIL and GAN-GCL example implementations

GAIL : discriminator prediction (Forward call), discriminator architecture (The ReLU net):
GAN-GCL : discriminator prediction, discriminator architecture:


Are ReLUs incapable of solving certain problems? Background
I've been interested in and reading about neural networks for several years, but I haven't gotten around to testing them out until recently.
Both for fun and to increase my understanding, I tried to write a class library from scratch in .Net. For tests, I've tried some simple functions, such as generating output identical to the input, working with the MNIST dataset, and a few binary functions (two input OR, AND and XOR, with two outputs: one for true, one for false).
Everything seemed fine when I used a sigmoid function as the activation function, but, after reading about the ReLUs, I decided to switch over for speed.
Problem
My current problem is that, when I switch to using ReLUs, I found that I was unable to train a network of any complexity (tested from as few as 2 internal nodes up to a mesh of 100x100 nodes) to correctly function as an XOR gate.
I see two possibilities here:

My implementation is faulty. (This one is frustrating, as I've re-written the code multiple times in various ways, and I still get the same result).

Aside from being faster or slower to train, there are some problems that are impossible to solve given a specific activation function. (Fascinating idea, but I've no idea if it's true or not).

My inclination is to think that 1) above is correct. However, given the amount of time I've invested, it would be nice if I could rule out 2) definitively before I spend even more time going over my implementation.
More details
For the XOR network, I have tried both using two inputs (0 for false, 1 for true), and using four inputs (each pair, one signals true and one false, per "bit" of input). I have also tried using 1 output (with a 1 (really, >0.9) corresponding to true and a 0 (or <0.1) corresponding to false), as well as two outputs (one signaling true and the other false).
Each training epoch, I run against a set of 4 inputs $\{ (00, 0), (01, 1), (10, 1), (11, 0) \}$.
I find that the first three converge towards the correct answer, but the final input (11) converges towards 1, even though I train it with an expected value of 0.

There are a variety of possible things that could be wrong, but let me give you some potentially useful information.
Neural networks with ReLU activation functions are Turing complete for a computation with on order as many steps as the network contains nodes - for a recurrent network (an RNN), that means the same level of turing completeness as any finite computer. In other words, for any function/algorithm that you want to compute, you can devise a neural network, potentially recurrent, that will approximate/compute it.
As an example, suppose that we want to compute the NOR function, which can be used to implement a Turing machine.  We can do it with the following neural network with a ReLU activation function.
Let the input be
$$ W = \begin{bmatrix}x_1 \ \ x_2 \end{bmatrix}$$
the weight matrix be
$$ W = \begin{bmatrix} -20 \\ -20 \end{bmatrix}$$
and the bias be
$$ b = \begin{bmatrix} 1 \end{bmatrix}$$
Then the ReLU unit (or neuron) performs the following operation
$$o = \max(Wx + b, 0)$$
So, $o = 1$ only when both $x_1$ and $x_2$ are $0$, otherwise, it's always $0$.
However, gradient descent is a finicky way to search for RNNs. There are a wide variety of ways that it might have been failing. In general, once you have very thoroughly checked your gradient, I'd make sure to use Adam as the optimizer and then play with the hyperparameters endlessly until I find an incantation that works.
For further reading on general understanding of this level of deep learning's capability limitations, I'd recommend this blog post by Ilya Sutskever, now an OpenAI researcher: http://yyue.blogspot.com/2015/01/a-brief-overview-of-deep-learning.html?m=1


Does the human brain use beam search for text generation? As far as I understand, beam search is the most widely used algorithm for text generation in NLP. So I was wondering: does the human brain also use beam search for text generation? If not, then what?

I thought the answer might be no.

In this 2020 ICLR paper: The Curious Case of Neural Text Degeneration, researchers found that beam search text is less surprising compared to human natural language. And they proposed a nucleus sampling method which generates more human like text.


How are mean and variance calculated in Bayesian curve fitting? This page is from Pattern Recognition by Bishop. What is the explanation for equations 1.70. 1.71 and 1.72?

Edit: The distributions are all Gaussian.

Since you only mention that their Gaussian, here is a screenshot depicting the relationship of gaussian conjugate priors

I took this from wikipedia. If you check with what ever your likelihood and priors parameters are it probably all works out (you did not add these in the question)

From a quick google, here's a link to someone going through the proof: link


Steps for final Logistic Regression Modal I am new for machine learning and I am tried to understand basic steps to get final modal of Logistic Regression.

I know Logistic Regression is supervisory learning technique. Therefore we want to give training data to training the modal. According to my understanding, I can take steps to get the final modal as follows.

Step 1 - Make an algorithm. For Logistic Regression is y = sigmoid(W x + B) function. Get zero or some value for W and B.

Step 2 - Give sample known training data and find W and B values. x(1), x(2)...x(m) inputs and get y(1), y(2)...y(m) outputs. Gradient Descent use for find W and B that minimizes the cost function.

Step 3 - Then apply W and B values which I found.y^ = sigmoid(W x + B). And then again apply sample known training data to get the predicting of y^.

Step 4 - Get the final modal to test unknown data.

are these steps right? Please give me basic fundamental steps to understand supervisory learning technique. I would like to know un-supervisory learning technique steps also.

In general these steps are correct, but are few clarifications would be good. For supervised learning, you can:

pick model to use
split data into training and test set
train model, or determine weights by optimization of our error function (inputting in training data)
use trained model to predict on test data
utilize metrics of interest to evaluate our model (e.g. MSE for continuous outcomes, AUC for binary)

This however is a very elementary pipeline to train a supervised machine learning model. There are many other steps taken to ensure our model is as robust as possible (like cross validation, feature selection, hyperparameter optimization, model selection...). I would strongly recommend learning about these other steps and seeing how they integrate with the above pipeline.

For unsupervised learning, the process depends a lot on the task, but in general it involves optimizing some objective function using the data provided. This is usually done using all the data in one go (not split into training and test) unlike supervised learning. This can be seen more clearly with examples, take methods like PCA and k clustering; the steps taken to apply these methods very much depend on the method used.

Going further, I would recommend Andrew Ng's coursera course on machine learning. It is very accessible and a great introduction to this field, covering both unsupervised/supervised learning.


Do VR, AR and MR use any machine learning or deep learning? I wonder if Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR) use any machine learning or deep learning?

For example in AR, the virtual objects are brought into the real world, does this process involve any object detection and localization?

TL;DR

Yes for all the 3 cases

Details

VR

Even if the environment is virtual there is another thing to perceive which or better who is real: the user

Deep Learning can be used for things like: eye tracking, gesture recognition, voice recognition, ...

Furthermore, in games,  you could think about more complex applications like understanding the way the user plays the game and react accordingly

AR and MR

In addition to what has been said for VR, in these cases you also have to integrate the environment in the final user experience so you ultimately have to perceive it and there multiple specific applications: object detection, planes detection, depth estimation, Visual Odometry (e.g. as feature detection module), SLAM, ...

These days Deep Learning is achieving Starte-Of-The-Art (SOTA) in many different tasks far beyond the classification and detection, but one things is research so published papers and so on and another thing is a product
so to sum up every VR, MR, AR vendor is probably relying on an hybrid solution where traditional algorithms and deep learning coexist but so far it seems the trend is deep learning is eating the traditional algos up day after day

References

The rising force of deep learning in VR and AR

Virtual Reality And Machine Learning Go Hand In Hand


What are some datasets to train an MLP on simple tasks? I have implemented an MLP. Now, I want to train it to solve simple tasks.

Are there any data sets to train the MLP on simple tasks, that is, tasks with a small number of inputs and outputs?

I would like to train it to solve problems which are slightly more complex than the XOR problem.

There are a ton of sample datasets our there you can play with. A bunch of good ones install with R in the datasets package.  Luckily you can download them independently if you're not an R user.  Try  https://vincentarelbundock.github.io/Rdatasets/datasets.html

You might also be interested in the MNIST database which is one of the canonical databases used in handwriting recognition research.

Beyond that, you can look at / ask on http://datasets.reddit.com and/or http://opendata.reddit.com and you'll find all sorts of useful datasets.

And finally, don't overlook the UCI Machine Learning Repository.


Does this diagram represent several LSTMs, or one through several timesteps? I'm trying to read this paper describing Google's LSTM architecture for machine translation. It features this diagram on page 4:

I'm interested in the encoder block, on the left. Apparently, the pink and green cells are LSTMs. However, I can't tell if the x-axis is space or time. That is, are the LSTM cells on a given row all the same cell, with time flowing forward from left to right? The diagram on the next page in the paper seems to suggest that.

are the LSTM cells on a given row all the same cell, with time flowing forward from left to right?

Yes this is correct
The x-axis on this figure is basically the time axis. Essentially all pink boxes in the same row are the same LSTM cell, with different inputs from the same sequence. At each timestep, the cell takes an input and produces an output which is fed to the next layer. At the 8-th layer, the outputs over all timesteps are inputted at the same time to the attention layer.


When doing transfer learning, which initial layers do we need to freeze, and how should I change the last layer for my task? I want to train a neural network for the detection of a single class, but I will be extending it to detect more classes. To solve this task, I selected the PyTorch framework.
I came across transfer learning, where we fine-tune a pre-trained neural network with new data. There's a nice PyTorch tutorial explaining transfer learning. We have a PyTorch implementation of the Single Shot Detector (SSD) as well. See also Single Shot MultiBox Detector with Pytorch — Part 1.
This is my current situation

The data I want to fine-tune the neural network with is different from the data that was used to initially train the neural network; more specifically, the neural network was initially trained with a dataset of  20 classes

I currently have a very small labeled training dataset.

To solve this problem using transfer learning, the solution is to freeze the weights of the initial layers, and then train the neural network with these layers frozen.
However, I am confused about what the initial layers are and how to change the last layers of the neural network to solve my specific task. So, here are my questions.

What are the initial layers in this case? How exactly can I freeze them?

What are the changes I need to make while training the NN to classify one or more new classes?

After a lot of browsing online for answers to these questions, this is what I came up with.

What are the initial layers in this case? How exactly can I freeze them?

The initial few layers are said to extract the most general features of any kind of image, like edges or corners of objects. So, I guess it actually would depend on the kind of backbone architecture you are selecting.
How to freeze the layers depends on the framework we use.
(I have selected PyTorch as the framework. I found this tutorial Some important Pytorch tasks - A concise summary from a vision researcher, which seems to be useful.)

What are the changes I need to make while training the NN to classify one or more new classes?

I just need to change just the number of neurons (or units) in the final layer to the number of classes/objects your new dataset contains.


Putting constraints on output of deep neural network I am training a deep neural network. There is a constraint on an output value of the network. (e.g. Output has to be between 0 and 180) I think some possible solutions are using sigmoid,tanh activation at the end of the layer. I wonder if there are better ways to put constraints on the output value of a neural network.

There are many ways of constraining the network's output.
Using an activation layer is a good one. If you sigmoid the output layer, the output is constrained between [0,1] and you can multiply that by 180 to adapt to your output. Since the layer is part of the optimization process, gradients will be learned correctly.
However, there are a few issues with it, namely sigmoid is not a linear curve, and is quite steep around 0. Maybe you can consider something like a ReLU, but with a maximum as well. This makes your output linearly between [0,180], and is also learned during optimization.
You could also just clip the output without any activation layers. This is simple and works, but the optimization process sees no difference between outputting 180 or 180000. This can lead to problems, depending on your problem.


Why is the reward in reinforcement learning always a scalar? I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about the problem of assigning credit to an action for a reward. Wouldn't a vector reward make it easier for an agent to understand the effect of an action? Specifically, a vector in which different components represent different aspects of the reward. For example, an agent driving a car may have one reward component for driving smoothly and one for staying in the lane (and these are independent of each other).

Rather than the survey by Liu et al. recommended above, I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a much more useful introduction to this area)

Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67-113.

Liu et al's survey, in my opinion, doesn't do much more than list and briefly describe the MORL algorithms which existed at that point. There's no deeper analysis of the field. The original version of their paper was also retracted due to blatant plagiarism of several other authors, including myself as can be confirmed here: https://ieeexplore.ieee.org/document/6509978
Our survey provides arguments for the need for multiobjective methods by describing 3 scenarios where agents using single-objective RL may be unable to provide a satisfactory solution that matches the needs of the user. Briefly, these are

the unknown weights scenario where the required trade-off between the objectives isn't known in advance, and so to be effective the agent must learn multiple policies corresponding to different trade-offs and then at run-time select the one which matches the current preferences (e.g. this can arise when the objectives correspond to different costs which vary in relative price over time);

the decision support scenario where scalarization of a reward vector is not viable (for example, in the case of subjective preferences, which defy explicit quantification), so the agent needs to learn a set of policies, and then present these to a user who will select their preferred option, and

the known weights scenario where the desired trade-off between objectives is known, but its nature is such that the returns are non-additive (i.e. if the user's utility function is non-linear), and therefore standard single-objective methods based on the Bellman equation can't be directly applied.

We propose a taxonomy of MORL problems in terms of the number of policies they require (single or multi-policy), the form of utility/scalarization function supported (linear or non-linear), and whether deterministic or stochastic policies are allowed, and relate this to the nature of the set of solutions which the MO algorithm needs to output. This taxonomy is then used to categorise existing MO planning and MORL methods.
One final important contribution is identifying the distinction between maximising Expected Scalarised Return (ESR) or Scalarised Expected Return (SER). The former is appropriate in cases where we are concerned about the results within each individual episode (for example, when treating a patient - that patient will only care about their own individual experience), while SER is appropriate if we care about the average return over multiple episodes. This has turned out to be a much more important issue than I anticipated at the time of the survey, and Diederik Roijers and his colleagues have examined it more closely since then (e.g. http://roijers.info/pub/esr_paper.pdf)


Best ways of leveraging AI for stock market trading What are the current popular approaches to leveraging AI for stock price prediction? It seems like there could be several approaches and problem formulations:

Supervised learning:

Regression: predict the stock price directly
Classification: predict whether the stock price goes up or down

Unsupervised learning: find clusters of stocks that move together
Reinforcement learning: let the agent directly maximize its stock market return
Other AI methods: rules, symbolic systems etc.

Which are most popular/performant? Are there other ways that people are using machine learning in stock trading (sentiment analysis on financial statments, news etc.).

Your list is complete for what is considered 'popular' by most practitioners who apply AI for stock trading.  Supervised learning and rule learning are at the top for accuracy.  There are more academic papers published on classifiers than on regression approaches;
classifiers are typically more accurate than regressors.


Feasibility of a team-based FPS AI? We have seen advances in top down, RTS team games like Dota 2 and Starcraft II from companies like OpenAI who developed agents to beat real pro players most of the time. How would similar learning techniques compare to games like Overwatch that require faster reaction times and complex understanding of 3d space and effect?
Or have we not developed solutions that could be tasked with this problem?

To get a better understanding, let's compare a couple of games DOTA2 and Overwatch. At first, both seen similar in terms of multiplayer and strategy, but there is a significant difference.

You have mentioned it in the question. reaction time

Even though both the games require strategy and understanding of the surroundings and team play, FPS games are heavily dependent on reaction time and accuracy. As much as you try, it's next to impossible to reach the reaction times and accuracy of computer agent.

In games such as DOTA2, the game is more dependent towards strategy and how the characters interact with each other (Strengths and Weakness).

Finally, it doesn't make sense to train an AI based agent to play FPS games, because the optimal solution is already present and there is no way that AI can reach those levels. It's called AIM BOT. As good as you can get at strategy and planning, team work and reaction times, you can almost never beat it. It's next to 100% accurate and the opponent is basically dead as soon as he's in sight.


Why aren’t heuristics for Connect Four Monte Carlo tree search improving the agent? I’ve created an agent using MCTS to play Connect Four. It wins against humans pretty well, but I’d like to improve upon it. I decided to add domain knowledge to the MCTS rollout stage. My evaluation function checks how “good” an action is and returns the best/highest value action to the rollout policy as the action to use.
I created a “gym” application for one agent, who’s not using the evaluation function, to play against an agent who is using the evaluation function.
I would have expected the agent using the heuristics to perform better than the agent who isn’t, but the inclusion of the heuristics doesn’t seem to make any difference! Any ideas why this might be the case?

It might be the case that if you perform a large number of random rollouts, the "best action" as chosen by the agent without the domain knowledge, is same as the agent with the domain knowledge. I guess what you can do is try to reduce the number of rollouts and see if the performance changes.


Why do we need Upsampling and Downsampling in Progressive Growing of Gans I was working recently on Progressive Growing of GANs (aka PGGANs). I have implemented the whole architecture, but the problem that was ticking my mind is that in simple GANs, like DCGAN, PIX2PIX, we actually use Transposed Convolution for up-sampling and Convolution for Down-sampling, but in PGGANs in which we gradually add layers to both generator and discriminator so that we can first start with 4x4 image and then increase to 1024x01024 step by step.

I did not understand that once we Increase 1x1x512 dimensional latent vector size to 4x4x512 sort of image we use convolution with high padding, and then once training for 4x4 images, we take still take 512 latent vector and then use the previously trained convolutional layers to convert it to 4x4x512 image, and then we up-sample then given image to 8x8 using nearest neighbor filtering and then again apply convolution and so-on.

My question is that why we need to explicitly up-sample and then apply convolution, when instead we could just use Transposed Convolution which can upsample it automatically and is trainable? Why do we not use it like in other GANs?

Here is the image of architecture:

Please explain me the intuition behind this.
Thanks

Use of Transposed Convolution can lead to checkerboard artifacts.  So we prefer to up-sample and then apply convolution. You can check this article for more information https://distill.pub/2016/deconv-checkerboard/.


Why is GPT-3 such a game changer? I've been hearing a lot about GPT-3 by OpenAI, and that it's a simple to use API with text in text out and has a big neural network off 175B parameters.
But how did they achieve this huge number of parameters, and why is it being predicted as one of the greatest innovations?

The main point in GPT-3 and already in 2 was the observation that performance was steadily increasing with increasing model size (As seen in Figure 1.2 in your linked paper). So it seems that while all progress made in NLP was definitely useful, it also is important to just scale up the model size.
This may not seem like surprising point, but it actually kind of is. Normally, performance would saturate or at least the gain would slope off, but this is not the case! So the main innovation may not be that big and is kind off brute-force but the point still stands: Bigger models are better.
Another point to mention is the way they did the training. Such a large model needs some tricks to be actually trained (and fast at that). You also want to make use of multiple GPUs for parallel training. This means they also had to develop new structures for training.
Why exactly it is predicted as a huge innovation may only be contained to some twitter demonstration, there are no real sources on this as far as I know. Especially because the model is not openly available.


Are there neural networks with very few nodes that decently solve non-trivial problems? I'm interested in knowing whether there exist any neural network, that solves (with  >=80% accuracy) any nontrivial problem, that uses very few nodes (where 20 nodes is not a hard limit). I want to develop an intuition on sizes of neural networks.

Even if it’s impossible to answer this question properly, as non trivial is not well defined (maybe the author will edit this questions later, to specify it better), I take the opportunity to point out this paper which looks interesting to me

Smallest Neural Network to Learn the Ising Criticality

Assuming you have a general idea of the Ising Model I think the problem of identifying the critical temperature from a data driven perspective can be considered as non trivial and the paper shows how the authors have improved the performance related to solve this task with NN passing from 100 Hidden Neurons, as performed in this paper Machine learning phases of matter from 2017, to only 2 Hidden Neurons

Just my cents:

reducing the neurons, while keeping good performance, should help in terms of neural processing interpretability which is notoriously obscure and its complexity grows (exponentially) with the number of neurons


How to use validation dataset in my logistic regression model? I am new to machine learning and recently I joined a course where I was given a logistic regression assignment in which I had to split 20% of the training dataset for the validation dataset and then use the validation dataset to capture the minimum possible loss and then use the test dataset to find the accuracy of the model.
Below is my code for implementing logistic regression
class LogReg(LinReg):
    def __init__(self, n_dim, bias=True):
        if bias:
            n_dim = n_dim + 1
        super(LogReg, self).__init__(n_dim)
        self.bias = bias

  def __call__(self, x):
      return x.mm(self.theta).sigmoid()

  def compute_loss(self, x, y, lambda_reg):
      # The function has a generic implementation, and can also work for the neural nets!
      predictions = self(x)
      loss = -(y * torch.log(predictions) + (1-y) * torch.log(1 - predictions)).mean()
      regularizer = self.theta.transpose(0, 1).mm(self.theta)
      return loss + regularizer.mul(lambda_reg)
  @staticmethod
  def add_bias(x):
      ones = torch.ones((x.size(0), 1), dtype=torch.float32)
      x_hat = torch.cat((ones, x), dim=-1)
      return x_hat

  def fit(self, x, y, num_iter=10, mb_size=32, lr=1e-1, lambda_reg=1e-2, reset=True):
      N = x.size(0)
      losses = []
      x_hat = x
      # Adding a bias term if needed
      if self.bias:
          x_hat = self.add_bias(x)
      if reset:
          self.reset() # Very important if you want to call fit multiple times
      num_batches = x.size(0) // mb_size
      # The outer loop goes over `epochs`
      # The inner loop goes over the whole training data
      for it in range(num_iter):
          loss_per_epoch = 0
          for batch_it in range(num_batches):
              # has been implemented for the linear model
              self.zero_grad()

              ind = torch.randint(0, N, (mb_size, 1)).squeeze()
              x_mb, y_mb = x_hat[ind, :], y[ind, :]

              loss = self.compute_loss(x_mb, y_mb, lambda_reg)

              loss.backward()
              self.theta.data = self.theta.data - lr*self.grad().data
              loss_per_epoch += loss.item()

          loss_per_epoch /= num_batches
          losses.append(loss_per_epoch)

      return losses

How should I use the validation set at the level of epoch to find the best loss?

So generally, when you seperate your training data to 80%-20% then you fit method should get 2 x,y. better to call them x_train,y_train, x_val, y_val or something similar.
Now its important you do the split before entering the fit, and not do it for each epoch or something alike.
Once you do that and the fit method should be something like:
def fit(self, x_train, y_train, x_val, y_val, num_iter=10, mb_size=32, lr=1e-1, lambda_reg=1e-2, reset=True):

Then you should, at the end of each epoch, test the performance of the model on the validation set entirely and calculate the desired metric for evaluation. If you improved it's better to save the current model. This is done repeatedly for each epoch until the end of the training and you will guarantee to have the model who gave you the best results on the validation set rather than on the training set, which might be overfitting it.
I will do it in a separate method with the following flow:

iterate on each sample in the validation set
for each one calculate the loss/metric
append it to some list
return the mean of that list

And if the average result is better from some previously saved one, save the new model


How does a neural network that has been trained keep learning while in a real world scenario Say I trained a Neural Network (not RNN or CNN) to classify a particular data set.
So I train using a specific data set & then I test using another and get an accuracy of 95% which is good enough.
I then deploy this model in a production level environment where it will then be processing real world data.
My question is, will this trained NN be constantly learning even in a production scenario? I can't figure out how it will because say it processes a dataset such as this:
[ [1,2,3] ] and gets an output of [ 0, 0.999, 0 ]
In a training scenario it will compare the predicted output to the actual output and back propagate but in a real world scenario it will not know the actual value.
So how does a trained model learn in a real world scenario?
I am still very much a beginner in this field and I am not sure if the technology used is going to affect the answer to this question, but I am hoping to use Eclipse Deeplearning4J to create a NN. That being said the answer does not need to be restricted to this technology in particular as I am hoping more for the theory behind it and how it works.

You are right. If you don't continuously train the neural network after you have deployed it, there is no way it can continuously learn or be updated with more information. You need to program the neural network to learn even after it has been deployed. There is no such thing as a neural network that decides what it does without a human deciding first what it needs to do: this is a very common misconception (probably caused by the media and science fiction movies). It's also true that you need to label your data if you intend to train the neural network in a supervised fashion, but there are other ways to train neural networks (e.g. by reinforcement learning), depending also on the problem you want to solve.
If you want to develop neural networks that can learn continually, you probably want to look into continual learning techniques for neural networks. Another term that you may be looking for is online machine learning.


Why does the error of my LSTM not decrease after 10 epochs? Despite the problem being very simple, I was wondering why an LSTM network was not able to converge to a decent solution.
import numpy as np
import keras

X_train = np.random.rand(1000)
y_train = X_train
X_train = X_train.reshape((len(X_train), 1, 1))

model= keras.models.Sequential()
model.add(keras.layers.wrappers.Bidirectional(keras.layers.LSTM(1, dropout=0., recurrent_dropout=0.)))
model.add(keras.layers.Dense(1))

optimzer = keras.optimizers.SGD(lr=1e-1)

model.build(input_shape=(None, 1, 1))
model.compile(loss=keras.losses.mean_squared_error, optimizer=optimzer, metrics=['mae'])
history = model.fit(X_train, y_train, batch_size=16, epochs=100)

After 10 epochs, the algorithm seems to have reached its optimal solution (around 1e-4 RMSE), and is not able to improve further the results.
A simple Flatten + Dense network with similar parameters is however able to achieve 1e-13 RMSE.
I'm surprised the LSTM cell does not simply let the value through, is there something I'm missing with my parameters? Is LSTM only good for classification problems?

I think there are some problems with your approach.
Firstly, looking at the Keras documentation, LSTM expects an input of shape (batch_size, timesteps, input_dim). You're passing an input of shape (1000, 1, 1), which means that you're having "sequences" of 1 timestep.
RNNs have been proposed to capture temporal dependencies, but it's impossible to capture such dependencies when the length of each series is 1, and the numbers are randomly generated. If you want to create a more realistic scenario, I would suggest you generate a sine wave, since it has a smooth periodic oscillation. Afterward, increase the timesteps from 1, and you can test on the following timestamps (to make predictions).
For the second part, if you think about a normal RNN (I will explain for a simple RNN but you can imagine a similar flow for LSTM) and a Dense layer when applied to 1 timestamp, there are not so many many differences. The dense layer will have $Y=f(XW + b)$, where $X$ is the input, $W$ is the weight matrix, $b$ is the bias and $f$ is the activation function. Whereas RNN will have $Y=f(XW_1 + W_2h_0 + b)$, since is the first timestamp $h_0$ is $0$, so we can reduce it to $Y=f(XW_1 +b)$, which is identical with the Dense layer. I suspect that the results differences are caused by the activation functions, by default Dense layer has no activation function, and LSTM has tanh and sigmoid.


Is there an ideal range of learning rate which always gives a good result almost in all problems? I once read somewhere that there is a range of learning rate within which learning is optimal in almost all the cases, but I can't find any literature about it. All I could get is the following graph from the paper: The need for small learning rates on large problems

In the context of neural networks trained with gradient descent, is there a range of the learning rate, which should be used to reduce the training time and get a good performance in almost all problems?

The 2015 article Cyclical Learning Rates for Training Neural Networks by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate.
The paper's primary focus is the benefit of using a learning rate schedule that varies learning rate cyclically between some lower and upper bound, instead of trying to choose a single fixed learning rate value. For this to work, you still need to select good lower and upper bounds, and Smith suggests training the model for a few epochs while increasing the learning rate between a large range of values. At first, the learning rate will be too small to make any progress at all. As the learning rate increases, eventually, the loss will begin to decrease, but, at some point, the learning rate will get too large, and the loss will stop decreasing and even begin increasing.  Your ideal range consists of the learning rate values where the loss was decreasing steeply.  After finding your range, you can reset the weights and biases on your model and restart training using whatever learning rate schedule you plan to use for training.
Here is a concrete example from one of my experiments:

In this case, I start my learning rate search at 1e-09 and plan to end with a learning rate of 0.99 (although I am actually able to stop sooner than that).  Your experiment may require different search bounds, but you could always start with that and adjust things as needed.  At first, the loss plot is flat, and then it begins to decrease, but is too gradual.  At the first red line, loss starts to decrease sharply, and once it reaches the second red line, the plot has begun to level off, so I can end my search.  For this particular experiment, my ideal learning rate range had a minimum of 4.01e-4 and a maximum of 2.58e-2.
For more information, I suggest reading this Keras Learning Rate Finder post, which contains more information on how the process works and a tutorial for how to program it using Keras and Tensorflow.


Viterbi versus filtering In Chapter 15 of Russel and Norvig's Artificial Intelligence -- A Modern Approach (Third Edition), they describe three basic tasks in temporal inference:

Filtering,
Likelihood, and
Finding the Most Likely Sequence.

My question is on the difference between the first and third task.  Finding the Most Likely Sequence determines, given evidences $e_1,\dots,e_n$, the most likely sequence of states $S_1,\dots,S_n$.  This is done using the Viterbi algorithm.  On the other hand, Filtering provides the probability distribution on states after seeing $e_1,\dots,e_n$.  You could then pick the state with the highest probability, call it $S'_n$. I am guessing that $S'_n$ should always be equal to $S_n$.  Likewise, you can already do the same after any prefix $e_1,\dots,e_i$, again picking the most likely state $S'_i$.  I would love to have a simple example where $S'_1,\dots,S'_n$ is not equal to the sequence $S_1,\dots,S_n$ produced by the  Viterbi algorithm.

Welcome to AI.SE @vdbuss, and great first question!

This point is touched on in Section 15.2.3 (page 576 in my copy), in the second paragraph, and there's a good exercise at the end of the chapter (15.4) that is designed to get you to think through exactly why these are different procedures. If you want to really absorb it, I suggest trying to work out that exercise! If you want the quick answer, read on.

The basic action of filtering is to generate a probability distribution $P(X_{t+1} | e_{1:t+1})$ using only two pieces of information, specifically the current state distribution $P(X_{t} | e_{1:t})$, and the new piece of evidence $e_{t+1}$. So, when computing the most likely sequence, the algorithm cannot take into account the sequences that are actually possible, while Vitirbi can.

Here's a simple example: suppose I tell you that I'm going to drop you in a maze at one of two locations. I drop you near the top right corner with probability 0.75, and near the bottom left corner with probability 0.25. Suppose further that a Grue is known (with certainty) to live somewhere near the bottom left corner. Using filtering, your maximum aposterori estimate for your location after being dropped in the maze ($t=1$) is that you are in the top right corner. You then move 1 step to the right and can see a Grue. Clearly your estimate for your position in the second timestep $(t=2)$ must be the bottom left, because Grues only live there. But, you definitely can't end up moving to the bottom left by moving right from the top right, so your sequence has probability zero overall, despite using the maximum aposterori estimate for position at every step. To avoid this, Vitirbi uses a linear amount of extra space to select the maximum aposterori sequence, which in this case is clearly that you are near the bottom left in both timesteps.


How does RL based neural architecture search work? I have read through many of the papers and articles linked in this thread but I haven't been able to find an answer to my question.

I have built some small RL networks and I understand how REINFORCE works. I don't quite understand how they are applied to NAS though. Usually RL agents map a state to an action and get a reward so they can improve their decision making (which action to choose). I understand that the reward comes from the accuracy of the child network and the action is a series of digits encoding the network architecture.

What is passed as the state to the RL agent? This doesn't seem to be mentioned in the papers and articles I read. Is it the previous network? Example input data?

(I will repeat a few details that you're already aware of, so that other users can also understand the context).

In the Neural Architecture Search (NAS) paper (that I mention in my answer to the question you link to in your question), the agent is the controller (see also this question Is there any difference between a control and an action in reinforcement learning?), which is implemented as a recurrent neural network (RNN). This controller produces actions (or controls), which are strings that represent the hyper-parameters of a neural network (see e.g. section 3.2), based on the reward that it receives, which, in this case, is the accuracy on the validation dataset that the designed (by the controller) and trained neural network obtains.

In this context, the controller is thus the reinforcement learning agent (or policy). The controller is an RNN that is represented by a vector of parameters, $\boldsymbol{\theta}_\text{c}$. This controller is trained using a policy gradient method to maximize the expected accuracy on the validation dataset (see e.g. section 3 of the paper). So $\boldsymbol{\theta}_\text{c}$ are adjusted using this policy gradient method so that the controller generates NN architectures that, after trained on some task, produce higher accuracy on a validation dataset.

The objective function of the controller is thus

$$
J(\boldsymbol{\theta}_\text{c}) = \mathbb{E}[R]
$$

where $R$ is the accuracy on the validation dataset. Why expected? You will be performing this operation multiple times, so, intuitively, you want an average of the accuracy that you obtain on the validation dataset using multiple neural network architectures that the controller might produce.

The accuracy on the validation dataset, represented by $R$, is actually non-differentiable, so the authors use the famous REINFORCE algorithm (which I will not explain here). The authors actually use an approximation of the REINFORCE algorithm (see section 3.2).

$$
\frac{1}{m} \sum_{k=1}^m \sum_{t=1}^T \nabla_{\boldsymbol{\theta}_\text{c}} \log P (a_t \mid a_{{t-1}:1}; \boldsymbol{\theta}_\text{c}) R_k
$$

So, given this formulation and taking into account the formulation of the REINFORCE algorithm, then the state of the agent seems to be $a_{{t-1}:1}$, that is, the previous actions of the agent. (Recall that the agent is a controller that is implemented as a recurrent neural network).


How do transformers understand data and answer custom questions? I recently heard of GPT-3 and I don't understand how the attention models and transformers encoders and decoders work. I heard that GPT-3 can make a website from a description and write perfectly factual essays. How can it understand our world using algorithms and then recreate human-like content? How can it learn to understand a description and program in HTML?

GPT-3 (and the likes) don't really have any understanding of the semantics nor pragmatics involved in the language. However, they are good at constructing text content similar to the contents created by a person (when the texts and the concepts are not too "complicated").


How can we estimate the transition model and reward function? In reinforcement learning (RL), there are model-based and model-free algorithms. In short, model-based algorithms use a transition model (e.g. a probability distribution) and the reward function, even though they do not necessarily compute (or estimate) them. On the other hand, model-free algorithms do not use such a transition model or reward function, but they directly estimate e.g. the state or state-action value functions by interacting with the environment, which allows the agent to infer the dynamics of the environment.
Given that model-based RL algorithms do not necessarily estimate or compute the transition model or reward function, in the case these are unknown, how can they be computed or estimated (so that they can be used by the model-based algorithms)? In general, what are examples of algorithms that can be used to estimate the transition model and reward function of the environment (represented as either an MDP, POMDP, etc.)?

Given that model-based RL algorithms do not necessarily estimate or compute the transition model or reward function, in the case these are unknown, how can they be computed or estimated (so that they can be used by the model-based algorithms)?

A generally reliable approach to creating learned models from interacting with the environment, then using those models internally for planning or explicitly model-based learning, is still something of a holy grail in RL. An agent that can do this across multiple domains might be considered a significant step in autonomous AI. Sutton & Barto write in Reinforcement Learning: An Introduction (Chapter 17.5):

More work is needed before planning with learned models can be effective. For example,the learning of the model needs to be selective because the scope of a model strongly affects planning efficiency. If a model focuses on the key consequences of the most important options, then planning can be efficient and rapid, but if a model includes details of unimportant consequences of options that are unlikely to be selected, then planning may be almost useless. Environment models should be constructed judiciously with regard to both their states and dynamics with the goal of optimizing the planning process. The various parts of the model should be continually monitored as to the degree to which they contribute to, or detract from, planning efficiency. The field has not
yet addressed this complex of issues or designed model-learning methods that take into account their implications.

[Emphasis mine]
This was written in 2019, so as far as I know still stands as a summary of state-of-the-art. There is ongoing research into this - for instance the paper Model-Based Reinforcement Learning via Meta-Policy Optimization considers using multiple learned models to assess reliability. I have seen a similar recent paper which also assesses reliability of the learned model and chooses how much it should trust it over a simpler model-free prediction, but cannot recall the name or find it currently.
One very simple form of learned model is to memorise transitions that have been experienced already. This is functionally very similar to the experience replay table used in DQN. The classic RL algorithm for this kind of model is Dyna-Q, where the data stored about known transitions is used to perform background planning. In it's simplest form the algorithm is almost indistinguishable from experience replay in DQN. However, this memorised set of transition records is a learned model, and is used as such in Dyna-Q.
The basic Dyna-Q approach creates a tabular model. It does not generalise to predicting outcomes from previously unseen state, action pairs. However, this is relatively easy to fix - simply feed experience so far as training data into a function approximator and you can create a learned model of the environment that attempts to generalise to new states. This idea has been around a long time. Unfortunately, it has problems - planning accuracy is strongly influenced by the accuracy of the model. This applies for both background planning and looking forward from current state. Approximate models like this to date typically perform worse than simple replay-based approaches.
This general approach - learn the model statistically from observations - can be refined and may work well if there is any decent prior knowledge that restricts the model. For example if you want to model a physical system that is influenced by current air pressure and local gravity, you could have free parameters for those unknowns starting with some standardised guesses, and then refine the model of dynamics when observations are made, with strong constraints about the form it will take.
Similarly in games of chance with hidden state, you may be able to model the unknowns within a broader well-understood model, and use e.g. Bayesian inference to add constraints and best guesses. This is typically what you would do for a POMDP with a "belief state".
Both of the domain-specific approaches in the last two paragraphs can be made to work better than model-free algorithms alone, but they require deep understanding/analysis of the problem being solved by the researcher to set up a parametric model that is both flexible enough to match the environment being learned, but also constrained enough that it cannot become too inaccurate.


Would minimizing influence into the world be a safe directive to a general AI? Let's take our standard paperclip maximizer General AI and attempt to obtain precisely one million paper clips, over course of a year, without destroying the universe in the process.

Most maximization directives make the process run-away. As cheaply as possible will crash world economy. As good clips as possible will turn the universe into super-synthesizer that assembles atom-perfect paperclips. Adding a deadline on these maximization processes will probably result in terrorizing the staff into readjusting the deadline, or invention of time travel (after consuming the solar system to invent it.) Minimizing resource usage would likely result in closure of all industry world-wide. You know, the standard horror scenarios.

What about the directive of minimizing AI's influence on the world while completing the task? Would it be safe, or can you spot a runaway scenario where it could result in dire effects?

I like mindrime's answer as it identifies fundamental, applicable notions.  I'll attempt another short answer from a Game Theory perspective.

Game Theory was founded in the minimax principle.  Specifically, maximizing benefit while minimizing potential downside in a condition of uncertainty.

Minimax works very well in contexts with easily definable parameters, such as combinatorial games and procedural optimization, but it gets more tricky in real-world scenarios where too many parameters may be present, resulting in a combinatorial explosion.

Another issue regarding real-world applications arises from the symbol grounding problem.  In the paper clip scenario, the benefit sought (the goal) is clearly definable, and can be expressed mathematically.  By contrast, the downsides to be avoided are more difficult to define: "Don't destroy the world", "Don't use up resources to a degree that humans suffer", "Don't harm the environment" all rely on language.  This is to say they rely on terms that, at present, constitute symbols that cannot be grounded.  Thus there is room for misinterpretation that may yield unintended consequences.

"Minimizing impact on the system" (minimizing influence in the world) would be the goal, but how do you guarantee an automata has a clear understanding of what this entails in every possible scenario?


Is there any way to classify Document Image without OCR? I have multiple invoices images which need to classify invoice types such as fright, utility, goods, etc. Is there any way to classify without OCR?

It is possible to classify invoice scans without passing through an OCR component if they are visually different (they demonstrate different visual features). On the other hand, if the invoices look very similar, then the classifier might not be very accurate.

Another challenge would be the number of images you need to train a deep network for image classification (you may start with pretrained models and only perform the finetuning of you do not have enough images). On the other hand, the combination of pretrained OCR models and NLP-based document classifier may not need that many samples for training (for this specific task).


Why state-action value function as an expected value of the return and state value function, does not need to follow policy? I often see, the state-action value function is expressed as:

$q_{\pi}(s,a)=\mathbb{E}_{\pi}[R_{t+1}+\gamma G_{t+1} | S_t=s, A_t = a] = \mathbb{E}[R_{t+1}+\gamma v_{\pi}(s') |S_t = s, A_t =a]$

Why does expressing the future return in the time $t+1$ as a state value function $v_{\pi}$ make the expected value under policy change to expected value in general?

Let's first write the state-value function as
$$q_{\pi}(s,a) = \mathbb{E}_{s_{t},r_{t} \sim E,a_t \sim \pi}[r(s_t,a_t) + \gamma G_{t+1} | S_t = s, A_t = a]\; ;$$
where $r(s_t,a_t)$ is written to show that the reward gained at time $t+1$ is a function of the state and action tuple we have at time $t$ (note here that $G_{t+1}$ would be just the sum of future reward signals). This allows us to show that the expectation is taken under the joint distribution of $s,r\sim E$ where $E$ is the environment and actions are taken from our policy distribution.
As we have conditioned on knowing $a_t$ then the only random variable in the expectation that is dependent on our policy $\pi$ is $G_{t+1}$ because this is the  sum of future reward signals and so will depend on future state-action values. Thus, we can rewrite again as
$$q_{\pi}(s,a) = \mathbb{E}_{s_{t},r_{t} \sim E}[r(s_t,a_t) + \gamma \mathbb{E}_{a_t\sim \pi}[ G_{t+1} |S_{t+1} = s'] | S_t = s, A_t = a]\;,$$
where the inner expectation (coupled with the fact its inside an expectation over the state and reward distributions) should look familiar to you as the state value function, i.e.
$$\mathbb{E}_{a_t\sim \pi}[ G_{t+1} |S_{t+1} = s'] = v_{\pi}(s')\;.$$
This leads us to get what you have
$$q_{\pi}(s,a) = \mathbb{E}_{s_{t},r_{t} \sim E}[r(s_t,a_t) + \gamma v_{\pi}(s') | S_t = s, A_t = a]\;,$$
where the only difference is that we have made clear what our expectation is taken with respect to.
The expectation is taken with respect to the conditional distribution $S_{t+1},R_{t+1}|A_t,S_t$, and we usually include the $\pi$ subscript to denote that they are also taking the expectation with respect to the policy, but here this does not effect the first term as we have conditioned on knowing $A_t$ and only applies to the future reward signals.


Trying to understand VGG convolution neural networks architecture Trying to understand the VGG architecture and I have these following questions.

I understand the general understanding of increasing filter size is because we are using max pooling and so its image size gets reduced. So in order to keep information gain, we increase filter size. But the last few layers in the VGG architecture, the filter size remained same when vgg was max pooling from 14x14 to 7x7 image size, the filter size remained same at 512x512. Why wasn’t there the need to increase filter size there?
Also few consecutive layers, in the end, was constructed with both same filter and image size, those layers were built just to increase accuracy? (experimentation?)
And I couldn’t wrap around that visualization at final filters have the entire face as the feature as I understood through convolution visualizing (Matt Zieler video explanation). But max pooling causes us to see only a subset part of the image right? When filter size is 512x512 (face as the filter/feature) the image size as 7x7, so how does entire face as a filter will work on images when we are moving over small subset of the image pixels?

Good questions. Let me reply one by one.

1- Filter size can be increased. There is no limit for it. However, think two cases:

DNN part. Shape will be 1024 x 7 x 7 and it will map to 4096 features, it will cause 204M parameters at dense_1 layer. This change will cause two possible problem. Overfitting and inference/training speed.
Sparsity. You can implement 1024 features at 5th conv block and train the network and check the accuracy to see if increased feature count will make any impact or not.

So, to decide the number of con layers, it is a good practice to check sparsity of feature layers.

2- If I did not understand your question wrong, those layers are DNN classifier as a fully connected layer.

3- Maxpooling is not causing us to see a subset of the image. It is just an algorithm to resizing. What you see in that video is the filter response to some objects exist in the image. In deeper level, do not expect to have visual shape like in shallow layers. You may even get single white pixel at deepest layer and it may be a response to an object like human face.


Minimax combined with machine learning to determine if a path should be explored I have an idea for a new type of AI for two-player games with alternating turns, like chess, checkers, connect four, and so on.

A little background: Traditionally engines for such games have used the minimax algorithm combined with a heuristic function when a certain depth has been reached to find the best moves. In recent days engines using reinforcement learning, etc (like AlphaZero in chess) have increased popularity, and become as strong as or stronger than the traditional minimax engines.

My approach is to combine these ideas, to some level. A minimax tree with alpha-beta pruning will be used, but instead of considering every move in a position, these moves will be evaluated with a neural net or some other machine learning method, and the moves which seem least promising will not be considered further. The more interesting moves are expanded like in traditional minimax algorithms, and the same evaluation are again done for these nodes' children.

The pros and cons are pretty obvious: By decreasing the breadth (number of moves in a position), the computation time will be reduced, which again can increase the search depth. The downside is that good moves may not be considered, if the machine learning method used to evaluate moves are not good enough.

One could of course hope that the position evaluation itself (from the neural net, etc) is good enough to pick the best move, so that no minimax is needed. However, combining the two approaches will hopefully make better results.

A big motivation for this approach is that it resembles how humans act when playing games like chess. One tends to use intuition (which will be what the neural net represents in this approach) to find moves which looks interesting. Then one will look more thoroughly at these interesting moves by calculating moves ahead. However, one does not do this for all moves, only those which seem interesting. The idea is that a computer engine can play well by using the same approach, but can of course calculate much faster than a human.

To illustrate the performance gain: The size of a minimax tree is about b^d, where b is the average number of moves possible in each position, and d is the search depth. If the neural net can reduce the size of considered moves b to half, the new complexity will be (b/2)^d. If d is 20, that means reducing the computation time by approx. 1 million.

My questions are:

Does anyone see any obvious flaws about this idea, which I might have missed?
Has it been attempted before? I have looked a bit around for information about this, but haven't found anything. Please give me some references if you know any articles about this.
Do you think the performance of such a system could compete with those of pure minimax or those using deep reinforcement learning?

Exactly how the neural net will be trained, I have not determined yet, but there should be several options possible.

The use of of a neural network to push the search algorithm to continually only along a promising path is the same that was described in the AlphaZero paper. In AlphaZero, the NN loop contained the search function and would encourage the continued search of high probability moves that were then simulated by the same NN that now contained the Value Net. The use of alpha-beta specifically is not necessary. Just a search function aptly known as PUCT (Predictor + Upper Confidence Bounds applied to Trees)


Anomaly Detection in distributed system using generated log file I am developing an AI tool for anomaly detection in a distributed system.  The system supports an interface that combines several individual logs into a single log file generating approx. 7000 entries/min. The logs entries are partially system generated (d-Bus, IPC, ….)  and human written statements (Status not received, initialized successfully, ….). The developers use the generated log for debugging. The entries have been configured to have a similar format depending on the generated system (timestamp, ids, component, context, verbosity level, description, ….).

Background:
1. The history of the identified anomalies is minimal and not archived.
2. Not many similar event templates in log files.
3. Software execution rules are not clearly documented.
4. The log events are co-related.

What are the recommended algorithms (Statistical, NLP, ML, Neural networks) that can be used to efficiently perform pattern extraction on the entries and identify existing and new anomalous behavior?

In the paper "Unsupervised real-time anomaly detection for streaming data" (by Subutai Ahmad, Alexander Lavin, Scott Purdy and Zuha Agha), 2017, an algorithm for anomaly detection (particularly suited for cases where a stream of data is continuously provided) is described. This algorithm is based on Numenta's Hierarchical Temporal Memory model.

I've actually never used it, but I know that Numenta's work is particularly suited for anomaly detection. You can have a look at it and see if it fits your needs. Have also a look at the Numenta Anomaly Benchmark (NAB).


Why can't DQN be used for self-driving cars? Why can't DQN be used for self-driving cars?
Why can't DQN and similar RL algorithms be used for self-driving cars?

The reason why I am curious is that it successfully plays go and other multistate games.

I'm not familiar with the ins and outs of self-driving cars, but I imagine that the action space is not discrete. For instance, the car may want to decide what angle it needs to turn (rather than left or right). The update in Q-Learning involves taking $\max_aQ(s, a)$; this is theoretically possible for a continuous action space, but it would itself require some expensive optimisation at each time step to find the maximum. It is more likely that if RL were to be applied to self-driving cars it would be through a method that easily allows for a continuous action space, like the methods detailed in this paper.
I found this survey of Deep RL for autonomous driving that you may want to look at.


In PyTorch, why does the sequence length need to be provided as the first dimension of the input tensor for an RNN? I am confused as to why the sequence length is the first dimension of the input tensor for an RNN, while the batch size is the first dimension for any other kind of network (linear, CNN, etc.).
This makes me think that I haven't fully grasped the concept of RNN batches. Is each independent batch a different sequence? And is the same hidden state across batches? Is the hidden state maintained between timesteps for a given sequence (for vanilla/truncated BPTT)?

As it says in the documentation, you can simply reverse the order of dimensions by providing the argument batch_first=True when constructing the RNN.
Then, the dimensionality will be: (batch, seq, feature), i.e. batch-size times sequence length times the dimension of your input (however dimensional that may be). Then, everything is gonna work as you are used to it.
To answer the second part of your question, normally, each sequence in a batch is independent of the others (since they commonly get sampled at random). So, there is no direct dependence between any two inputs in a batch (except, of course, for the fact that they are commonly expected to stem from some underlying shared data generating process which you want to approximate by the RNN).
And a hidden state is commonly maintained per batch element, i.e. there is one hidden state per batch element (i.e. per sequence).


How do I apply the value iteration algorithm when there are two goal states? I am working through the famous RL textbook by Sutton & Barto. Currently, I am on the value iteration chapter. To gain better understanding, I coded up a small example, inspired by this article.

The problem is the following

  There is a rat (R) in a grid. Every square is accessible. The goal is to find the cheese (C). However, there is also a trap (T). The game is over whenever the rat either find the cheese or is trapped (these are my terminal states).

The rat can move up, down, left, and right (always by one square).

I modeled the reward as follows:

-1 for every step
5 for finding the cheese
-5 for getting trapped

I used value iteration for this and it worked out quite nice.

However, now I would like to add another cheese to the equation. In order to win the game, the rat has to collect both cheese pieces.

I am unsure how to model this scenario. I don't think it will work when I use both cheese squares and the trap square as terminal states, with rewards for both cheese squares.

How can I model this scenario? Should I somehow combine the two cheese states into one?

What you could do is to trigger environment termination when rat either:

steps into the trap
picks both cheese pieces

The problem with such setup is that, when the rat picks a single piece, it would move one step to the side, and then it would come back to the same cheese spot so it would keep exploiting the same spot indefinitely.

The solution to that would be to simply remove the cheese piece once the rat picks it up, so that it can't exploit it indefinitely.

Sadly, another problem would arise which is partial observability: Markov property wouldn't be fulfilled because the current action wouldn't depend on the current state solely, it would be important whether cheese piece was picked before or not.

The solution to that would be to make environment fully observable. You could accomplish that by expanding the amount of information about your current state. Before, only your position on the grid was important, but now you would also add state features that tell you whether cheese piece at specific position was picked or not. You would basically add a flag for each cheese piece that has value of 1 if piece was picked, or value of 0 if it wasn't. That way you could remove cheese piece it rat picks it, and you would still have full information.

I believe this setup would work.


Can reinforcement learning algorithms be applied to computer vision problems? Can reinforcement learning algorithms be applied to computer vision problems? If yes, what are some examples of these applications?

Yes, there are a couple of ways to apply reinforcement learning in computer vision problems. This mainly employs the principle of "applying the algorithm -> evaluating the outcome -> adopting the best outcome".

The following are a couple of examples that use reinforcement learning in computer vision.

CAD2RL: Real Single-Image Flight Without a Single
Real Image.
End-to-End Training of Deep Visuomotor
Policies


Is some kind of dropout used in the human brain? I've read that ANNs are based on how the human brain works. Now, I am reading about dropout.
Is some kind of dropout used in the human brain? Can we say that the ability to forget is some kind of dropout?

The human brain works by having neurons constantly fire at different rates. So, if the firing rate increases, the neuron is transmitting overly exciting or calming information to further neurons connected to it. How other neurons connected to the former neuron respond on the messages sent by it, depends on the strength of the connection between the connected neurons.
However, having true dropout in the brain would be like having a neuron stop firing entirely for some time. As far as I know, this is not gonna happen unless a neuron dies. So, no, there doesn't seem to be any real biological equivalent to dropout in the brain. If you forget something, that is just caused by a weakening of some connections in the brain. But such a weakening (or strengthening alternatively) is a gradual process and is a function of the degree of firing synchrony between any two connected nodes.
It is true that the introductory texts to Neural Nets tell you about those nets' neuro-scientific inspiration, but this similarity holds only in the most abstract way. Yes, you have neurons in both biological and artificial neural networks. But communication in the brain works in an asynchronous, continuous bio-chemical way, while communication in an artificial neural net works by taking the numeric state of one node, scaling it by the weight of the connection between two nodes, and then adding the weighted contribution of the previous node to the activation of the receiving node. So, already the fundamentals are very different between the two approaches (with the biological variant being much more complex and elaborate).
If you want to find something that possesses more biologically inspired properties, look at how Convolutional Neural Networks are theoretically inspired by the working of the (human) visual system. Besides that, dropout is a purely mechanical strategy to regularize a model well, in order to enhance the model's generalization ability, i.e. making the model work well also on unseen data. It's just part of an algorithmic optimization procedure.


Is there any value given to each chess piece in AlphaZero? Recently, DeepMind's AlphaZero chess algorithm did better than the prior best chess software Stockfish. I read the paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm about it, but I'm not sure if:

is there a value given for each piece (e.g. 1 for pawn, 3 for the knight, 9 for the queen, etc.) to train the algorithm, or does the algorithm learn this by himself?

I read that the algorithm uses Monte Carlo Tree Search, but what are the key improvements to prior chess algorithms already using MCTS?

Is there hope for being able to run it an average computer? They said it required 9 hours learning (starting with nearly 0 knowledge except for rules (and maybe value for a piece?)), and 24 millions of games. Is it something doable in maybe 1 month with an average computer?

is there a value given for each piece (e.g. 1 for pawn, 3 for knight, 9 for queen, etc.) to train the algorithm, or does the algorithm learn this by himself?

No, there are no such explicit values assigned to pieces, no manually-constructed evaluation functions. The paper states that "no domain knowledge" is given to the algorithm other than the game's rules (necessary to run simulations / run search algorithms like MCTS).

  I read that the algorithm uses Monte Carlo Tree Search, but what are the key improvements to prior chess algorithms already using MCTS?

The key improvements are in the way that Deep Learning (deep neural networks), Reinforcement Learning, and self-play is combined with MCTS. This is quite similar to the methods previously used by AlphaGo Zero in the game of Go as well. There (likely) have been combinations of Deep Learning + MCTS before (e.g. using a learned Neural Network to bias rollouts in MCTS), but the specific way in which they are combined in AlphaZero is critical (specifically, using the "search behaviour" of MCTS as one of the training signals for the Neural Network). It probably also helped that we're talking about Google here, who could afford to use thousands of Tensor Processing Units (TPUs) for training.

  Is there a hope for being able to run it an average computer? They said it required 9 hours learning (starting with nearly 0 knowledge except rules (and maybe value for piece?)), and 24 millions of games. Is it something doable in maybe 1 month with a average computer?

Based on the information in the paper, I highly doubt this. As mentioned above, training was done with thousands of TPUs (specifically; 5,000 first-generation TPUs and 64 second-generation TPUs) in parallel. One month is only about a factor 80 times greater than the 9 hours training time reported in the paper, whereas all of those TPUs is much more than 80 average computers.

After training, it likely could run on a fairly average computer (or maybe high-end computer) to play though. But it'll need much more power to train first.


How could we build a neural network that is invariant to permutations of the inputs? Given a neural network $f$ that takes as input $n$ data points: $x_1, \dots, x_n$. We say $f$ is permutation invariant if
$$f(x_1 ... x_n) = f(\sigma(x_1 ... x_n))$$
for any permutation $\sigma$.

How could we build such a neural network? The output can be anything (e.g. a number or vector), as long as it does not depend on the order of inputs. Is there any research work on these types of neural networks?

Here is a few that might be what you are looking for:

Deep Sets, https://papers.nips.cc/paper/6931-deep-sets.pdf
BRUNO: A Deep Recurrent Model for Exchangeable Data, https://arxiv.org/pdf/1802.07535.pdf
Deep Learning with Sets and Point Clouds, https://openreview.net/pdf?id=HJF3iD9xe
Permutation-equivariant neural networks applied to dynamics prediction, https://arxiv.org/pdf/1612.04530.pdf


What do echo state networks give us over a generic RNN resevoir? Slightly generalizing the definition in Jaeger 2001, let's define a reservoir system to be any system of the form
$$h_{t}=f(h_{t-1}, x_t)$$
$$y_t=g(Wh_t)$$
where $f$ and $g$ are fixed and $W$ is a learnable weight matrix. The idea is we feed a sequence of inputs $x_t$ into the system, which has some fixed initial state $h_0$, and thereby generate the sequence of outputs $y_t$. $f$ is fixed (for example, a randomly generated RNN) we can then attempt to learn $W$ in some way in order to get the system to have the behavior that we want.
Now we add the echo state condition: the system has the echo state condition iff for any left-infinite sequence $...x_{-3}, x_{-2}, x_{-1}, x_0$, there is only one sequence of states $h_t$ consistent with this input sequence.
Seen from this perspective, any training procedure that could be applied to an echo state system could be applied to a generic reservoir system. So what do we get out of the echo state condition? Is there some reason to think echo state systems will generalize better, or be more quickly trainable? Jaeger does not seem to attempt to argue in this direction, he just describes how to train an ESN, but as I've said, nothing about these training methods seems to require the echo state property.

The echo state condition states that differences in the input sequence results in separate trajectories of reservoir states.
Which means that when you make the reservoir large enough any difference in the input sequence result in a linearly separable difference in the state space.
This is quite similar to the Kerneltrick in e.g. Support Vector machines where the data becomes linearly separable in the feature space. In reservoir computing the reservoir is in a way a random feature space.
To sum up, the echo state condition ensures that the signal becomes linearly separable.


TF Keras: How to turn this probability-based classifier into single-output-neuron label-based classifier Here's a simple image classifier implemented in TensorFlow Keras (right click to open in new tab): https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/advanced.ipynb

I altered it a bit to fit with my 2-class dataset. And the output layer is:

Dense(2, activation=tf.nn.softmax);

The loss function and optimiser are still the same as in the example in the link above.

loss_fn   = tf.losses.SparseCategoricalCrossentropy();
optimizer = tf.optimizers.Adam();

I wish to turn it into a classifier with single output neuron as I have only 2 classes in dataset, and sigmoid does the 2 classes good. Tried some combinations of output activation functions + loss functions + optimisers, but the network doesn't work any more (ie. it doesn't converge).

For example, this doesn't work:

//output layer
Dense(1, activation=tf.sigmoid);

//loss and optim
loss_fn   = tf.losses.mse;
optimizer = tf.optimizers.Adagrad(1e-1);

Which combination of output activation + loss + optimiser should work for the single-output-neuron model? And generically, which loss functions and optimisers should pair well?

Advice from Neil, yes, output targeting class labels is still classification.

Output range is in contiguous range:

This is regression
For example: Linear activation, it has full numeric range of outputs.

Output targeting class labels is classification:

Single output neuron with sigmoid-like functions. This will classify to 2 classes, although Y data can be normalised to classify more classes.
Multiple output neurons (probabilities of classes) with sigmoid-like functions (softmax is mainly used). This will classify to 2 or more classes.

Multiple combinations of loss functions and optimisers can make the single-neuron output layer work, with different configs for them. Note that learning rates of different optimisers are different, some take 1e-1, some need 1e-3 for good training.

For example, this combination should work:

loss_fn   = tf.losses.LogCosh();
optimizer = tf.optimizers.RMSprop(1e-3);

From my trying out, these other combinations also work for single output neuron with my data (Adam, Adamax, Nadam, RMSprop work when learning_rate=1e-3 instead of 1e-1):

                         Adadelta  Adagrad  Adam  Adamax  Ftrl  Nadam  RMSprop  SGD
BinaryCrossentropy       Yes       Yes      --    --      --    --     --       Yes
CategoricalCrossentropy  Yes       --       --    --      --    --     --       --
CategoricalHinge         --        --       --    --      --    --     --       --
CosineSimilarity         --        --       --    --      --    --     --       --
Hinge                    Yes       Yes      --    --      --    --     --       Yes
Huber                    Yes       Yes      --    --      --    --     --       Yes
LogCosh                  Yes       Yes      --    --      --    --     --       Yes
Poisson                  Yes       Yes      --    --      --    --     --       Yes
SquaredHinge             Yes       Yes      --    --      --    --     --       Yes

KLD: lambda a,b: KLD(a,b)
MAE,MAPE,MSE,MSLE: lambda a,b: Mxxx(a,b)
The above lambdas are direct functions, not classes like SGD, Adadelta, etc.
SparseCategorialCrossentropy: Seems not working with single output neuron.


Is there any AI system for finding the best way to schedule university classes? I was wondering whether there is an AI system which could be used to resolve the class clashes problem which mostly happens in universities. In almost every university students face this problem, where two or more courses that many students want to take together get scheduled at the same time. Does anyone know about a system which resolves this issue or someone who works on this problem?

Welcome to AI.SE @Israr Ali.

The problem of scheduling a timetable is an example of a constraint satisfaction problem, a topic long studied in AI.

There are many possible techniques to apply to this kind of problem. They can be organized into three broad categories:

Global search algorithms, like backtracking search can be used to try and find an assignment of times to classes that results in no conflicts at all. With proper heuristics, these can be quite fast in practice, even though their worst-case runtimes are poor. Russell & Norvig's AI: A Modern Approach has a very good overview of these approaches in Chapter 6.
Local search algorithms, like hill climbing search start by assigning every class a randomly selected time, and then making small adjustments that monotonically decrease the number of conflicts (e.g. swapping the times of two classes). These are often faster than backtracking approaches and are especially so if you are willing to accept a "pretty good" schedule that might still have fewer conflicts than an optimal one. Chapter 4 of AI: A Modern Approach provides a good introduction.
Planning approaches, particularly non-classical algorithms like GraphPlan can also be used for these domains. They make use of the structure of planning or scheduling problems in particular to re-frame the search problems addressed by Backtracking techniques. By using this domain-specific representation, they are able to achieve very high-quality solutions very quickly. Chapter 11 of AI: A Modern Approach covers Graph Plan in some detail, and could be a good introduction to these more specialized techniques.


Why would someone use NEAT over other machine learning algorithms? Why would someone use a neuroevolution algorithm, such as NEAT, over other machine learning algorithms? What situation would only apply to an algorithm such as NEAT, but no other machine learning algorithm?

The main difference leading to strengths and weaknesses of NEAT algorithm, is that it does not use any gradient calculations. That means for NEAT, neither the cost function, nor the activation functions of the neurons are required to be differentiable. In some circumstances - e.g. where agents directly compete, and you can select them for next generation if they win versus one or more opponents - you may not even need a cost function to optimise. Quite often the cost function can be a simple number generated from a complex evaluation of the network.

Therefore, NEAT can be used in situations where the formula for a cost function, in terms of single feed-forward runs of the network, is not very clear. It can also be used to explore activation functions such as step functions or stochastic firing neurons, where gradient based methods are difficult to impossible to apply.

NEAT can perform well for simple control scenarios, as a policy network that outputs actions given some sensor inputs. For example, it is popular to use it to create agents for racing games or simulated robotics controllers. When used as a policy-based controller, NEAT is in competition with Reinforcement Learning (RL), and has basic similarities with policy gradient methods - the nature of the controller is similar and often you could use the same reward/fitness function.

As NEAT is already an evolution-inspired algorithm, it also fits well with a-life code as the "brains" of simulated creatures.

The main disadvantage of NEAT is slow convergence to optimal results, especially in complex or challenging environments. Gradient methods can be much faster, and recent advances in Deep RL Policy Gradients (algorithms like A3C and DDPG) means that RL can tackle much more complex environments than NEAT.

I would suggest to use NEAT when:

The problem is easy to assess - either via a measurement of performance or via competition between agents - but might be hard to specify as a loss function
A relatively small neural network should be able to approximate the target function
There is a goal to assess a non-differentiable activation function

If you are looking at a sequential control problem, and could use a standard feed-forward neural network to approximate a policy, it is difficult to say in advance whether NEAT or some form of Deep RL would be better.


Bayesian Perceptron: Why to marginalize over neuron's output instead of it's weights? I found a very interesting paper on the internet that tries to apply Bayesian inference with a gradient-free online-learning approach: Bayesian Perceptron: Towards fully Bayesian Neural Networks.
I would love to understand this work, but unfortunately I am reaching my limits with my Bayesian knowledge. Let us assume that we have the weights $\mathcal{w}$ of our model and observed the data $\mathcal{D}$. Using the Bayes rule, we obtain the posterior according to $$p(\mathcal{w}|D)=\frac{p(D|\mathcal{w})p(\mathcal{w})}{p(D)}$$.
In words: we update our prior belief over our weights by multiplying the prior with the likelihood and divide everything by the evidence. In order to calculate the true posterior, we would need to calculate the evidence by marginalizing over (intergrating out) our unknown parameters. This gives the integral $$p(D) = \int p(D|\mathbf{w})p(\mathbf{w})dw$$.
So far so good. Now I refer to the paper mentioned above. Here, the approach is presented exemplarily on a neuron whose weighted sum is called $a$, which is then given to the activation function $f(.)$. Moreover it is assumed that $\mathbf{w}\sim N (\mu_w, \mathbf{C}_w)$. Because of the linearity it can be exploited that also $\mathbf{a}\sim N (\mu_a, \mathbf{C}_a)$.
What I am confused about now is formula (14), which seems to show the compute the true posterior:
$$p(w) = \int p(a, w|D_i)da = \int p(w|a, D_i)p(a|D_i)da$$
Why is $a$ integrated out here and not $w$? We want a distribution over $w$, don't we? But without marginalization over $w$ there is still uncertainty inside $w$
Glad about any help and food for thought;)

We want a distribution over $w$, don't we?

Yes. You want to obtain a distribution over the parameters, which models the uncertainty about the parameters. This distribution over the parameters can induce a probability distribution over the possible functions consistent with your data.

Why is $a$ integrated out here and not $w$?

This is just the definition of marginalization.
For simplicity, consider 2 discrete random variables $X$ and $Y$, which can take values $x_i$ and $y_i$, respectively, and for which you can have (at least conceptually) a joint distribution $p_{(X, Y)}(x, y)$. Then the marginal probability that $X$ takes the value $x_i$ independently of any outcome of the r.v. $Y$ (and this is the intuition behind the marginal distribution!) is given by
$${\displaystyle p_{X}(x_{i})=\sum _{j}p(x_{i},y_{j})}, \tag{1}\label{1}$$
So, in the case of equation 14 of the paper, there's the assumption that you have $p(w, a)$. Then $p(w)$ is a marginal distribution over $w$ (the parameters or weights of the neural network), independently of the values of the activations. Of course, $w$ and $a$ can be dependent or correlated, and that's why you also have $p(w, a)$, i.e. the probability distribution that describes how these two r.v.s vary together, but what you want to know is the probability distribution over $w$ independently of $a$, so you "marginalize out" the possible values of $a$ (not $w$).
Remember that $p(x, y) = p(x \mid y) p(y) = p(y \mid x) p(x)$ (this is known as the chain rule of probability for two r.v.s, which leads to the Bayes' theorem), so you could write equation (\ref{1}) as follows
$${\displaystyle p_{X}(x_{i})=\sum _{j}p(x_{i},y_{j})} = \sum _{j}p(x_{i} \mid y_{j}) p(y_j), \tag{2}\label{2}$$
In the case of real-valued r.v.s, you will have an integral rather than a summation, but the idea is roughly the same.

But without marginalization over $w$ there is still uncertainty inside $w$.

What we want to know is $p(w)$, i.e. a distribution over $w$, which models the uncertainty about the possible values that $w$ can take. So, once you know $p(w)$, depending on its shape, we will have more or less uncertainty about the possible values of $w$. For example, let's say $p(w)$ is a Gaussian distribution. Then, depending on the variance, you will have more or less uncertainty about the possible values of $w$.
Your doubt seems to be that you thought that marginalization is a way to not have uncertainty, but that's not true. So, the goal in Bayesian machine/deep learning is not to be certain about the values of the parameters, but to model the uncertainty about the values of the parameters.


How can I make meaningful English sentences from given set of words? I have a set of topics and each topic consists of a set of words. I want to make meaningful English sentences from these words. Each topic consist of 5 to 10 words and these words are relevant to each other, like {code, language, test, write and function} and {class, public, method, string, int} are two sets. I want to generate a sentence from these set of words using API.

Two Approaches:

Naive Bayes
LSTM

Train Naive Bayes on a whole dataset learning the probability of the next word given a word.

You can even go with any LSTM approaches, but I'd bet on Naive Bayes.

Eg:

text: hello how are you hello how are you hello No how

to get the suggestion of next word depending on current word - hello

p(how | hello) = 3/4

p(No | hello) = 1/4

take argmax of probabilities.

Also remember to smooth, and train on huge dataset. Training is just finding the probabilities before hand.

Hope it helps ;)


How to quantify the reflectance in an image? I am working on a problem where I have to train a CNN to recognize different kinds of surfaces. One important characteristic of the surfaces I am interested is is how reflective they are. I have been trying to find a method that quantifies how "shiny" a surface is, but I have not found much. I am hoping that someone can point me toward a method or some research into this kind of problem.

You should first know about the layer-wise working of a convolutional neural network. Read this https://distill.pub/2018/building-blocks/

Each layer of a cnn forms representations that are increasingly complex , which are  combinations of simpler representations formed by its previous layers.
"shine" is not a pattern (from image processing perspective). shine occurs due to change in intensity values.

And for quantifying "shine" you dont have to go for convolutional networks only , there are many more simpler methods for such tasks you could try fitting simple linear model and progress towards more complex ones.

and for classifying textures , there is interesting work going with "invariant scattering convolutional networks" you should check that.


If the AI goal is the protection of humans, will it always pursue this goal? If the AI goal is to serve humans and protect them (if this ever happens) and AI someday realizes that humans destroy themselves, will it try to control people for their own good, that is, will it control man's will to not destroy himself?

If the main goal of AI (which I assume you mean an AGI) is to protect humans and AI will be effective, then AI will always attempt to pursue its main goal (otherwise the assumption of its effectiveness does not hold), even at the expense of other less important goals that it might have. However, if the destruction of a human (or a group of humans) protected or avoided the destruction of other humans, then AI would face a dilemma. In that case, I think it is hard to predict the actions of the AI. Will it act rationally or irrationally? What would it mean for the AI to act rationally? Which parameters will it take into account? Only the number of deaths, or will take into account the future and weight the importance of the lives? How will it define the importance of a human life?


Has an AI ever solved a detective mystery? In detective novels, the point is often that the reader gets enough information to solve the crime themselves. This "puzzle" aspect of detective novels is part of the attraction.

Often the difficulty for humans is to keep track of all the variables - events, items, motivations.
An AI would have an easier time keeping track of all the details, but would rely on real-world knowledge to prevent making crazy mistakes. For example, if it was stated that a character took the train, the AI would need to know that this is a method of transportation - that it changes the location property of an agent over time.

Has an AI ever been able to solve a detective mystery?

I might be wrong, but I do not believe that something of the scope you describe would be possible with the current state of technology. It would require a lot of things which are still in relatively early stages of research.

For one, just extracting relevant information from text is a huge task by itself. Doubly so with a novel which contains a large amount of unimportant details.

It might perhaps be easier if the input was presented in the form of some sort of list of important facts. But it would still be rather difficult for the AI to connect them and find a solution.

As an example, let's say that we have these two facts:

Alice died of a snake bite.
Bob was seen buying a couple of mice recently.

To a human, it seems obvious that the mice were bought to feed a venomous snake. However, it would probably require a tremendous effort to teach an AI to make such connections.

Disclaimer: I don't do text processing myself, so I'm not quite up-to-date on the current state-of-the-art. It's possible that some of these things have already been done in some form. If anyone knows more about the subject, please correct me if I'm wrong.


Has any schema-agnostic database engine been implemented? Has any schema-agnostic database engine been implemented?

A very large one, the world wide web with highly scaled and optimized indexing by Google.com is the most distributed and robust schema-agnostic database known today.  Without the schema-awareness Google brought to the table by applying more rigorous information science to the table, it was almost useless to those that did not know the URL of the target document in advance.

Schema agnosticism is another way of saying that the database cannot

Provide meta information to the services accessing it,
Normalize the structure using simple SQL query-insert combinations
Proactively optimize the keys automatically as is now possible with machine learning, or
Validate insertions

Without first detecting a schema from data patterns.  Moving away from structure is appealing because you can just jam data in like a librarian without a book shelf.   However, the data scientist will point out that adding entropy working alongside thermodynamic devolution into stochasm.

The purpose of storing data is to be able to retrieve it.  Feature extraction is an opportunity to improve structure automatically during the storing structure, rather than store documents chaotically, a trend that will not lead anywhere good for the world of IT.

Consider whether Google is successful because it organizes its data as it crawls or later as we enter key phrases.  Which is the efficient sequence?

One more point, Wikipedia is a blog, and they know this, which is why they want peer review for everything now (after much of the information was added without peer review).  It is a good place to find lists but not verified facts.  The existence of a Wikipedia page is definitely not an indication of the value of the concept on it.