How can I train a neural network to describe the characteristics of a picture? I have collected a set of pictures of people with a text explaining the characteristics of the person on the picture, for example, "Big nose" or "Curly hair".

I want to train some type of model that takes in any picture and returns a description of the picture in terms of characteristics.

However, I have a hard time figuring out how to do this. It is not like labeling "dog" or "apple" because then I can create a set of training data and then evaluate its performance, now I can not. If so I would probably have used a CNN and probably also VGG-16 to help me out.

I only have two ML courses under my belt and have never really encountered a problem like this before. Can someone help me to get in the right direction?

As of now, I have a data set of 13000 labeled images I am very confident it is labeled well. I do not know of any pre-trained datasets that could be of help in this instance, but if you know of one it might help.

Worth noting is that every label is or should at least be unique. If for example there exist two pictures with the same label of "Big nose" it is purely coincidental.

The term you are looking for is multi-label classification, i.e. where you are making more than one classification on each image (one for each label). Most examples you'll find online are in the NLP domain but it is just as easy with CNNs since it's essentially defined by the structure of the output layer and the loss function used. It's not as complicated as it might sound if you are already familiar with CNNs.

The output layer of a neural network (for 3 or more classes) has as many units as there are targets. The network learns to associate each of those units with a corresponding class. A multi-class classifier normally applies a softmax activation function to the raw unit output, which yields a probability vector. To get the final classification, the max() of the probability vector is taken (the most probable class). The output would look like this:

                 Cat    Bird   Plane   Superman  Ball   Dog
Raw output:      -1     2      3       6         -1     -1
Softmax:         0.001  0.017  0.046   0.934     0.001  0.001
Classification:  0      0      0       1         0      0

Multi-label classification typically uses a sigmoid activation function since the probabilities of a label occuring can be treated independently. The classification is then determined by the probability (>=0.5 for True). For your problem, this output could look like:

                 Big nose  Long hair  Curly hair  Superman  Big ears  Sharp Jawline
Raw output:      -1        -2         3           6         -1        10
Sigmoid:         0.269     0.119      0.953       0.998     0.269     1.000
Classification:  0         0          1           1         0         1

The binary crossentropy loss function is normally used for a multi-label classifier since a n-label problem is essentially splitting up a multi-class classification problem into n binary classification problems.

Since all you need to do to get from a multi-class classifier to a multi-label classifier is change the output layer, its very easy to do with pre-trained networks. If you get the pre-trained model from Keras its as simple as including include_top=False when downloading the model and then adding the correct output layer.

With 13000 images, I would recommend using Keras' ImageDataGenerator class with the flow_from_dataframe method. This allows you to use a simple pandas dataframe to label and feed in all your images. The dataframe would look like this:

Filename  Big nose  Long hair  Curly hair  Superman  Big ears  Sharp Jawline
0001.JPG  0         0          1           1         0         1
0002.JPG  1         0          1           0         1         1
   .      .         .          .           .         .         .

flow_from_dataframe's class_mode parameter can be set to raw or multi_output along with x_col to 'Filename' and y_col to ['Big nose', 'Long hair', 'Curly hair', 'Superman', 'Big ears', 'Sharp Jawline'] (in this example). Check out the documentation for more details.

The amount of data you need for each label depends on many factors and is essentially impossible to know without trying. 13000 sounds like a good start but it also depends on how many labels you have and how evenly distributed they are between the labels. A decent guide (one of many) on how to set up a multi-label classifier and how to implement it with Keras can be found here. It also covers imbalances on label frequency and is well worth a read. I'd highly recommend that you become as intimately familiar with your dataset as possible before you start tuning your neural network architecture.


Restoration of localized damaged areas (time signals, but guess also applicable to images) I am starting to study the capabilities of neural networks for the reconstruction/restoration/... of communication signals.

I am feeding my neural network with a signal which has some parts which have been damaged because of the transmission through a communication system, and my targets are given by the signal with these areas undamaged.

The problem is that the areas damaged represent a very small portion of the whole signal, and my neural network spends lot of time learning only from the portions which actually do not present any problem.

Is there any solution to make the neural network to jump on those areas which show significant differences to respect to the targets? Is there anything I could do for example initializing my neural networks (as conventionally done, they are initialized randomly)? Or shall I accept that I need to train for longer time?

So , you want to remove the "noise" (damaged signal) from the signal, by giving the perfect signal as output. This model is called as a "denoising autoencoder". If you have enough training data, the noise is not at all a problem for the neural network , if you really feel that the network is capturing the noise , then you have to increase the training epochs , or sometimes it can be a bad initialized network.


Why is KL divergence used so often in Machine Learning? The KL Divergence is quite easy to compute in closed form for simple distributions -such as Gaussians- but has some not-very-nice properties. For example, it is not symmetrical (thus it is not a metric) and it does not respect the triangular inequality.
What is the reason it is used so often in ML? Aren't there other statistical distances that can be used instead?

This question is very general in the sense that the reason may differ depending on the area of ML you are considering. Below are two different areas of ML where the KL-divergence is a natural consequence:

Classification: maximizing the log-likelihood (or minimizing the negative log-likelihood) is equivalent to minimizing KL divergence as typical used in DL-based classification where one-hot targets are commonly used as reference (see https://stats.stackexchange.com/a/357974). Furthermore, if you have a one-hot vector $e_y$ with $1$ at index $y$, minimizing cross-entropy $\min_{\hat{p}}H(e_y, \hat{p}) = - \sum_y e_y \log \hat{p}_y = - \log \hat{p}$ boils down to maximizing the log-likelihood. In summary, maximizing the log-likelihood is arguably a natural objective, and KL-divergence (with 0 log 0 defined as 0) comes up because of its equivalence to log-likelihood under typical settings, rather than explicitly being motivated as the objective.
Multi-armed bandits (a sub-area of reinforcement learning): Upper confidence bound (UCB) is an algorithm derived from standard concentration inequalities. If we consider MABs with Bernoulli rewards, we can apply Chernoff's bound and optimize over the free parameter to obtain an upper bound expressed in terms of KL divergence as stated below (see https://page.mi.fu-berlin.de/mulzer/notes/misc/chernoff.pdf for some different proofs).

Let $X_1, \dots, X_n$ be i.i.d. Bernoulli RVs with parameter $p$.
$$P(\sum_i X_i \geq (p+t)n) \leq \inf_\lambda M_X (\lambda) e^{-\lambda t} = \exp(-n D_{KL}(p+t||p)).$$


Can ML/DL solve my classification problem? I'm new to AI but would still like to try and get a project off the ground.
I've read a lot about ML/DL the past few days but I just can't figure out if my problem can be solved with ML/DL. What I'm trying to do looks like a classification job to me but maybe isn't.
I have 100s of images of compacted soil samples, on these images there may be multiple layers visible.
I will include a picture below, this sample had a sticker on it, normally they don't. On the image there are 3 layers, separated above and under the sticker.
With every image there is data (xml file) available on the size of the layer(s) and the type of soil in that layer, which costs a lot of time to produce, so I want to automate this classification in the future.
The data files contains info like:
layer0:
    type 004
    2cm
    12cm
layer1:
    type 003
    12cm
    25cm

If there would be just one layer, the AI could learn what these layers look like and sort them in the right soil class.
But I don't know if my problem can be solved with AI as there could be 1, 2, 3 or 4 different layers (classes) on one image and I haven't seen any examples on classification where there can be multiple classes in one image.
As AI is quite a steep learning curve I would like to know if my problem is suited for ML/DL before I spend more of my nights reading for something that might not work.
I've read numerous websites and a few short books but can't find an answer to my questions.
Can ML/DL solve my multi-class single-image classification problem and which strategy should I read into?

A simple sanity-check on whether an image classifier can perform a task in theory is:

Can a human expert, using the same image plus a list of catgeories that they are familiar with, perform the same task?

It is important you only consider the contents of the image (or in general the data you are prepared to supply to the classifier) and the expert's general knowledge. The expert is not allowed to collect more data for instance, or interact with the sample other than maybe take a few measurements on the pixels.
This sanity check doesn't tell you how hard the problem is. It also does rule out problems that are very hard or impossible for humans but actually quite easy for computers. However, it is a good start because nowadays single-purpose computer vision classifier tasks often rate similarly to or better than humans performing the same task. You are effectively checking "is the data I need for the inference actually in the image?"
Multi-class classifiers are possible in several ways. One way would be to have separate heads to the neural network to classify each layer, and maybe a layer present binary flag to allow for varying numbers of layers. This is similar to an architecture called YOLO which classifies 0 or 1 objects and their locations over multiple grid squares within an image. Your architecture would need to be different to YOLO, but you could use a lot of the ideas from it, such as having an output with multiple multi-class classifiers, one for each soil layer.

I have 100s of images of compacted soil samples

One issue you will face is that you would need a very large number of labelled images in order to train a neural network with cutting-edge performance for this task from scratch. So you will want to look into transfer learning, which involves taking an existing image classifier trained on e.g. ImageNet and adapting it to your problem before training with your smaller dataset.
The small amount of sample image data you have will be a major limiting factor in your case. Sadly no-one can tell you before you attempt the project whether you have enough for a deep learning approach. That is probably where your largest risk of failure is.


Can machine learning algorithms be used to differentiate between small differences in details between images? I was wondering if machine learning algorithms (CNNs?) can be used/trained to differentiate between small differences in details between images (such as slight differences in shades of red or other colours, or the presence of small objects between otherwise very similar images?)? And then classify images based on these differences? If this is a difficult endeavour with our current machine learning algorithms, how can it be solved? Would using more data (more images) help?

I would also appreciate it if people could please provide references to research that has focused on this, if possible.

I've only just begun learning machine learning, and this is something that I've been wondering from my research.

Attentive Recurrent Comparators (2017) by Pranav Shyam et al. is an interesting paper that helps to answer the question you're wondering, along with a blog post that helps to describe it in easier terms.

The way it's implemented is actually rather intuitive. If you have ever played a "what is different" game with two images usually what you'd do is look back and forth between the images to see what the difference is.  The network that the researchers created does just that!  It looks at one image and then remembers important features about that images and looks at the other image and goes back and forth.


Reinforcement Learning with Adaptive Action Magnitude How to select an action in a state if the action does not necesarily cause the environment to change state?

Given 10 states ($S_0$ to $S_9$) and in each state $i$ there are two actions defined $(1,-1)$. $1$ increases a parameter of the environment and $-1$ decreases it.

For example, if the parameter is speed and it is currently 1 rad/s, corresponding to ${S_i}$, an action can be either increase or decrease this and therefore transition to the next state, ideally $S_{i-1}$ or $S_{i+1}$.

It is unclear how to formulate a reinforcement learning problem in this case. The problem is the same magnitude of the parameter change through an action.

The range of the parameter is (3, 25) and step size is 1. The problem is thath the responce of the environment is not the same in every satate. In some states a parameter change with magnitude 1 results in a state transition in some state the magintude proves to be to small to provoke a transition change.

For example, if the envitonment is instate $S_1$ and action 1 is applied, how can the magintude of the paremter be adopted in a way which assures that the environment will transition to state $S_2$?  How can the step in the paremeter change be made adaptive? Actually my environment is uncertain that is why I don't know exactly whether this action will take me to next state or not.

For your information, I am using Q learning off policy algorithm. Suppose state 9 is the goal state and  my Q table is $10 \times 2$.

There are more possible approaches to tackle this problem:

Use a reinforcement learning method which can cope with a continous state space. This would eliminate the need for discretizing the state space which in turn, if I understood correctly, leads to problems in transitioning between states. You can also consider selecting a reinforcement learning method that can cope with a continous state space and a continous action space.
Actions do not have to be immediate. If your action 1 is transitioning form $S_i$ to $S_{i+1}$ you can have a function wich continously increases the spead until the next state is reached and only then you consider the execution of the action complete. Furthermore, the magnitude of the actions can be dependent on the current state, if it assures state transition.
You can add more actions. Use RL for your advantage and define not just $+1$ and $-1$ but $+0.1$, $0.3$, $+0.5$, $+0.7$, $+1$ (and same for deacreasing the speed) etc. Add a negative reward for all actions which do not cause a state transition or which cause higher jump then needed. However, care must be taken to make sure you have the same speed fro each sampling. Eg. if you are in $S_1$ and the action +0.1 does not cause a state transition to $S_2$ you have to reset the state (if possible) or fail the epoch, since the state you are in is not only dependent on the action and the previous state. In other words the speed of e.g. $1 rad/s$ (with a tolarance band) defined as $S_1$ will be slightly higher and ending up in $S_2$ after applying a slight increase in the speed takes less change in the speed than from $S_1$ without the "unnoticed" speed change.


How to deal with episode termination in Advantage Actor-Critic algorithm? Advantage Actor-Critic algorithm may use the following expression to get 1-step estimate of the advantage:

$ A(s_t,a_t) = r(s_t, a_t) + \gamma V(s_{t+1}) (1 - done_{t+1})  - V(s_t) $

where $done_{t+1}=1$ if $s_{t+1}$ is a terminal state (end of the episode) and $0$ otherwise.

Suppose our learning environment has a goal, collecting the goal gives reward $r=1$ and terminates the episode. Agent also receives $r=-0.1$ for every step, encouraging it to collect the goal faster. We're learning with $\gamma=0.99$ and we terminate the episode after $T$ timesteps if the goal wasn't collected.

For the state before collecting a goal we have the following advantage, which seems very reasonable: $A(s_t,a_t) = 1 - V(s_t)$.

For the timestep $T-1$, regardless of the state, we have: $ A(s_{T-1},a_{T-1}) = r(s_{T-1}, a_{T-1}) - V(s_{T-1}) \approx -0.1 -\frac{-0.1}{1-\gamma} = -0.1 + 10 = 9.9 $
(this is true under the assumption that we're not yet able to collect the goal reliably often, therefore the value function  converges to something close to $\frac{r_{avg}}{1-\gamma} \approx -10 $ ).

Usually, $T$ is not a part of the state, so the value function has no way to anticipate the sudden change in reward-to-go. So, all of a sudden, we got a (relatively) big advantage for the arbitrary action that we took at the timestep $T-1$.  Following the policy gradient rule, we will significantly increase the probability of an arbitrary action that we took at the end of the episode, even if we didn't achieve anything. This can quickly destroy the learning process.

How do people deal with this problem in practice? My ideas:

Differentiate between actual episode terminations and ones caused by the time limit, e.g. for them we will not replace next step value estimate with $0$.
Somehow add $t$ to the state such that the value function can learn to anticipate the termination of the episode.

As I noticed, the A2C implementation in OpenAI baselines does not seem to bother with any of that:

A2C implementation - calculation of discounted rewards
discount_with_dones function

First a small note: I don't think your expression for $A(s_{T-1}, a_{T - 1})$ looks correct. If we assume that $V(s_T) = 0$ (i.e., assume that we cannot possibly reach the goal in one single step from $s_{T - 1}$), we have:

\begin{align}
A(s_{T-1}, a_{T-1}) &= r(s_{T-1}, a_{T-1}) + \gamma V(s_{T}) (1 - done_{T})  - V(s_{T-1}) \\
&= r(s_{T-1}, a_{T-1}) - V(s_{T-1}).
\end{align}

In this expression, we'd normally have that $r(s_{T-1}, a_{T-1}) = -0.1$, whereas you seem to have mistakenly taken $+0.1$ in your post.

Those details aside; yes, there can be sudden bumps of positive advantage estimates as you described. This is not a problem though, this is exactly what we'd expect to happen given your description of the environment.

You describe an environment in which the agent is likely going to be wandering around randomly (at least when it hasn't learned a good policy yet), and incurring negative rewards over and over again. This naturally leads to negative value estimates for all encountered states. Suddenly, it does something and the episode terminates; it receives a nice reward of $0$ rather than yet another negative reward (this actually "feels" like a bonus, a positive reward, something more than was expected). When your agent has not yet learned a good policy that can reach the better reward of $1$, this is indeed a good result, a good action, and it rightfully should get reinforced.

Because this event of the episode terminating is mostly uncorrelated with the state (I say "mostly", because in theory it probably ends up being a slightly rarer event in states close to the goal than in states far away from the goal), it will eventually (after sufficient training time) end up occurring approximately equally often in all states. From the perspective of an agent that is oblivious to the current time step, this will be perceived as an event that can simply occur by pure chance in a nondeterministic environment.

This is not necessarily a problem. It can slow down learning due to increased variance in your reward observations (which can be addressed by using low learning rates / large batch sizes), but Reinforcement Learning algorithms are almost always naturally built to handle nondeterministic environments, it can work this out, it can average out all the different outcomes observed for the same state+action pairs. This is not a problem that requires dealing with.

  My ideas:

  Differentiate between actual episode terminations and ones caused by the time limit, e.g. for them we will not replace next step value estimate with $0$.
  Somehow add $t$ to the state such that the value function can learn to anticipate the termination of the episode.

The first idea fundamentally changes the quantity that your algorithm is learning, it will essentially make your learning algorithm incorrect. There's always a chance that it might still appear to learn something useful (many Machine Learning/Reinforcement Learning algorithms can still appear to be okay even when there are bugs/technically incorrect parts), but it'll very likely perform worse.

The second idea, while not necessary as explained in my answer above, may still be beneficial to learning speed provided it is done well. It may help because it can add to the power of your algorithm to "explain" its observations, and more importantly "explain" the variance in its observations.

The main problem I see with adding $t$ to your input is that it is not naturally a binary variable. Very often you'll find that we're just using a bunch of binary inputs in (Deep) Reinforcement Learning algorithms. When all inputs are of the same magnitude like that, it tends to be easier to get the learning process to run well, tune hyperparameters like learning rate, etc. If you suddenly plug in an additional input which can take significantly larger values (like straight up adding $t$ as an input), this will be more difficult. Adding $\frac{t}{T}$ as an input may be better, since that will always still be bounded between $0$ and $1$.


How can I make ELIZA more realistic? I’ve coded a simple ELIZA chatbot for a high school coding competition. The chatbot is part of an app that’s designed to help its user cope with depression, anxiety, and similar mental health disorders. It uses sentiment analysis to identify signs of mental illness, and to track it's user's progress toward "happiness" over time.

My question is, what steps can I take to make it more realistic (without using some pre-existing software, library, etc, which isn't allowed)? Also, are there any existing tables of questions/responses I can add to my ELIZA bot's repertoire so that it can handle more conversations?

One 'easy' way would be to have some sort of conversational memory, where you track what the user has said already. I don't know how complex your patterns are, but if you could recognise names and track references, you could try and build up a mental model of the user's relationships with other people, and perhaps refer to that in your bots responses.

The latter will be quite advanced, but keeping track of things said earlier and referring back to them on occasion might make it appear a lot more capable.

As an added bonus, track changes in the user's sentiment scores, and see if you spot a pattern in the conversation (maybe over the course of multiple conversations) to see which bot utterance have the biggest (positive or negative) effect on the user's mood.


Why are RNNs used in some computer vision problems? I am learning computer vision. When I was going through implementations of various computer vision projects, some OCR problems used GRU or LSTM, while some did not. I understand that RNNs are used only in problems where input data is a sequence, like audio or text.
So, in kernels of MNIST on kaggle almost no kernel has used RNNs and almost every repository for OCR on IAM dataset on GitHub has used GRU or LSTMs. Intuitively, written text in an image is a sequence, so RNNs were used. But, so is the written text in MNIST data. So, when exactly is it that RNNs(or GRUs or LSTMs) need to be used in computer vision and when don't?

There are tasks in computer vision where recurrent neural networks (RNNs) can be useful because there's some sequential sub-task in the main task.
For instance, in the paper Long-Term Recurrent Convolutional Networks for Visual Recognition and Description, the authors investigate the use of a neural network that is both recurrent and convolutional to solve certain computer vision tasks that also have a sequential component/part, such as video recognition tasks, image to sentence generation problems, and video narration challenges.
There are other papers that investigate the combination of convolutional and recurrent layers, such as Recurrent Convolutional Neural Networks: A Better Model of Biological Object Recognition, which also has a biological motivation.


How to predict Q-values based on stack of frames I decided to train deep Q-learning agent based on getting raw pixels from environment.I have one particular problem:when I input stack of frames, suppose 4 consecutive frames, if action space is 6,then output is 4 by 6 matrix.So which one is real Q-value?I mean, I input batch of frames and it outputs batch of values and question is which is real Q-value out of those batch values?

You do not output the batch of Q-values. Input frame stacking is needed to gain full observability of the environment. In your case the output would be 6 elements for your current frame. If $F$ is a frame then you would stack 4 frames $[F_{k-3}, F_{k-2}, F_{k-1}, F_k]$ and the output would be 6 Q-values for frame $F_k$.


Neural Networks unmatched on MNIST? I remember reading or hearing a claim that at any point in time since the publication of the MNIST dataset, it has never happened that a method not based on neural networks was the best given the state of science of that point in time.

Is this claim true?

To quote the relevant Wikipedia article: "The original creators of the database keep a list of some of the methods tested on it.[5] In their original paper, they use a support vector machine to get an error rate of 0.8 percent"

Feel free to look up that original paper, but to me the quote strongly suggests that the first record holder was a support vector machine.

Edit: As liori points out the quote is misleading: In the original paper Yann LeCun et al. actually tried a slew of methods and one version of ConvNet scored best (0.7).

But according to the MNIST-webpage, after that initial paper, DeCoste and Scholkopf, MLJ 2002, reached an error of 0.56 with a SVM. So if that webpage is complete, the claim is still false.


A solution for a famous problem in RL I'm here to ask you for a solution on this problem which is: how to use Reinforcement Learning in Immersive Virtual Reality to make a person move to a specific location in a virtual environment. As you know reinforcement Learning is a sub-area of Machine Learning in which an active entity called an agent interacts with its environment and learns how to act in order to achieve a pre-determined goal. The Reinforcement Learning had no prior model of behaviour and the participants no prior knowledge that their task was to move to and stay in a specific place. The participants were placed in a virtual environment where they had to avoid collisions with virtual projectiles. Following each projectile the agent analysed the movement made by the participant to determine paths of future projectiles in order to increase the chance of driving participants to the goal position and make them stay there as long as possible.

Update 1: Download: Reinforcement Learning as a tool to make people move to a speciﬁc location in Immersive Virtual Reality

Selecting a Scenario for Comprehension's Sake

Movement through a modeled reality is an area that was under development when I entered the research community.  It is not a problem though.  It is a nearly infinite set of problems, an area of research of interest to robotics engineering and gaming.

The solution cannot be very specific when so many details are left out of the scenario definition.  Although I am fine with specifying a general solution, pure math and system architecture is not met with much enthusiasm by most of those interested in answers in this exchange, so I will make some assumptions.

Those with good mathematical and systems design backgrounds will be able to extrapolate the general approach from the more specific scenario in this answer.  I'll place pertinent general theory inline to facilitate that generalization.

Narrowing the Specification Incompletely

The vehicle for movement is not specified, so I will assume the participants are knights on horses.  Linear movements in Cartesian coordinates as in a CAM move command for a CNC machine head is an unlikely use of reinforcement, and aircraft lift can stall, complicating the problem.  Horses were picked from among the remaining list of typical vehicles (cars, bicycles, on foot, and horses).
Time constraints were not specified, so it will be assumed there are none except the time to reach the goal cannot be infinite unless there is no available path at all to the destination.
The nature of the obstacles and their movements were not specified, so I will assume they are solid and that only other knights move on their own horses.
Risks associated with making contact with an obstacle was not specified, so I will assume the horses will not run into one another or stationary objects and will step over small ground objects or jump over slightly larger ones if approaching at sufficient speed.
What will happen when a collision is about to occur was not specified, so I will assume that the horse will decelerate.
What will happen if a horse passes under an obstacle will be assumed the hospitalization or death of the knight, since they cannot easily duck in their armor, leaving the objective unmet.
The capabilities of the participants is not specified, so I will assume that the knights can see stereoscopically, are equipped with compasses, can turn their head and eyes, can control the reigns in the usual horseback riding way, can urge the horse forward with the vocal sound, "Ya!" and/or a light poke with both feet, hear and detect the volume of an auditory beacon, and sense acceleration in 3D to detect horse behavior.
The specs of the other participants is not given, so I will assume the knights along with their horses are of equal volume.
The obstacle statistics are not given, so I will assume the objects to be chaotically sized, shaped, colored, and placed such that their total volumetric space consumption is 1% and the mean object volume is the same as the volume of a knight and her or his associated horse.
The scene is lit from one distant source so that shadows and shadowing are present and the entire scene is bounded by constants in altitude, longitude, and latitude dimensions.
The coordinate system is not specified, the system is modeling equatorial terrestrial space, so latitude and longitude coordinates are not substantially ansiotropic.
The start and end positions are of specific latitude and longitude and are sufficiently distant from one another so that line of sight between the start and end positions is extremely unlikely.
How the end position will be known is not specified, so I will assume the end position is equipped with an auditory beacon.
We will assume a zero sum game, where the achievement of the goal is not mutually exclusive, and we will assume collaboration linguistically is not possible, although collaborative strategy may emerge out of learned behavior organically.  (That's an entire other topic.)

Mathematical precision is missing in the above definition, with several parameters only roughly defined, and the chaotic sizing, shaping, and positioning becomes a realistic challenge for reinforcement software engineering for the general scenario.

Conversely, engineering a solution for this specific case within the field of VR motion with the application of reinforcement concepts can proceed without mathematical abstractions that require much advance (and advanced) study and laboratory experience.

Generalization

The above adequately defines system E (environment) in combination with system V (vehicles) of which there are N, one for each participant.  Discrete changes to command signals C are multidimensional and A (acquired samples) are also multidimensional.

Acquisition Channels

Sampled audio vectors of spectral spectral distributions in musical half tones (frequency ratio of 21/24), sampled in frames of constant period.
Sampled visual matrices in yuv420p form, sample
Sampled tactile vectors of 3D acceleration force

The audio and  vectors are maintained at constant levels as inputs to the base neural net until the next vector is acquired.  The video matrices are fed into a convolutional neural net as is customary.  (See the work of Google researchers Sergey Levin, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen in August 2016 for their approach and references for background research that guided them.)

Control Channels

Head position relative to the vehicle direction
Left reign position
Right reign position
Activation of, "Ya!" vocalization
Activation of the light poke

The output layer of the base neural net must be a real number with the appropriate range for all five, since the volume of the vocalization and the lightness of the poke have informational meaning to the motor control of the vehicle.

Real Time Learning

Real time learning requires at least one model of wellness to provide the reinforcement signal to the base behavioral network.  In many cases, as suggested in early cybernetics work prior to the advent of digital systems, convergence requires more than one wellness variable.

Vector control of reinforcement is not well developed in open software yet, however the concepts of multidimensional gradients (Jacobian and Hessian matrices) are standard elements in artificial network theory and can be extended from matrices to cubes.  Any intermediate calculus text will provide the theory applicable to gradient descent with a curved surface.

Performing the back-propagation effectively when more than one degrees of freedom are present is an interesting problem that is certainly researched and has been deployed to production in commercial and military applications.

Such cannot be described here because the mechanics of back-propagation with vector reinforcement signaling of which I am aware is currently either company confidential or classified.  As with much technology, it may be released for publication in the future as open source code emerges independently over time.

These are probably the best choices for the first and second channels of learning reinforcement signaling to implement and tune.  I doubt (but have only intuition to offer as reasoning) that only one channel will produce a very effective reinforced learning.  The base network necessary for this scenario will be too deep to train with out both types of proximity estimations (1) beacon and (2) nearest obstacle.

Wellness of Behavior Modelling

The simplest and first cut at modelling participant behavioral wellness (proficiency in searching for the beacon) is the rate of change in beacon volume.  It is a distorted estimate of proximity to the beacon, but systematically so.  The distortions in the correlation between the differential of volume with respect to position (not time) is related to the effects of obstacles on sound.

Wellness of Position Modelling

Further development could add filtering out of transient attenuation created by close proximity to an object, which would require two additional neural nets, (a) to detect closest object proximity from patterns in audio and visual inputs and (b) to use acceleration to approximate change in position in the latitude-longitude plane and correlate filtering of audio volume changes with movement to better filter out transient changes not related to proximity of the beacon.

The tone of the beacon may be added as a third to improve rate of learning.

Crash level accelerations can be added over multiple game plays as a fourth.

Additional Determinations

The determination of initial state and meta-parameters for the base neural network and the connectivity between the models and the reinforcement signal to the base network is beyond the scope of this answer, requiring experimentation and possibly months of intensive analysis for this (or any) semi-specific case.

The only known systems that handle general cases without defining environment E, participant P, and their quantity N are DNA based systems that have developed such general adaptive capabilities over billions of years.


What would be a typical pre-processing and data normalization pipeline for time series data (for non-linear models such as neural networks)? I've started to work on time series. I was wondering what would be the best data normalizing and pre-processing technique for non-linear models, specifically, neural networks.
One I can think of is min-max normalization
$$z = \frac{x - min(x)}{max(x) - min(x)}$$

I would say any normalization such as min-max or standard deviation is fine as far as the scaling factor is provided as a feature, since time-series of different scale might behave differently.


Is it possible to control asymptotic behaviour of neural network models? Is it possible to specify what the asymptotic behaviour of a Neural Networks (NN) model should be?
I am thinking of a NN which tries to learn a mapping $\vec y=f(\vec x)$ with $\vec x$ a vector of features of dimension $d$ and $\vec y$ a vector of outputs of dimension $p$.
Is it possible to specify that, for instance, the NN should have a fixed value when $x_1$ goes to infinite?
I mean:
$$
\lim_{x_1\to \infty} f(\vec x) = \vec c
$$
If it is not possible with NN, do you know other machine learning models (for instance  Gaussian Process Regression or Support Vector Regression) which have a known asymptotic behaviour?

For standard NNs, their extrapolation behavior an important aspect for financial applications cannot be controlled due to complex functional forms typically involved.
Neural Networks with Asymptotics Control discuss how they overcome this significant limitation and develop a new type of neural networks that incorporate large-value asymptotics, when known, allowing explicit control over extrapolation. This new type of asymptotics-controlled Neural network is based on two novel technical constructs:

A Multi-Dimensional Spline Interpolator with Prescribed Asymptotic Behavior
and A Custom NN layer that guarantees zero asymptotics in chosen directions.

Let $f(x)$  be the Function that we want to approximate, while preserving its asymptotics. (It goes without saying that the asymptotics needs to be known.) As the first step, we find a control variate function $S(x)$ that has the same asymptotics as $f(x)$.

Multi-Dimensional Inputs - the most interesting case - these asymptotics could either be known in all directions, or only in some; naturally, as much information about the asymptotics as is available should be incorporated into the control variate function $S(x)$.

For step one, they show how to construct a universal control variate, a multi-dimensional spline $S(x)$ that has the same asymptotics as $f(x)$; and that can be used in all situations. In specific applications, of course, if more is known about the function $f$; a better choice of the control variate could be available.

For step two, we design a custom NN layer that guarantees zero asymptotics in all directions, with fine control over the regions where the NN interpolation is used and where the asymptotics kick in. they approximate the residual function $R(x) = S(x) - f(x)$ with a special NN that has vanishing (zero) asymptotics in all, or some, directions.

One-Dimensional Spline with Asymptotics Control:
The paper discussed that Instead of using the natural boundary conditions, i.e. setting second derivatives
to zero at the boundaries, they have fixed the first derivatives at boundary points to
arbitrary values. $\color{blue}{\text{Clearly, this paves the way to control asymptotics}}$.
Suppose that we know the behavior of the original, calculation-heavy function in its tails, i.e. $f(x) \simeq f_{−}(x)$ for $x<h_0$ and $f(x) \simeq f_{+}(x)$  for $x>h_{N+1}$
for large negative $h_0$ and large positive $h_{N+1}$. First, we
expand the set of spline nodes to include points $h_0$ and $h_{N+1}$ and make sure
the spline passes through them, $S(h_0) = f_{−}(h_0)$ and $S(h_{N+1}) = f_{+}(h_{N+1})$. To
finish, they have specifies first order derivatives at these new boundary points.
$$S^{\prime}_0(h_0) = f^{\prime}_{−}(h_0), S(h_{N+1}) = f^{\prime}_{+}(h_{N+1})$$
This fully specifies the approximating function $\tilde{S}(x)$,
$$\tilde{S}(x)=\left\{\begin{array}{ll}f_{-}(x), & x \leq h_{0} \\ S(x), & h_{0}<x<h_{N+1} \\ f_{+}(x), & h_{N+1} \leq x\end{array}\right.$$
One-Dimensional Spline with Asymptotics Control: Continuous Second Derivatives
where they have discussed how they have constructed a control variate function based on the spline like above, but with the continuous second derivative at the endpoints. In order to do that, they have picked a point between $h_0$ and $h_1$, say $h_{\frac{1}{2}}$ , and find, analytically, the value $S(h_{\frac{1}{2}})$ such that the second derivative at $h_{0}, S^{\prime \prime}\left(h_{0}\right)= f_{-}^{\prime \prime}\left(h_{0}\right)$ is matched.

Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems. This paper deals with studying the asymptotical properties of multilayer neural networks models used for the adaptive identification of wide class of nonlinearly parameterized systems in stochastic environment.

A study of the asymptotic behavior of neural networks. where they have studied of neural networks modeled as a set of nonlinear differential equations of the form $TX+X=Wf(X)+b$, where $X$ is the neural membrane potential vector, $W$ is the network connectivity matrix, and $F(X)$ is the nonlinearity (an essentially sigmoid function). Topologies of neural networks that exhibit asymptotic behavior are established as behavior depends solely on the topology of the network. Moreover, the connectivity $W$ need not be symmetric. The simulated behavior of typical neural networks is presented

Refrence
Neural Networks with Asymptotics Control
Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems
A study of the asymptotic behavior of neural networks


What kind of education is required for researchers in AI? Suppose my goal is to collaborate and create an advanced AI, for instance one that resembles a human being and the project would be on the frontier of AI research, what kind of skills would I need?

I am talking about specific things like what university program should I complete to enter and be competent in the field. Here are some of the things that I thought about, just to exemplify what I mean:

Computer sciences: obviously the AI is built on computers, it wouldn't hurt to know how computers work, but some low level stuff and machine specific things does not seem essential, I may be wrong of course.
Psychology: if AI resembles human beings, knowledge of human cognition would probably be useful, although I do not imagine neurology on a cellular level or complicated psychological quirks typical to human beings like the Oedipus complex would be relevant, but again, I may be wrong.

As a full-time AI researcher myself, I'd say that a PhD in machine learning would certainly be one useful option.

However, in order make much-needed progress, AI needs avoid falling into the trap of thinking that currently fashionable methods are any kind of 'silver bullet'. There's some danger that a PhD that heads straight into (say) some sub-sub-sub area of DL would end up imposing too much bias on the student's subsequent perspective.

AI research is an essentially multi-disciplinary activity. Other possible backgrounds therefore include:

Mathematics or physics (to first degree or PhD level). A strong background in either of these never did anyone any harm. People who are competent in these fields tend to be able to turn their abilities to new domains relatively easily.
Software Engineering. One of the things that AI needs are integrative architectures for knowledge engineering. Here's why. I believe that one of the reasons that we haven't yet managed to do OCR at the level of a 5 year old is that we've yet to accept that we have to 'build a sledgehammer to crack a nut'. Software architects are used to managing large-scale complexity, so they may be able to help.
Cognitive Science, Psychology, Cognitive Linguistics. The reasons here are obvious.

Above all, I personally think that a good AI researcher should be creative, inquisitive and prepared to question received wisdom, all of which are more important in practice than specifics of their background.


Will it ever be possible to construct a neural network that could have the features of human brain? I have a question. Will we be able to build a neural network that thinks abstractly, has the creativity, feels and is conscious?

Maybe in the distant future they could build a computer powerful enough to simulate the individual neurons of an entire human brain. Then they could carefully copy/paste the connectivity of a sample brain into the computer simulation.

Given that this extreme seems physically possible, it stands to reason that much simpler/smaller alternatives could be engineered in the future assuming continuous advancements in technology.


Looking for approach for monetary evaluation of AI potential in 2025 I need to evaluate the monetary potential of AI at my company for the year 2025 to justify a long term investment in a dedicated AI center of competence. The departments in which AI would be used later on (Purchasing, R&D, Manufacturing, …) are generally very interested and some of them have already built some prototype use-cases. However, they are unwilling to predict and quantify any future potential. The investments in AI can only be justified if we can state the monetary benefit.

Are there any creative approaches out there to estimate the monetary potential of AI?

Any help is greatly appreciated!

It would be irresponsible to try. Several in the past, dating back centuries led to various trends that some call winters. I don't agree they are winters. They are just people returning to reality. Creating intelligence in computers has been progressing forward steadily since Charles Babbage. It has accelerated due to advances in digital electronics, in discrete mathematics, and in brain imaging.

It is possible to predict, with some unavoidable uncertainty, the impact on the profitability of existing products or services five years into the future, provided the AI efforts in the corporation are planned at a high level first.

The main issue is that what we mean when we say the word intelligence is really a collection of abilities. Tesla died with no assets yet his designs are in our appliances and the grids that power them. A billionaire may not be able to draw a sketch of himself worth a penny, and Michelangelo may not have been able to design a way to transmit power to homes and businesses.

There is no one thing called intelligence. It is many things. That's why computers exceeded humans in arithmetic long ago, but there are no combinations of hardware and software that can even come close to writing a physics textbook or even passing an arbitrary high school physics final exam. There is much more not known about intelligence then there is known about it.

Monetary evaluation may be necessary as part of the solution, but the end goal is monetary projection. That means one must evaluate the state and trajectory of something AI experts don't yet understand well. We can guess accurately that, if the direction of an AI department or AI groups within multiple existing departments is well managed, gains can be made. We can evaluate the use of AI in the past within the same business type and use the trend we see to make projections along the same line based on how technology trends usually progress. We can also factor in the emergence of competitive businesses also progressing with their use of AI.

That purchasing, R&D, manufacturing, and other departments are interested in AI, have done some initial play with examples and maybe built out some proof of concept code but are unwilling to quantify future potential is not surprising. It is not any easy financial modelling task. To begin the task, some information must be collected and scrutinized to ensure it is an accurate starting assessment.

What are the kinds of products and services of the business?
What are the critical corporate functions that support those?
What are the key technological, market, and production characteristics of the business that places it in a group of like businesses?
What currently developed and known-to-be-effective AI technologies most likely produce profit enhancements for this business?
What past technology advancements are most like those current ones?
Quantitatively, what were the financial outcomes of the leveraging of those past technologies over time?
How is competitive leveraging of the same technologies by competitive businesses likely to draw business and divide profit margins?

Once this information is gathered and its accuracy is ensured, a financial model can be constructed and implemented in software and projections of various scenarios can be made. It is a development project in itself. Many businesses are bypassing this project and just assuming that the current AI craze must have a high ROI because large corporations have made the choice down that road. This question exhibits a more prudent course of action.

In addition to the departments listed, there are a few more to consider.

Marketing
Finance
Strategic planning (at the board and senior management level)
Accounts receivable


About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends? I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are all non linear, and they all can be different. And in the first layer, they are all linear.
Why?
On what does this depends?

When it is said "the neural network learns automatically", in colloquial words, what does it mean?

AFAIK, one first train the NN, then at some point NN learns. When does the "automatically" enters then?
Thanks in advance for your help.

Rosenblatt was probably discussing a specific architecture, for which there are many.  However, for general purpose feed-forward back-propagation ANNs used for function aproximation and classification analysis, you can use whatever activation functions you want on the input-side, hidden layers, and output-side.  Examples are identity, logistic, tanh, exponential, Hermite, Laguerre, RBFN, ReLu, softmax, etc.  "Automatically," likely refers to the iterative learning process, which tends to be similar to gradient descent, during which partial derivatives of prediction error w.r.t to coefficients decrease.


What are some use cases of few-shot learning? Besides computer vision and image classification, what other use cases/applications are for few-shot learning?

Few-short learning (FSL) can be useful for many (if not all) machine learning problems, including supervised learning (regression and classification) and reinforcement learning.

The paper Generalizing from a Few Examples: A Survey on Few-Shot Learning (2020) provides an overview (including examples of applications and use cases) of FSL. Their definition of FSL provided is based on Tom Mitchell's famous definition of machine learning.

  Definition 2.1 (Machine Learning [92, 94]). A computer program is said to learn from experience $E$ with respect to some classes of task $T$ and performance measure $P$ if its performance can improve with $E$ on $T$ measured by $P$.

Here's the definition of FSL.

  Definition 2.2. Few-Shot Learning (FSL) is a type of machine learning problems, specified by $E$, $T$ and $P$, where $E$ contains only a limited number of examples with supervised information for the task $T$.

Specific examples of applications of FSL are

character generation
drug toxicity discovery
sentiment classification from short text
object recognition


Why Autoencoder Weights Are Not Always Tied To me, tying weights in an autoencoder makes sense if we think of the auto encoder as doing PCA. Why in any situation would it make sense to not tie the weights? If we don't tie the weights, would it not try to learn something that is PCA anyway or rather something that might not be as optimal as PCA?
Also, if weights are not tied, it doesn't make sense to me that the auto-encoder is invertible i.e. if the decoder is looking for an inverse operation because it's a mapping between spaces of different dimension which should not invertible.
So, if the weights are not tied then why do we expect the decoder to learn anything meaningful i.e neither PCA nor an inverse operation?

We expect the decoder to learn anything meaningful without tying the weights because the loss function is calculated between the input and reconstructed output and training will minimize that loss. The untied autoencoder's decoder will learn to transform the embedding back into the input. Not tying the weights gives more representational power and also increases chances of overfitting.
If you are running into overfitting while training your autoencoder tying your weights is an option. It is a regularization method. Not tying them is fine if you are getting any overfitting or want to solve overfitting by an alternative method.


what will i be able to do in the end of AI: modern approach? i just started the book and i was wondering , what will i be able to do in AI by the end of the book ?
and more particularly, what is my position with Reinforcement Learning, deep neural networks and NEAT algorithm, will i study them enough in the book

will i need to read another more specialized book, if so what would you recommend

"Artificial Intelligence: A Modern Approach" (AIMA) by Russel and Norvig is a general introductory book on AI. That means it not only covers sub-symbolic AI (like machine learning) but also symbolic AI. Therefore, it can "only" give you an overview of each topic (I put only in quotation marks since it is actually quite ambitious to cover all topics of AI in a single book!).

Regarding the topics you have mentioned that means:

you will have read about the basics of neural networks but will not have seen much about deep nets (the 3rd edition only has 10 pages on neural networks, i.e. it's really basic)
you will have read the basics of reinforcement learning (the 3rd edition has 23 pages on this, i.e. it's basic but more comprehensive than neural networks and enough to read papers about the topic)
you will not have read anything about NEAT but have read about the basics of genetic algorithms and neural networks which NEAT is based on (and can read the corresponding papers afterwards)

So, regarding your preferred topics, AIMA is one way to get started if you're not only looking for a selectively deep but also a broad understanding of AI. The road could go like this:

Read a general AI book, e.g. AIMA
Read a general ML book, e.g. "The Elements of Statistical Learning" by Hastie, Tibshirani, Friedman
Read deep dive books on selected topics, e.g. "Deep Learning" by Goodfellow, Bengio, Courville
Read papers for your topics of interest, e.g. "Mastering the Game of Go without Human Knowledge" by Silver et al.

The quicker start would be of course to start at step 2 or 3 and only go to previous steps to look something up if it is required.


Mathematical Analysis of the Loss function of GAN I was pondering on loss function of GAN and following thing turned out
\begin{aligned}
 L(D, G)
& = \mathbb{E}_{x \sim p_{r}(x)} [\log D(x)] + \mathbb{E}_{x \sim p_g(x)} [\log(1 - D(x)] \\
& = \int_x \bigg( p_{r}(x) \log(D(x)) + p_g (x) \log(1 - D(x)) \bigg) dx \\
& =-\left[CE(p_r(x), D(x))+CE(p_g(x), 1-D(x)) \right] \\
\end{aligned}
Where CE stands for cross-entopy. Then, by using law of large numbers:
\begin{aligned}
L(D, G)
& = \mathbb{E}_{x \sim p_{r}(x)} [\log D(x)] + \mathbb{E}_{x \sim p_g(x)} [\log(1 - D(x)] \\
& =\lim_{m\to \infty}\frac{1}{m}\sum_{i=1}^{m}\left[1\cdot \log(D(x^{(i)}))+1\cdot \log(1-D(x^{(i)}))\right]\\
& =- \lim_{m \to \infty} \frac{1}{m}\sum_{i=1}^{m} \left[CE(1, D(x))+CE(0, D(x))\right]
\end{aligned}
As you can see, I got very strange result.This should be wrong intuitively because in last equation first part is for real samples, and second is for generated samples. However, I am curious about where is the mistakes?
(Please explain with math)
(Also, I am aware of this post and it explained nothing )

I guess the issue is you lost track of where the samples came from and since you requested a math explanation I'll try to go step by step using my notation and without checking other material to avoid being biased by how other authors present it
So we start from
$$ L(D,G) = E_{x \sim p_{r}(x)} \log(D(x)) + E_{x \sim p_{g}(x)}\log(1 - D(x)) $$
then you apply the definition of $E_{\cdot}(\cdot)$ operator in the continuous case
$$ L(D,G) = \int_{x} \log(D(x)) p_{r}(x)dx + \int_{x}\log(1 - D(x))p_{g}(x)dx $$
then you Monte Carlo sample it to approximate it
$$ L(D,G) = \frac{1}{n} \sum_{i=1}^{n} \log(D(x_{i}^{(r)}))  + \frac{1}{m} \sum_{j=1}^{m}\log(1 - D(x_{j}^{(g)})) $$
As you can see here I have kept the samples from the 2 distributions separated and used a notation that allows to track their origin so now you can use the right label in the Cross Entropy
$$ L(D,G) = \frac{1}{n} \sum_{i=1}^{n} L_{ce}(1, D(x_{i}^{(r)}))  + \frac{1}{m} \sum_{j=1}^{m} L_{ce}(0, D(x_{j}^{(g)})) $$
But you could also have decided to merge the 2 integrals before to have
$$ L(D,G) = \int_{x} \left( \log(D(x)) p_{r}(x) + \log(1 - D(x))p_{g}(x) \right) dx $$
which is mathematically legit operation, however the issue is when you try to
discretize this with Monte Carlo sampling.
You can't just replace the integral with one sum since you are Monte Carlo sampling and here, contrary to what we have done above, you do not have 1 distribution per integral to sample but in the same integral you have 2 distributions and for each sample you have to say what distribution it comes from which is where the issue is in your notation since you lost track of this information and it seems all the samples come from one distribution


Transposition table is only used for roughly 17% of the nodes - is this expected? I'm making a Connect Four game using the typical minimax + alpha-beta pruning algorithms. I just implemented a Transposition Table, but my tests tell me the TT only helps 17% of the time. By this I mean that 17% of the positions my engine comes across in its calculations can be automatically given a value (due to the position being calculated previously via a different move order).

For most games, is this figure expected? To me it seems very low, and I was optimistically hoping for the TT to speed up my engine by around 50%. It should be noted though that on each turn in the game, I reset my TT (since the evaluation previously assigned to each position is inaccurate due to lower depth back then).

I know that the effectiveness of TT's are largely dependent on the game they're being used for, but any ballparks of how much they speed up common games (chess, go, etc) would be helpful.

EDIT - After running some more tests and adjusting my code, I found that the TT sped up my engine to about 133% (so it took 75% as much time to calculate). This means those 17% nodes were probably fairly high up in the tree, since not having to calculate the evaluation of these 17% sped up things by 33%. This is definitely better, but my question still remains on whether this is roughly expected performance of a typical TT.

I don't think that's necessarily a strange number. It's impossible for anyone to really tell you whether that 17% is "correct" or not without reproducing it, which would require much more info (basically would have to know every single tiny detail of your implementation to be able to reproduce).

Some things to consider:

The size of your transposition table / the number of bits you use for indexing into the TT. If you have a relatively small TT, meaning you use relatively few bits for indexing, you'll have bigger probabilities of collisions. That means you will have to replace existing entries more often, which means they might no longer be in the table anymore by the time you encounter transpositions during the search.
Where in the search tree are the nodes located that are recognized as transpositions already in the table? If you detect transpositions very high up in the search tree, you save a lot more search time than if you detect a transposition somewhere deep down in the search tree; once you detect a transposition that has already been searched sufficiently deep for the value stored in the table to be valid, you can cut off the complete subtree below that node from the search. This becomes more valuable as it happens closer to the root. So, just the number "17% of nodes" doesn't really tell us much.
Are you using iterative deepening? Since you mentioned only minimax + alpha-beta pruning in the question, I suspect you're not using iterative deepening. TTs become significantly more valuable once you do use iterative deepening, because then almost every state encountered becomes a "transposition". You'll already have seen all those states in a previous iteration with a lower search depth limit. Now, it is important to note with this combo of ID + TTs, that you can no longer completely cut off searches for all recognized transpositions. If an entry in the table holds a value that was computed with a search depth of $d$, that value will no longer be valid when performing a subsequent iteration of ID with a max search depth of $d + 1$ for example. However, that "outdated" value stored in the TT can still be used for move ordering, which can lead to significantly more prunings from alpha-beta pruning.
How efficient is the remainder of your engine? A TT is not 100% free, it takes a bit of additional time too (for example to compute the hash values for your game states). If the rest of your engine is relatively slow (i.e. inefficient implementation for playing moves, copying game states, etc.), the computational overhead of the TT won't matter much and even a low number of recognized transpositions will still be valuable. If the rest of your engine is very fast, it'll be more important to have a high number of transpositions for the TT to be really valuable.

As an "educated guess", I'd say the number of 17% you describe is not necessarily strange. Especially given your edit to the question, where you indeed mention that it is likely that transpositions are found high up in the tree (close to the root). When this happens, you immediately remove the probability of recognizing all those states deeper down in the tree of getting recognized as transpositions yourself. So, the pool of states that could "potentially" be found in the TT is much less than 100% of the states stored in the TT.

It's really just that though, just an educated guess. It's going to be very difficult for anyone to give a conclusive "yes" or "no".


How does the policy gradient's derivative work? I am trying to understand the policy gradient method using a PyTorch implementation and this tutorial.

My first question is about the end result of this gradient derivation,

\begin{aligned}
\nabla \mathbb{E}_{\pi}[r(\tau)] &=\nabla \int \pi(\tau) r(\tau) d \tau \\
&=\int \nabla \pi(\tau) r(\tau) d \tau \\
&=\int \pi(\tau) \nabla \log \pi(\tau) r(\tau) d \tau \\
\nabla \mathbb{E}_{\pi}[r(\tau)] &=\mathbb{E}_{\pi}[r(\tau) \nabla \log \pi(\tau)]
\end{aligned}

Mainly in this equation

$$\nabla \mathop{\mathbb{E}_\pi }[r(\tau )] = \mathop{\mathbb{E}_\pi }[r(\tau )\nabla log \pi (\tau )]$$

Does expectation follow a distributive or associative property?

I know that expectations of a function can be written as below

$$\mathop{\mathbb{E}}[f(x)] =\sum p(x)f(x)$$

Then can we rewrite the first equations as

$$\mathop{\mathbb{E}_\pi }[r(\tau )\nabla log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\nabla log \pi (\tau )] \\= \sum p(\tau)r(\tau ) \,\, \sum p(\tau)\nabla log \pi (\tau ) \\
= p(\tau) \sum r(\tau ) \nabla log \pi (\tau )$$

The problem is when I compare  this to PyTorch implementation (line 71-74)

for log_prob, R in zip(policy.saved_log_probs, returns):
    policy_loss.append(-log_prob * R)
optimizer.zero_grad()
policy_loss = torch.cat(policy_loss).sum()

The  pytorch implementation simply multiplied log probability and reward -log_prob * R and then summed the vector torch.cat(policy_loss).sum() there is no $p(\tau)$. What is really happening here?

The second question is the multiplication of log probability and reward in PyTorch implementation -log_prob * R, PyTorch implementation has a negative log probability and derived equation has a positive one $\mathop{\mathbb{E}_\pi }[r(\tau )\nabla log \pi (\tau )]$. What is the need for multiplying log probability with a negative value in PyTorch implementation?

I have only a basic understanding of maths and that's why I am asking this question here.

Edit: found a better derivation of above equation https://youtu.be/Ys3YY7sSmIA?t=3622

You cannot do this:

  $\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\bigtriangledown log \pi (\tau )]$

That is because $r(\tau )$ and $\bigtriangledown log \pi (\tau )$ are correlated by their dependence on $\tau$. In a simpler concrete example, if your expectation was over simple equiprobable discrete distribution where $\tau$ could be any integer in range $[1,10]$, then $\mathop{\mathbb{E} }[\tau^2] = 38.5$ whilst $\mathop{\mathbb{E} }[\tau]\mathop{\mathbb{E} }[\tau] = 30.25$

  The  pytorch implementation simply multiplied log probability and reward -log_prob * R and then summed the vector torch.cat(policy_loss).sum() there is no $p(\tau)$. What is really happening here?

The purpose of transforming the gradient into an expectation $\mathbb{E}$ for the policy gradient theorem, is so that you can estimate it using samples taken from the distribution. Typically, you don't know $p(\tau)$, but you do know that if you follow the same process where $p(\tau)$ applies (i.e. measure the return from the environment whilst following the policy represented by the policy function) that you will get an unbiased sample from that distribution.

So what is going on here is that you throw away the outer expectation $\mathbb{E}_{\pi}[]$ and replace it with a stochastic estimate for the same value based on taking samples. The samples are naturally obtained with distribution described by $p(\tau)$, if you follow the policy function when making action choices.

  The second question is the multiplication of log probability and reward in pytorch implementation -log_prob * R, pytorch implementation has a negative log probability and derived equation has a positive one $\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )]$. What is the need for multipling log probability with negative value in pytorch implementaion?

I don't know the code, but this very likely because of a sign change brought on by considering how to respond to the gradient estimate.

There is a clue in the use of the name "loss". To maximise return in policy gradient methods, you can perform gradient ascent based on the estimated gradient as the goal is to find higher values. However, it is more usual in NN libraries to perform gradient descent in order to minimise a loss function. That is a likely cause of the sign reversal here.


How to implement AI strategy for Mastermind I'm looking to implement a AI for the turn-based game Mastermind in Node.JS, using Google's Tensorflow library. Basically the AI needs to predict the 4D input for the optimal 2D output [0,4] with a given list of 4D inputs and 2D outputs from previous turns in the form of [input][output].

The optimal output would be [0,4], which would be the winning output. The training data looks like this:

[1,2,3,4][0,1] [0,5,2,6][3,1] [0,2,5,6][2,2] [6,5,2,0][4,0] [5,2,0,6][0,4]

So given these previous turns

[1,2,3,4][0,1] [0,5,2,6][3,1] [0,2,5,6][2,2] [6,5,2,0][4,0]

the AI would predict an input of [5,2,0,6] for the output [0,4].
I've looked at this post but it talks about only inferring input for a output without any context. In Mastermind, the context of previous guesses and results from them are critical

My algorithm would need to use the information from previous turns to determine the best input for the winning output ([0,4]).

So my question is: How can I implement AI for Mastermind?

You could possibly apply neural networks, reinforcement learning to summarise results of previous choices (what you are calling context) and use score predictions to suggest the next turn's guess. However, the game of Mastermind has a small search space and it is possible to process this "context" more directly by refining a set of guesses. This will be much more efficient and simpler to understand than a neural network approach. It would be very hard to make a neural network variant which was as efficient - either in terms of CPU time, or in terms of number of turns it takes to find a solution.

In practice, a Mastermind solver is much like a Hangman solver, or a Guess Who? solver. You have an initial large set of all possible answers, and need to narrow it down to a single correct answer. You do this by processing after each guess to reduce the set of answers that meet all the constraints that the game has given you so far.

The agent needs to know the score function that compares a target value with the guess and returns the score. Let's call that score(guess, target)

The algorithm looks like this:

(Opponent sets unknown_target)
Initialise possible_answers as list of all valid targets

For each turn:
  Select one of possible_answers as this turn's guess
  Ask opponent for gscore = score(guess, unknown_target)
  If gscore is [0,4] then exit(win)
  If it was last guess then exit(lose)
  For each possible_answer in possible_answers:
    pscore = score(guess, possible_answer)
    If pscore != gscore, then remove possible_answer from possible_answers

You can finesse this for the stage Select one of possible_answers as this turns guess by trying to optimise either by psychological model of opponent or trying to find choices that are likely to cause the best reduction in size of possible_answers. However, a simple random choice should do quite well.

Also worth noting is that the algorithm does not depend on the exact nature of the scoring function, so it is applicable for many variations of guessing games. It does rely on the score providing information that will reduce the remaining set of guesses. In some games that may mean taking more care about the precise nature of a guess, in order to maximise this effect.

Out of interest, I implemented this algorithm and tested it when there were 10 choices at each position (i.e. digits 0 to 9), maximum of 10 guesses allowed, and the target to guess was set randomly. Using random guesses and the algorithm exactly as written, the above approach guessed correctly 9,996 times out of 10,000, and on average the guesser won the game in 6.2 turns.


Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs I have implemented the total loss of my PPO objective as follows:-

total_loss = critic_discount * critic_loss + actor_loss -  entropy_beta * K.mean(-(newpolicy_probs * K.log(newpolicy_probs)))

After training for a few epochs, the entropy term becomes "nan" for some reason. I used tf.Print() to see the new policy probabilities when the entropy becomes undefined, it is as follows-

  new policy probs: [[6.1029973e-06 1.93471514e-08
  0.000299338106...]...]

I am not clear as to why taking log of these small probabilities is coming out as nan. Any idea how to prevent this?

I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.


Why are neural networks preferred to other classification functions optimized by gradient decent Consider a neural network, e.g. as presented by Nielsen here. Abstractly, we just construct some function $f: \mathbb{R}^n \to [0,1]^m$ for some $n,m \in \mathbb{N}$ (i.e. the dimensions of the input and output space) that depends on a large set of parameters, $p_j$. We then just define the cost function $C$ and calculate $\nabla_p C$ and just map $p \to p - \epsilon \nabla_p C$ repeatedly.
The question is why do we choose $f$ to be what it is in standard neural networks, e.g. a bunch of linear combinations and sigmoids? One answer is that there a theorem saying any suitably nice function can be approximated using neural networks. But the same is true of other types of functions $f$. The Stone-Weierstrass theorem gives that we could use polynomials in $n$ variables: $$f(x) =  c^0_0 + (c^1_1 x_1 + c^1_2 x_2 + \cdots + c^1_n x_n) + (c^2_{11}x_1 x_1 + c^2_{12} x_1x_2 + \cdots + c^2_{1n} x_1 x_2 + c^2_{21} x_2x_1 + c^2_{22} x_2x_2 + \cdots) + \cdots,$$
and still have a nice approximation theorem. Here the gradient would be even easier to calculate. Why not use polynomials?

You can indeed fit a polynomial to your labelled data, which is known as polynomial regression (which can e.g. be done with the function numpy.polyfit). One apparent limitation of polynomial regression is that, in practice, you need to assume that your data follows some specific polynomial of some degree $n$, i.e. you assume that your data has the form of the polynomial that you choose, which may not be true.
When you use a neural network to solve a classification or regression problem, you also need to choose the activation functions, the number of neurons, how they are connected, etc., so you also need to limit the number and type of functions that you can learn with neural networks, i.e. the hypothesis space.
Now, it is not necessarily a bad thing to limit the hypothesis space. In fact, learning is generally an ill-posed problem, i.e. in simple terms, there could be multiple solutions or no solutions at all (and other problems), so, actually, you often need to limit the hypothesis space to find some useful solutions (e.g. solutions that generalise better to unseen data). Regularisations techniques are ways of constraining the learning problem, and the hypothesis space (i.e. the set of functions that your learning algorithm can choose from), and thus making the learning problem well-posed.
Neural networks are not preferred over polynomial regression because they are theoretically more powerful. In fact, both can approximate any continuous function [1], but these are just theoretical results, i.e. these results do not give you the magical formula to choose the most appropriate neural network or polynomial that best approximates the desired unknown function.
In practice, neural networks have been proven to effectively solve many tasks (e.g. translation of natural language, playing go or atari games, image classification, etc.), so I would say that this is the main reason they are widely studied and there is a lot of interest in them. However, neural networks typically require large datasets to approximate well the desired but unknown function, it can be computationally expensive to train or perform inference with them, and there are other limitations (see this), so neural networks are definitely not perfect tools, and there is the need to improve them to make them more efficient and useful in certain scenarios (e.g. scenarios where uncertainty estimation is required).
I am not really familiar with research on polynomial regression, but it is possible that this and other tools have been overlooked by the ML community. You may want to have a look at this paper, which states that NNs are essentially doing polynomial regression, though I have not read it, so I don't know the details about the main ideas and results in this paper.


Should the exploration rate be reset after each trial in Q-learning? As the title says, should I reset the exploration rate between trials?

I am currently doing the Open AI pendulum task and after a number of trials my model started playing but did not take any actions (i.e. didn't perform any significant swing). The Actor-Critic tutorial I followed did not reset the exploration rate (link) but it seems like there are lots of mistakes in general.

I assume that it should be reset since the model might start from a new unknown situation in a different trial and not know what to do without exploring.

The exploration rate, typically parameterized as epsilon / ε, can be changed on every trial. It depends on the complexity of the model and the goals.

The simplest thing to do is keep exploration rate high and fixed. That means the model will continue to explore new options, even at the cost of not "exploiting" the best available option.

Another option is setting the exploration rate high at the beginning of learning so the model will search the space for possible successful solutions. Then as the model creates a set of policies that are successful for given states, the exploration rate can be lowered or decay. Exploration rate decay can be fixed (i.e., over time there is consistently less exploration and more exploitation). Exploration rate decay can be dynamic and learned. This last option is often the best but also the most complex to implement.

"Dare to Discover: The Effect of the Exploration Strategy on an Agent’s Performance" goes into greater detail on this topic.


In novelty search, are the novel structures or behaviour of the neural network rewarded? I have been reading a lot lately about some very promising work coming out of Uber's AI Labs using mutation algorithms enhanced with novelty search to evolve deep neural nets. See the paper Safe Mutations for Deep and Recurrent Neural Networks through Output Gradients (2018) for more details.

In novelty search, are the novel structures or behavior of the neural network rewarded?

In the paper Exploiting Open-Endedness to Solve Problems Through the Search for Novelty (2008), by Joel Lehman and Kenneth O. Stanley, which introduced the novelty search approach, it is written

  Thus this paper introduces the novelty search algorithm, which searches with no objective other than continually finding novel behaviors in the search space.

and

  instead of searching for a final objective, the learning method is rewarded for finding any instance whose functionality is significantly different from what has been discovered before

and

  The novelty of a newly generated individual is computed with respect to the behaviors (i.e. not the genotypes) of an archive of past individuals whose behaviors were highly novel when they originated

Therefore, the goal of a novelty search is to search for novel behavior and not necessarily novel chromosomes (or genotypes).

In the experiments reported in the novelty search paper, the authors use neural networks to represent the policy that controls a robot that needs to navigate a maze, while evolving these neural networks with NEAT (a neuroevolution method) and a novelty metric (rather than a fitness metric, which is used in the original NEAT). In the same experiments section, Lehman and Stanley write

  Thus, for the maze domain, the behavior of a navigator is defined as its ending position. The novelty metric is then the Euclidean distance between the ending positions of two individuals. For example, two robots stuck in the same corner appear similar, while one robot that simply sits at the start position looks very different from one that reaches the goal, though they are both equally viable to the novelty metric.

Therefore, the evolution of the neural networks, which represent the controllers, is not necessarily guided by the novelty of (the architecture of) the neural networks but by the novelty of the behavior generated by the neural networks, even though novel neural networks might correspond or lead to novel behaviors.


What is the most effective way to build a classifier? At the moment, I am working on a project which requires me to build a naive Bayes classifier. Right now, I have a form online asking for people to submit a sentence and the subject of the sentence, in order to build a classifier to identify the subject of a sentence. But before I train the classifier, I intend on processing all entries for the parts-of-speech and the location of the subject. So my training set will be formatted as:

Sentence: Jake moved the chair.   Subject: Jake
POS-Tagged: NNP VBD DD NN         Location: 0

Would this be an effective way to build the classifier, or is there a better method?

Your approach would definitely work. I would recommend training a variety of classifiers and comparing their performance using multiclass roc analysis. Also, think about other useful features in addition to the ones you mentioned (e.g. pos tag). Feature engineering is one of the most important factors in building good predictive models. Another thing to keep in mind is that the classes could be highly imbalanced which might influence your model's performance.


What does "we wrap the individual and reuse the codons" mean in the paper "Grammatical Evolution" by Neill and Ryan? I've just started learning Grammatical Evolution and I'm reading the paper Grammatical Evolution by Michael O'Neill and Conor Ryan.
On page 3 (section IV-A), they write:

During the genotype-to-phenotype mapping process it is possible for individuals to run out of codons and in this case, we wrap the individual and reuse the codons.

I'm not an English native speaker and I don't understand the meaning of the word "wrap" here. What does it mean?
I understood that, if none of the symbols are terminals, we have to start from the beginning of the genotype again and replace the nonterminal symbols until we have only terminal symbols. But, if I'm correct, when do I have to stop? In the paper, they also talk about non-valid individuals.

In GE, the genotype is a linear sequence of codons. By "wrapping" it, you make it a circular sequence that never ends. It allows you to build a bigger tree, while having only a few codons. Still, it is possible to find such a combination of a genotype and a grammar that defines an infinitely deep expansion — such combinations are hardly suited for practical purposes.
I learned about GE recently. I implemented a GP/GE system to solve the Santa Fe Trail problem. I chose not to perform wrapping to make genotype-to-phenotype mapping more predictable (I always generate enough codons to complete a grammar expansion, and prune unused tails). I also went with a subtree crossover, and a subtree-local mutation, which effectively make the system more a classical GP (with a fancy genotype-to-phenotype mapping) than GE. So there are some options.


How to map X to Y for TensorFlow RNN training data Usually for DNN, I have the training data of matching X (2D) to Y (2D), for example, XOR data:

X = [[0,0],[0,1],[1,0],[1,1]];
Y = [[0],  [1],  [1],  [0]  ];

However, RNN seems strange, I don't get it how to match X to Y, input of RNN layer is 3D and output is 2D (rightclick to open in new tab): https://colab.research.google.com/drive/17IgFuxOYgN5fNO9LKwDijEBkIeWNPas6

import tensorflow as tf;

x = [[[1],[2],[3]], [[4],[5],[6]]];
bsize = 2;
times = 3;

#3d input
input = tf.placeholder(tf.float32, [bsize,times,1]);

cell  = tf.keras.layers.LSTMCell(20);
rnn   = tf.keras.layers.RNN(cell);
hid   = rnn(input);

sess = tf.Session();
init = tf.global_variables_initializer();
sess.run(init);

#results in 2d
print(sess.run(hid, {input:x}));

The example data seen on https://www.tensorflow.org/tutorials/sequences/recurrent are:

 t=0  t=1    t=2  t=3     t=4
[the, brown, fox, is,     quick]
[the, red,   fox, jumped, high]

How to map these data from X (3D input for RNN layer) to Y (2D)? (Y is 2D because RNN layer output is 2D).

I found out how to get 3D output from LSTMCell so that I can matmul with output weights + biases and subtract with expected values:

Inputs & expecteds should be: placeholder(,[times,batch_size,num_inp]) instead of batch_size first then times. However, tf.keras.layers.LSTM will ask for [batch_size,times,num_inp]
Use tf.nn.static_rnn with a list of inputs, instead of 1 input

Source code:

import tensorflow as tf;

x = [[[1],[2],[3]],[[4],[5],[6]]];
times = 2;
bsize = 3;

#3d input
inputs = tf.placeholder(tf.float32, [times,bsize,1]);

cell   = tf.nn.rnn_cell.BasicRNNCell(20);
hids,_ = tf.nn.static_rnn(cell,tf.unstack(inputs,times),dtype=tf.float32);

sess = tf.Session();
init = tf.global_variables_initializer();
sess.run(init);

#results in 2d
print(sess.run(hids, {inputs:x}));


Which techniques can achieve neural doodle in real-time? Ideally I'd like to watch movie which is deep dreamed in real-time. Most algorithms which I know are too slow or not designed for real-time processing.

For example I'm bored with some movie which I've watched thousands of time and I'd like to add some "dreaming" to it which is real-time filter which takes input frames, then it's processing and enhances the images through artificial neural network to achieve doodled output.

Doesn't have to be exactly DeepDream or hallucinogenic technique (which could be too much to watch for 2h), but with any similar ANN algorithm. I'm more interested into achieving desired real-time use.

What kind of techniques can achieve such efficiency?

Most of the algorithms (based on image synthesis and style transfer, e.g. neural-doodle) haven't been proven to be highly effective in terms of real-time image processing.

However the following studies discusses such algorithms for real-time texture synthesis:

Feed-forward Synthesis of Textures and Stylized Images

The approach is to move the computational burden to a learning stage, making trained network (CNN) light-weight and compact in order to generate multiple samples of the same texture. This can generate textures as good as comparable to Gatys~et~al, but significantly faster.
Perceptual Losses for Real-Time Style Transfer and Super-Resolution

This method uses parallel work which can generate high-quality images by defining and optimizing loss functions based on high-level features extracted from pretrained networks.
Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks

This uses precomputed feed-forward networks that captures the feature statistics of Markovian patches in order to generate outputs of arbitrary dimensions. This can be applied to texture synthesis, style transfer and video stylization.

Source: Above list suggested on neural-doodle project.


Who stands a chance against Google in the AI race? At Google I/O the Google Duplex just came out and now releasing new public self-driving car, Waymo. I can't think of any Tech Giants that can actually compete with Google AI other than Amazon. I'm thinking other companies will team up such as Nvidia, Intel, AMD, and etc. Apple seems to be lacking.

Pretty much any "kid in a garage" so to speak. Nobody knows how to build AI yet and a major breakthrough could come from anywhere. That could knock Google off their perch easily enough.


How to Mask an image using Numpy/OpenCV? I am detecting wheels with a deep learning algorithm. The algorithm gives me the coordinates of those rectangles. I want to keep data that is in the rectangles of the image. I created rectangles as a mask of the area I want to keep.

Here is the output of my system

I read my image

im = cv2.imread(filename)

I created the rectangles with:

height,width,depth = im.shape
cv2.rectangle(img,(384,0),(510,128),(0,255,0),3)
cv2.rectangle(rectangle,(width/2,height/2),200,1,thickness=-1)

How can I mask out the data outside of the rectangle from the original image? and keep those rectangles?

Edited: I wrote this code and it only gives me one wheel. How can I have multiple masks and get all the wheels?

  mask = np.zeros(shape=frame.shape, dtype="uint8")

# Draw a bounding box.
# Draw a white, filled rectangle on the mask image
cv.rectangle(img=mask,
             pt1=(left, top), pt2=(right, bottom),
             color=(255, 255, 255),
             thickness=-1)

# Apply the mask and display the result
maskedImg = cv.bitwise_and(src1=frame, src2=mask)

cv.namedWindow(winname="masked image", flags=cv.WINDOW_NORMAL)
cv.imshow(winname="masked image", mat=maskedImg)

You can use cv2.bitwise_and and pass rectangle as a mask.

im = cv2.imread(filename)
height,width,depth = im.shape
cv2.rectangle(img,(384,0),(510,128),(0,255,0),3)
cv2.rectangle(rectangle,(width/2,height/2),200,1,thickness=-1)

masked_data = cv2.bitwise_and(im, im, mask=rectangle)

cv2.imshow("masked_data", masked_data)
cv2.waitKey(0)


How to train a neural network with few weights and biases held constant? I am a beginner in neural networks. I am building a neural network with 3 layers. The input $X$ has 7 features and the output $Y$ is a real number. In the hidden layer, there are two nodes. The bottom node contains weights and biases which should be hard set.

Now, I want to train this neural network with the training data $X$ and $Y$, such that the red weights are held constant while all other weights are learnable.
Is there a way of doing this during the training of the neural network? I'm using TensorFlow and Keras, so, if you could provide also the code necessary to do this, that would be very useful.

The strategy is turning b2 into a separate model, initialise b2 the way it should be, and train your network without b2 as usual.
In the middle of the main network, combine the output of b1 layer and b2 network using concatenation function, for example in TensorFlow:
# Axis 0 is the batch dimension, axis 1 for the dimension of value in every sample
tf.concat([b1out,b2out],axis=1)

Example source code (paste to Google Colab to test):
%tensorflow_version 2.x
%reset -f

import tensorflow                   as     tf
from   tensorflow.keras             import *
from   tensorflow.keras.layers      import *
from   tensorflow.keras.activations import *
from   tensorflow.keras.models      import *
from   tensorflow.keras.callbacks   import *

class MyModel(Model):
    def __init__(self):
        super(MyModel,self).__init__()
        self.dense1 = Dense(200, activation=relu)
        self.dense2 = Dense(1,   activation=tf.identity)

    @tf.function
    def call(self,x):
        # MIND THE CONCATENATION IN THIS PART:
        h1a = self.dense1(x)
        h1b = b2(x) # b2 won't get trained, so it stays fixed

        # CONCATENATION AT AXIS 1, COZ AXIS 0 IS BATCH DIMENSION
        h1  = tf.concat([h1a,h1b],axis=1)
        u   = self.dense2(h1)
        return u

class ModelB2(Model):
    def __init__(self):
        super(ModelB2,self).__init__()
        self.dense1 = Dense(200)
        # self.dense2 = ...
        # Init weights of B here or B is pre-trained model

    @tf.function
    def call(self,x):
        u = self.dense1(x)
        return u

# PROGRAMME ENTRY POINT ========================================================
if __name__=="__main__":
    inp = [[1,2,3,4,5,6,7],[7,6,5,4,3,2,1]] # Example values
    exp = [[0],            [1]            ]

    mm = MyModel()
    b2 = ModelB2()

    mm.compile(loss=tf.losses.MeanSquaredError(), optimizer=tf.optimizers.Adam(1e-3))
    b2.compile(loss=tf.losses.MeanSquaredError(), optimizer=tf.optimizers.Adam(1e-3))
    mm_loss = mm.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) # Init weights in here
    b2_loss = b2.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) # Init weights in here

    print("\nbefore training:")
    print("mm weights:",mm.get_weights()[0][0][:3],"...")
    print("b2 weights:",b2.get_weights()[0][0][:3],"...")
    print("mm loss:",mm_loss)
    print("b2 loss:",b2_loss)

    mm.fit(x=inp,y=exp, batch_size=len(inp), epochs=500, verbose=0)
    mm_loss = mm.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1)
    b2_loss = b2.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1)

    print("\nafter training:")
    print("mm weights:",mm.get_weights()[0][0][:3],"...")
    print("b2 weights:",b2.get_weights()[0][0][:3],"... <-- UNCHANGED AS WANTED")
    print("mm loss:",mm_loss)
    print("b2 loss:",b2_loss,"<-- UNCHANGED AS WANTED")
# EOF

Google colab: https://colab.research.google.com


Does Rice's theorem prove safe AI is undecidable? According to Wikipedia

  In computability theory, Rice's theorem states that all non-trivial,
  semantic properties of programs are undecidable. A semantic property
  is one about the program's behavior (for instance, does the program
  terminate for all inputs), unlike a syntactic property (for instance,
  does the program contain an if-then-else statement). A property is
  non-trivial if it is neither true for every computable function, nor
  false for every computable function.

A syntactic property asks a question about a computer program like "is there is a while loop?"

A semantic properties asks a question about the behavior of the computer program. For example, does the program loop forever (which is the Halting problem, which is undecidable, i.e., in general, there's no algorithm that can tell you if an arbitrarily given program halts or terminates for a given input)?

So, Rice's theorem proves all non-trivial semantic properties are undecidable (including whether or not the program loops forever).

AI is a computer program (or computer programs). These program(s), like all computer programs, can be modeled by a Turing machine (Church-Turing thesis).

Is safety (for Turing machines, including AI) a non-trivial semantic question? If so, is AI safety undecidable? In other words, can we determine whether an AI program (or agent) is safe or not?

I believe that this doesn't require formally defining safety.

Every program halts, or continues

Given N steps, enough time and space (*), halting within N steps is provable

3 (from 2). Halting always has proof: run program until halt; count number of steps; verify claim of halting (within number of steps)

(Program is safe) implies (program is proved safe)

(Safety proved) implies (public understands the proof)

(Program is safe) implies (program always halts (safely) or continues (safely)) [from 1, 6] and ((public) understands (safety proof)) [from 7]

(Public not understand claimed safety proof this moment) implies (don't run program this moment) [common sense]

(*) This universe is finite. Some numbers too big to be computed in this universe
Have you seen perfect software?
Have you seen software make mistakes?
Why trust life-death decisions to software?
Why trust government decisions to software?
Why trust business decisions to software?
If scientists may not recognize AI is intelligent, what if you don't recognize something beyond AI in front of you?
(After enough doubt, all you can do is TRUST)


What is a high dimensional state in reinforcement learning? In the DQN paper, it is written that the state-space is high dimensional. I am a little bit confused about this terminology.
Suppose my state is a high dimensional vector of length $N$, where $N$ is a huge number. Let's say I solve this task using $Q$-learning and I fix the state space to $10$ vectors, each of $N$ dimensions. $Q$-learning can easily work with these settings as we need only a table of dimensions $10$ x number of actions.
Let's say my state space can have an infinite number of vectors each of $N$ dimensions. In these settings, Q-learning would fail as we cannot store Q-values in a table for each of these infinite vectors. On the other hand, DQN would easily work, as neural networks can generalize for other vectors in the state-space.
Let's also say I have a state space of infinite vectors, but each vector is now of length $2$, i.e., small dimensional vectors. Would it make sense to use DQN in these settings? Should this state-space be called high dimensional or low dimensional?

Usually when people write about having a high-dimensional state space, they are referring to the state space actually used by the algorithm.

  Suppose my state is a high dimensional vector of $N$ length where $N$ is a huge number. Let's say I solve this task using $Q$-learning and I fix my state space to $10$ vectors each of $N$ dimensions. $Q$-learning can easily work with these settings as we need only a table of dimensions $10$ x number of actions.

In this case, I'd argue that the "feature vectors" of length $N$ are quite useless. If there are effectively only $10$ unique states (which may each have a very long feature vector of length $N$)... well, it seems like a bad idea to make use of those long feature vectors, just using the states as identity (i.e. a tabular RL algorithm) is much more efficient. If you end up using a tabular approach, I wouldn't call that a high-dimensional space. If you end up using function approximation with the feature vectors instead, that would be a high-dimensional space (for large $N$).

  Let's also say I have a state space of infinite vectors but each vector is now of length $2$ i.e. very small dimensional vectors. Would it make sense to use DQN in these settings ? Should this state-space be called high dimensional or low dimensional ?

This would typically be referred to as having a low-dimensional state space. Note that I'm saying low-dimensional. The dimensionality of your state space / input space is low, because it's $2$ and that's typically considered to be a low value when talking about dimensionality of input spaces. The state space may still have a large size (that's a different word from dimensionality).

As for whether DQN would make sense in such a setting.. maybe. With such low dimensionality, I'd guess that a linear function approximator would often work just as well (and be much less of a pain to train). But yes, you can use DQN with just 2 input nodes.