Is there a rigorous proof for finding Hopfield minima? I am looking for a rigorous mathematical proof for finding the several local minima of the Hopfield networks. I am searching for something rigorous, a demonstration, not just let the network keep updating its neurons and wait for noticing a stable state of the network.

I have look virtually everywhere but I found nothing.

Is there a rigorous proof for Hopfield minima ? Could you give me ideas or resources ?

Thank you in advance

See the paper On the Convergence Properties of the Hopfield Model (1990), by Jehoshua Bruck.

In the first section of the paper, J. Bruck describes the Hopfield network (popularized by J. J. Hopfield in 1982 in his paper Neural networks and physical systems with emergent collective computational abilities, hence the name of the network), then he describes the notation that is used throughout the paper and he gives some examples where a simple Hopfield network (with two nodes) converges to stable states (of the network) and cycles (of which the author also gives a definition).

The usual proofs of the convergence properties of Hopfield networks involve the concept of an energy function, but, in this paper, J. Bruck uses an approach (based on an equivalent formulation of the Hopfield network as an undirected graph) that does not involve an energy function, and he unifies three apparently different convergence properties related to Hopfield networks (described in part $C$ of the section Introduction). More specifically, finding the global maximum of the energy function associated with the Hopfield network operating in a serial mode (which is defined in part $A$ of the Introduction section of the paper) is equivalent to find a minimum cut in the undirected graph associated with this Hopfield network.

Furthermore, note that the proofs of convergence of the Hopfield networks actually depend on the structure of the network (more specifically, its weight matrix). For example, if the weight matrix $W \in \mathbb{R}^{n \times n}$ (where $n$ is the number of nodes in the network) associated with the Hopfield network is a symmetric matrix with the elements of the diagonal being non-negative, then the network will always converge to a stable state.

See also the chapter 13 The Hopfield Model of the book Neural Networks - A Systematic Introduction (1996) by Raul Rojas.


Which nonfictional documentaries about Artificial Intelligence are available? From the subjective perspective, the number of documentaries about the subject Artificial Intelligence and robotics is small. It seems, that the topic is hard to visualize for the audience and in most cases, the assumption is, that the recipient isn't familiar with computers at all. I've found the following documentaries:

The Computer Chronicles - Artificial Intelligence (1985)
The Machine That Changed the World (1991), Episode IV, The Thinking Machine
Robots Rising (1998)
Rodney's Robot Revolution (2008)

The subjective awareness is, that the quality of the films in the 1980s was higher than in modern documentaries, and in 50% of the documentaries Rodney Brooks is the host. Are more documentaries available which can be recommended to watch?

Focus on non-fictional documentaries

Some fictional movies were already mentioned in a different post. For example Colossus: The Forbin Project (1970), Bladerunner (1982) or A.I. Artificial Intelligence (2001). They are based on fictional characters which doesn't exist and the presented robots are running with a Hollywood OS. This question is only about nonfictional motion pictures.

AlphaGo (2017) is quite a good watch, given that it is a documentary about the AlphaGo program, how DeepMind developed it, the help they had, and doesn't get too technical.  You can watch the trailer here.

Another documentary which wasn't exactly AI but something that is an interesting watch or at least was when it came out was The Human Face of Big Data.


Which neuron represents which part of the input? In a neural network, each neuron represents some part of the input. For example, in the case of a MNIST digit, consider the stem of the number 9. Each neuron in the NN represents some part of this digit.

What determines which neuron will represent which part of the digit?
Is it possible that if we pass in the same input multiple times, each neuron can represent different parts of the digit?
How is this related to the back-propagation algorithm and chain rule? Is it the case that, before training the neural network, each neuron doesn't really represent anything of the input, and, as training proceeds, neurons start to represent some part of the input?

The determination of what each neuron represents is dictated by the initial weights of the network. As you may know, the common practice is to initialize the weights randomly. This means that given the same input data, the network may switch what each neuron means. It may also function differently, taking longer or shorter times to train. However, if you set the random seed to some constant, you can make the network deterministic.


What is the typical AI approach for solving blackjack? I'm currently developing a blackjack program. Now, I want to create an AI that essentially uses the mathematics of blackjack to make decisions.

So, what is the typical AI approach for solving blackjack?

It doesn't have to be language-specific, but if it will help with an answer, the language I plan to use to do this would be Python.

Blackjack is usually modelled using Monte Carlo (MC) Methods. There is a lot of literature on MC methods which is interesting on its own right but here is a paper describing how MC is applied to Blackjack. There is also a good description on page 110 of the Introduction to Reinforcement Learning.

Good luck!


How to define the "Pre-Processing" in machine learning? Is every process1 that is done on the data before we train the model is always called the pre-processing part? Or are there some processes which are not included?

1"Every process" is includes data acquisition, splitting the data for validation, data cleaning, feature engineering, etc.

Data preprocessing consists of all those techniques used to generate the final datasets (with an appropriate size, structure, and format) for the machine learning algorithms or models. Data acquisition should not be part of data preprocessing, but the step preceding it, which gathers the raw data (which may e.g. be noisy).

The book Data Preprocessing in Data Mining (2014), by Salvador García et al., which provides a good overview of the data preprocessing techniques and their connection with data mining and machine learning algorithms and models, defines data preprocessing as follows.

  Data preprocessing includes data preparation, compounded by integration,
  cleaning, normalization and transformation of data; and data reduction tasks; such as feature selection, instance selection, discretization, etc. The result expected after a reliable chaining of data preprocessing tasks is a final dataset, which can be considered correct and useful for further data mining algorithms.

From page 10 onwards, there is a description and categorization of the main data preprocessing techniques. I will just list them, so refer to the book for a definition and explanation of each of these techniques.

Data Preparation

Data Cleaning
Data Transformation
Data Integration
Data Normalization
Missing Data Imputation
Noise Identification

Data Reduction

Feature Selection
Instance Selection
Discretization
Feature Extraction/Instance Generation

Here are two screenshots (from the cited book) that illustrate some of the data preparation

and data reduction techniques.


What algorithm would you advise me to use for my task? I have an image and a mask. I want the image to be the same, but rotated, scaled and positioned like mask. What can I use?

Assuming that the image is blank everywhere but where the face is drawn...
The first step is to scale the image to the mask.  That doesn't require a detailed explanation here as it is too trivial a problem.
Second, rotate the image by 90 degrees three times and save each one.
Third, for the four versions of the image (the original and three rotations), do the image addition.  The only one with information will be the one that is rotationally aligned with the mask.


Understanding a paragraph about object detection with two objects I was reading this article on detecting rectangles in an image. My doubt is in the part where the model works fine with detecting a single object, but struggles with two rectangles detection.
The author reasons this as follows:

We train our network on the leftmost image in the plot above. Let’s say that the expected bounding box of the left rectangle is at position 1 in the target vector (x1, y1, w1, h1), and the expected bounding box of the right rectangle is at position 2 in the vector (x2, y2, w2, h2). Apparently, our optimizer will change the parameters of the network so that the first predictor moves to the left, and the second predictor moves to the right. Imagine now that a bit later we come across a similar image, but this time the positions in the target vector are swapped (i.e. left rectangle at position 2, right rectangle at position 1). Now, our optimizer will pull predictor 1 to the right and predictor 2 to the left — exactly the opposite of the previous update step! In effect, the predicted bounding boxes stay in the center.

I don't understand how this reasoning is correct, apart from the fact that when they try to flip the rectangles to mitigate this error, accuracy actually improves (so there's experimental observation, but not much theoretical reasoning).
The reason I think so is that in the case of a single rectangle also the network has to learn for all differently placed objects just as in the two-rectangle-case, so there too, it should predict boxes in the somewhat the center only.
I have to concede I am only a noob in this, so I would love to find out where I am wrong in my reasoning, because experimentally I am wrong (i.e. accuracy does improve when rectangles are flipped).
Thoughts? Also if this is not the correct sub/forum for these type of questions, please feel free to guide me towards those that better suit the content.

If I understood that post right (I just skimmed through, so it's possible I missed some details), they are using several predictors on the inputs with several rectangles. This basically means separate, not shared, weights to detect each rectangle.

It's fairly likely that these neurons will adjust to certain areas on the input image. As far as the neurons are concerned, they are "punished" if they detect the wrong rectangle, as if the other rectangle is the noise that the network should learn to ignore. As a result, the "left" predictor learns to fire more actively when it detects a rectangle on the left, because there's high chance it's the correct one. Same for the "right" predictor.

When, all of a sudden, the training image swaps the targets, each predictor gets confused. Because now what they used to detect as noise becomes the ground truth and vice versa. It certainly hurts the performance and makes the network be sure the most, when the target is around the center. When the targets are flipped, the dataset becomes more consistent and less confusing for the network, it's no wonder that the accuracy goes up.


What is the difference between pixel-based object recognition and feature-based object recognition? From my understanding and text I found in research papers online :

Pixel-based object recognition: neural networks are trained to locate individual objects based directly on pixel data.
Feature-based object recognition: contents of a window are mapped to a feature space that is provided as input to a neural classifier.

What do the above 2 definitions imply in a simpler language? Especially point number 2.

Also, can someone point me to papers/resources/articles that would explain the above approaches in greater detail?

Pixel based object recognition, like the name says, works by analyzing the individual pixels of an image. For example: You analyze an image with a lot of different shades of blue and some grey pixels - you might assume that this is the picture of a plane in the sky or a ship in the water.

You could also look for similar pictures by calculating the difference between each individual pixel of 2 images and sum up the differences. The smaller the sum of the difference, the more the pictures look alike.

It depends on your field of work, but those methods will most likely not be good enough for solid image recognition.

You also mentioned neural networks. Of course you can feed the raw pixel data to a neural network, but that's not state of the art. You will almost always use a CNN, which brings me to your next point.

Feature based object recognition, like the name says, tries to extract certain features from an image to perform classification. You use several convolutional layers for analysis before you feed the output to a neural network. This architecture is called CNN (Convolutional Neural Network).

What those features actually look like is hard to explain without showing examples. I recommend to watch the following two lectures from the Stanford course CS231n:

Lecture 2 | Image Classification
Lecture 5 | Convolutional Neural Networks

The first one explains image classification in general and shows some implementations for pixel based algorithms. The second one dives into CNNs and explains feature based approaches. I highly recommend to watch the full series, but if you are in a hurry, those 2 lectures will tell you what you want to know.


Is there a way of representing the minimax algorithm mathematically? I have successfully figured out how the minimax algorithm works for a game like chess, where a game tree is used, and you assign a value to the terminal nodes and propagate that value up the tree.

Is there a way to represent this algorithm mathematically? If so, how would I go about showing it?

A simple google search gives plenty of results.

If you have a look at the entry in wikipedia for Minimax it has mathematical representations as well as some basic pseudocode and tree representations to help grasp the concept.

Proving it would be a matter of going through the regular methods of a mathematical proof and would probably be a bit complicated.  It is far easier to discuss this by way of graphs.


When is bias values updated in back propagation? I am new to deep learning. I have doubts on modifying bias values during back propagation. My doubts are

Does the back propagation algorithm modifies the weigh values and bias values in the same pass?
How does the algorithm decide whether it has to change the weight value or bias value to reduce the error in a pass?
Will the learning rate same for bias and weights?

Thanks!

Does the back propagation algorithm modifies the weigh values and bias values in the same pass?

Yes.

  How does the algorithm decide whether it has to change the weight value or bias value to reduce the error in a pass?

It differentiates the loss function (like MSE) with respect to the weights and biases, that is, it finds the partial derivative of the loss function with respect to each of the parameters. You can think of the partial derivative of the loss function with respect to one of the parameters of the model as representing the "contribution" of that parameter to the loss of the model.

The partial derivatives of the loss function with respect to each of the parameters (weights and biases) is collectively called the gradient, which is thus a vector of $N$ partial derivatives, where $N$ is the number of parameters of the model.

  Will the learning rate same for bias and weights?

It is usually the same, but, in theory, nobody prevents you from updating the biases using a different learning rate, so you could use a different learning rate to update the biases and weights.


Neural Nets: CNN confirming layer/filter arithmetic I was hoping someone could just confirm some intuition about how convolutions
work in convolutional neural networks. I have seen all of the tutorials on
applying convolutional filters on an image, but most of those tutorials
focus on one channel images, like a 128 x 128 x 1 image. I wanted to clarify
what happens when we apply a convolutional filter to RGB 3 channel images.

Now this is not a unique question, I think a lot of people ask this question as well. It is just that there seem to be so many answers out there, each with their own variations, that it is hard to find a consistent answer. I included a post below that seems to comport with what my own intuition, but I was hoping one of the experts on SE could help validate the layer arithmetic, to make sure my intuition was not off.

Convolutional neural nets and reduction of the layers

Consider an Alexnet network with 5 convolutional layers and 3 fully connected
layers. I borrowed the network from this post.  Now, say the input is 227 x 227, and the filter is specified
as 11 x 11 x 96 with stride 4. That means there are 96 filters each with dimensions 11x11x3, right?
So there are a total of 363 parameters per filter--excluding the bias term--
and there are 96 of these filters to learn. So the 363*96 = 34848 filter values are learned
just like the weights in the fully connected layers right?

My second question deals with the next convolutional network layer. In the next
layer I will have an image that is 55 x 55 x 96 image. In this case, would the
filter be 5x5x96--since there are now 96 feature maps on the image? So that means
that each individual filter would need to learn 5x5x96 = 2400 filter values (weights),
and that across all 256 filters this would mean 614,400 filter values?

I just wanted to make sure that I was understanding exactly what is being learned
at each level.

Your first point is correct. The filters are stored in 4d arrays, with dimensions of (height, width, input channels, filter number) . The order may differ.
Your second point is correct too. The filtered result get stacked together so the output dimensions are (Height,width,filter numbers) the next layer's filters are of size( filter width, filter width, last layer's filter number). Your understanding of CNN is correct. If you want additional resources on CNN, you can try Andrew Ng's class on CNN in Coursera. Hope you can learn more about CNN.


Does Musk know what gradient descent is? From Futurism.com:

  Musk indicates that internet infrastructure is "particularly susceptible" to a method called gradient descent algorithm, a mathematical problem-solving process. Bad news is, AI is excellent at doing gradient descents, which can become devastating digital weaponry.

A screenshot of the tweet is provided below.

He is an intelligent man, so why would he say something like this? Does Musk know what gradient descent is?

I think Musk was using the terminology correctly though perhaps with hyperbole.

I believe this was tweeted in the context of the botnet attacks on name-resolution services that broke Netflix and a large number of other internet services for a time.  He was expressing the idea that you could train a botnet-based system to attack the internet by giving it a toolbox of targets and hacks and using well-known machine-learning techniques to optimize the effectiveness of using the attacks in combination to take down the internet.

Gradient descent itself isn't harmful; it is using gradient descent to train a botnet that could theoretically result in a really devastating attack.

I also don't think he is necessarily implying that there is an actual machine learning system at work; rather that the hackers were training themselves with a methodology analogous to gradient descent.


How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent? In this video, the lecturer states that $R(s)$, $R(s, a)$ and $R(s, a, s')$ are equivalent representations of the reward function. Intuitively, this is the case, according to the same lecturer, because $s$ can be made to represent the state and the action. Furthermore, apparently, the Markov decision process would change depending on whether we use one representation or the other.

I am looking for a formal proof that shows that these representations are equivalent. Moreover, how exactly would the Markov decision process change if we use one representation over the other? Finally, when should we use one representation over the other and why are there three representations? I suppose it is because one representation may be more convenient than another in certain cases: which cases? How do you decide which representation to use?

In general the different reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ are not equivalent mathematically, so you will not find any formal proof.
It is possible for the functions to resolve to the same value in a specific MDP, if, for instance, you use $R(s, a, s')$ and the value returned only depends on $s$, then $R(s, a, s') = R(s)$. This is not true in general, but as the reward functions are often under your control, it can be the case quite often.
For instance, in scenarios where the agent's goal is to reach some pre-defined state, as in the grid world example from the video, then there is no difference between $R(s, a, s')$ or $R(s)$. Given that is the case, for those example problems you may as well use $R(s)$, as it simplifies the expressions that you need to calculate for algorithms like Q-learning.
I think the lecturer did not mean "equivalent" in the mathematical sense, but in the sense that future lectures will use one of the functions, and a lot of what you will learn is going to be much the same as if you had used a different reward function.

Finally, when should we use one representation over the other and why are there three representations?

Typically, I don't use any of those representations by default. I tend to use Sutton & Barto's $p(s', r|s, a)$ notation for combined state transitions and rewards. That expression returns probability of transitioning to state $s'$ and receiving reward $r$ when starting in state $s$ and taking action $a$. For discrete actions, you can re-write the expectation of the different functions $R$ in terms of this function as follows:
$$\mathbb{E}[R(s)] = \sum_{a \in \mathcal{A}(s)}\sum_{s' \in \mathcal{S}}\sum_{r \in {R}}rp(s', r|s, a)\qquad*$$
$$\mathbb{E}[R(s,a)] = \sum_{s' \in \mathcal{S}}\sum_{r \in {R}}rp(s', r|s, a)$$
$$\mathbb{E}[R(s,a, s')] = \sum_{r \in {R}}r\frac{p(s', r|s, a)}{p(s'|s, a)}$$
I think this is one way to see how the functions in the video are closely related.
Which one would you use? Depends on what you are doing. If you want to simplify an equation or code, then use the simplest version of the reward function that fits with the reward scheme you set up for the goals of the problem. For instance, if there is one goal state to exit a maze, and an episode ends as soon as this happens, then you don't care how you got to that state or what the previous state was, and can use $R(s)$
In practice, what happens if you use a different reward function is that you need to pay attention to where it appears in things like the Bellman equation for theoretical treatments. When you get to implement model-free methods like Q-learning, $R(s)$ or its variants don't really appear except in the theory.

* This is not technically correct in all cases. The assumption I have made is that $R(s)$ is a reward granted at the point of leaving state $s$, and is independent of how the state is left and where the agent ends up next.
If this was a fixed reward for entering state $s$, regardless of how, then it could be written around $R(s')$ as follows:
$$\mathbb{E}[R(s')] = \sum_{s \in \mathcal{S}}\sum_{a \in \mathcal{A}(s)}\sum_{r \in {R}}rp(s', r|s, a)$$
i.e. by summing all the rewards that end up at $s'$


Several questions related to UCT and MCTS In Bandit Based Monte-Carlo Planning, the article where UCT is introduced as a planning algorithm, there is an algorithm description in page 285 (4 of the pdf).
Comparing this implementation of UCT (a specific type of MCTS algorithm) to the application normally used in games, there is one major difference. Here, the rewards are calculated for every state, instead of only doing an evaluation at the end of the simulation.
My questions are (they are all related to each other):

Is this the only big difference? Or in other words, can I do the same implementation as in MCTS for games with the 4 stages: selection, expansion, simulation and backpropagation, where the result of the simulation is the accumulated reward instead of a value between 0 and 1? How would the UCT selection be adjusted in this case?

What does the UpdateValue function in line 12 does exactly? In the text it says it is used to adjust the state-action pair value at a given depth, will this be used to the selection? How is this calculated exactly?

What is the depth parameter needed for? Is it related with the UpdateValue?

Finally I would like to ask if you know any other papers where a clear implementation of UCT for planning is used with multiple rewards, not only on the end of the simulation.

Is this the only big difference? Or in other words, can I do the same implementation as in a simple MCTS with the 4 stages: selection, expansion, simulation and backpropagation, where the result of the simulation is the accumulated reward instead of a value between 0 and 1? How would the UCT selection be adjusted in this case?

No, this is not the only difference. The important differences between the "original UCT" (as described in the paper you linked), and the "standard UCT" (as typically implemented in game AI research) are nicely summarised in Subsection 2.4 of "On Monte Carlo Tree Search and Reinforcement Learning". I'd recommend reading the entire paper, or large parts of it, if you have the time, but to summarise the key differences:

Original stores as many nodes as allowed by memory limitations per iteration, standard only expands one node per iteration.
Original uses transpositions, standard implementation does not
Original uses discount factor $\gamma$, standard does not (equivalent to picking $\gamma = 1$)
Original takes into account at which point in time rewards were observed, standard does not

That last point is the one your question mainly seems to be about. Your idea of using a standard implementation, but accumulating any intermediate rewards and treating them as one large terminal reward, would not be the same. Actually taking into account the times at which rewards were observed, as done in the original UCT, can lead to more accurate value estimates in nodes in the middle of the tree (or at least to finding them more quickly). Your idea would be equivalent to learning from Monte-Carlo backups in Reinforcement Learning literature (e.g. Sutton and Barto's book), whereas the original UCT implementation would be more similar to learning from TD($0$) backups.

Note that, when talking about the "standard" implementation of UCT, that's very often in papers about two-player zero-sum games in which all intermediate rewards have a value of $0$, discounting is considered to be unimportant (i.e. $\gamma = 1$), and only terminal game states have a non-zero reward. In this special case, the two ideas do become equivalent.

  What does the UpdateValue function in line 12 does exactly? In the text it says it is used to adjust the state-action pair value at a given depth, will this be used to the selection? How is this calculated exactly?

This would be done in the "standard" way. The visit count is incremented, and $q$ could be added to a sum of all $q$ values ever observed during the search for that state-action pair, such that the average score can be computed as the sum of $q$ scores divided by visit count. This average score is the $\bar{X}$ used in Selection.

  What is the depth parameter needed for? Is it related with the UpdateValue?

I think it's usually not really needed. In the paper they describe that it may be considered to have a depth cut-off. From paper:

  "This can be the reach of a terminal state, or episodes can be cut at a certain depth (line 8)."


How should I model all available actions of a chess game in deep Q-learning? I just read about deep Q-learning, which is using a neural network for the value function instead of a table.
I saw the example here: Using Keras and Deep Q-Network to Play FlappyBird and he used a CNN to get the Q-value.
My confusion is on the last layer of his neural net. Neurons in the output layer each represent an action (flap, or not flap). I also see the other projects where the output layer also represents all available actions (move-left, stop, etc.)
How would you represent all the available actions of a chess game? Every pawn has a unique and available movement. We also need to choose how far it will move (rook can move more than one square). I've read Giraffe chess engine's paper and can't find how he represents the output layer (I'll read once again).
I hope somebody here can give a nice explanation about how to design NN architecture in Q-learning, I'm new in reinforcement learning.

To model chess as a Markov decision problem (MDP) you can refer to the AlphaZero paper (Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm). The exact details can be found starting from the bottom of page 13.

Briefly, an action is described by picking a piece and then picking a move with it. The size of the board is 8 by 8 so there can be 8x8 possibilities for picking a piece. Then we can either pick linear movements (in 8 directions) and then pick the number of steps in that direction (maximum 7 steps) or we can make a knight movement (maximum 8 possibilities). So far that is 8x7 + 8. Furthermore, we also need to consider underpromotions (promoting a pawn into a non-queen piece). In this scenario we can have 3 types of pawn movements (forward, left diagonal or right diagonal capture) and 3 types of promotions (rook, knight, bishop) so that makes it 9. So the total dimension of the action space is 8x8x(8x7+8+9) and this will be the number of neuron outputs you will need to use.

Note that this action space representation covers every possible scenario and for example at the start of the game the action of picking the tile E4 and promoting it to a bishop doesn't make sense (there are no pieces on tile E4 at the beginning of the game). Or if we pick a tile where there is a rook we cannot make a knight movement with it. Therefore you will also need to implement a function that can return you the set of possible actions in a given state and ignore all neural network outputs that is not contained in this set.

Obviously this action representation is not set into stone so if you can come up with something better or more compact you can use that one too. You can also make restrictions to your game for example by not allowing underpromotions.


What's the intuition behind contrastive learning? Recently, I have seen a surge of papers w.r.t contrastive learning (a subset of semi-supervised learning).

Can anyone give a detailed explanation of this approach with its advantages/disadvantages and what are the cases in which it gives better results?

Also, why it's gaining traction amongst the ML research community?

Contrastive learning is a framework that learns similar/dissimilar representations from data that are organized into similar/dissimilar pairs. This can be formulated as a dictionary look-up problem.
If I conceptually compare the loss mechanisms for:

SimCLR: A Simple Framework for Contrastive Learning of Visual Representations @ https://arxiv.org/abs/2002.05709
MoCo: Momentum Contrast for Unsupervised Visual Representation Learning @ https://arxiv.org/abs/1911.05722

Both MoCo and SimCLR use varients of a contrastive loss function, like InfoNCE from the paper Representation Learning with Contrastive Predictive Coding
\begin{eqnarray*}
\mathcal{L}_{q,k^+,\{k^-\}}=-log\frac{exp(q\cdot k^+/\tau)}{exp(q\cdot k^+/\tau)+\sum\limits_{k^-}exp(q\cdot k^-/\tau)}
\end{eqnarray*}
Here q is a query representation, $k^+$ is a representation of the positive (similar) key sample, and ${k^−}$ are representations of the negative (dissimilar) key samples. $\tau$ is a temperature hyper-parameter. In the instance discrimination pretext task (used by MoCo and SimCLR), a query and a key form a positive pair if they are data-augmented versions of the same image, and otherwise form a negative pair.

The contrastive loss can be minimized by various mechanisms that differ in how the keys are maintained.
In an end-to-end mechanism (Fig. 1a), the negative keys are from the same batch and updated end-to-end by back-propagation. SimCLR, is based on this mechanism and requires a large batch to provide a large set of negatives.
In the MoCo mechanism i.e. Momentum Contrast (Fig. 1b), the negative keys are maintained in a queue, and only the queries and positive keys are encoded in each training batch.


Why can't OCR be perceived as a good example of AI? On the Wikipedia page about AI, we can read:

  Optical character recognition is no longer perceived as an exemplar of "artificial intelligence" having become a routine technology.

On the other hand, the MNIST database of handwritten digits is specially designed for training and testing neural networks and their error rates (see: Classifiers).

So, why does the above quote state that OCR is no longer exemple of AI?

Whenever a problem becomes solvable by a computer, people start arguing that it does not require intelligence. John McCarthy is often quoted: "As soon as it works, no one calls it AI anymore" (Referenced in CACM).

One of my teachers in college said that in the 1950's, a professor was asked what he thought was intelligent for a machine. The professor reputedly answered that if a vending machine gave him the right change, that would be intelligent.

Later, playing chess was considered intelligent. However, computers can now defeat grandmasters at chess, and people are no longer saying that it is a form of intelligence.

Now we have OCR. It's already stated in another answer that our methods do not have the recognition facilities of a 5 year old. As soon as this is achieved, people will say "meh, that's not intelligence, a 5 year old can do that!"

A psychological bias, a need to state that we are somehow superior to machines, is at the basis of this.


Equilateral and One-of-n encoding I was reading AI For Humans Vol. 1 by Jeff Heaton when I came across the terms "equilateral encoding" and "one-of-n encoding." The explanations unfortunately made no sense to me and the reddit threads on the Web are blocked by my Internet provider (I use a high-school machine). Is anyone here able to provide basic explanations regarding the two procedures for me? Thanks in advance.

Please check this book with the normalization infos:

https://jamesmccaffrey.wordpress.com/2014/06/03/neural-networks-using-c-succinctly/

Freely available and gives good explanation to encoding with code example.

Check the part: "Effects Encoding and Dummy Encoding"


What types of AI agents are Djikstra's algorithm and Prim's Minimum Spanning Tree algorithm? From the perspective of the type of AI Agents, I would like to discuss Prim's Minimum Spanning Tree algorithm and Dijkstra's Algorithm.
Both are model-based agents and both are "greedy algorithms".
Both have their memory to store the history of vertices and their path distance. Prim's is more greedy than Dijkstra's algorithm whereas Dijkstra's algorithm is more efficient than Prim's.
Can we say that Dijkstra's algorithm is a utility-based agent, whereas Prim's is a goal-based agent, with the justification that Prim's is more goal-oriented as compared to finding the optimum (shortest) path?

Why do you want to think of these algorithms as agents?

An agent is an abstract and higher-level concept than the concept of an algorithm, which is just a set of instructions.

You could have two agents, one that is supposed to find the minimum spanning tree and another that is supposed to find the shortest path between a source and goal nodes.  In both cases, these two agents have a goal, so they are both goal-oriented agents, but the algorithms they use to reach the goal are irrelevant, as long as they have one or more goals. If you have an agent that performs optimally with respect to some metric, you may want to call that agent an optimal agent.

A similar logic applies to utility-based agents.

So, in short, algorithms are used by agents to act, but the algorithms themselves are not the agents (unless you want to have a philosophical debate or use different definitions of agents). See this answer for more details about the definition of agents.


Proof of uniqueness of value function for MDPs with undiscounted rewards How does one prove the uniqueness of the value function obtained from value iteration in the case of bounded and undiscounted rewards? I know that this can be proven for the discounted case pretty easily using the Banach fixed point theorem.

In the case where the reward is undiscounted, there is no guarantee of convergence as the iteration procedure is not a strict contraction.

Unfortunately I can't find the math mode on the ai stackexchange so my answer can't be very precise.

But an easy example is the following: Take a 'running' reward R of 0 to make things simpler, and a MDP with two states a and b. Take a transition matrix with 0's on the diagonal and 1 off diagonal. You will see that the algorithm will always flip the values of V(a) and V(b), and hence no convergence.


How could emotional intelligence be implemented? I've seen emotional intelligence defined as the capacity to be aware of, control, and express one's emotions, and to handle interpersonal relationships judiciously and empathetically.

What are some strategies for artificial intelligence to begin to tackle this problem and develop emotional intelligence for computers?
Are there examples where this is already happening to a degree today?
Wouldn't a computer that passes a Turing test necessarily express emotional intelligence or it would be seen as an obvious computer?

Perhaps that is why early programs that pass the test represented young people, who presumably have lower emotional intelligence.

Architectures for recognizing and generating emotion are typically somewhat complex and don't generally have short descriptions, so it's probably better to reference the literature rather than give a misleading soundbite:
Some of the early work in affective computing was done by Rosalind W. Picard. There is a research group at MIT specializing in this area.
Some of the more developed architectural ideas are due to Marvin Minsky.
A pre-publication draft of his book, The Emotion Machine, is available via Wikipedia.
Emotional intelligence would certainly seem to be a necessary component of passing the Turing test - indeed, in the original Turing test essay in Computing Machinery and Intelligence implied some degree of "Theory of Mind" about Mr. Pickwick's preferences:

Yet Christmas is a Winter’s day, and I do not think Mr. Pickwick would mind the comparison.


max pooling size=2,stride=1 outputs same size While working with darkflow, I encountered something that I can't understand.

I understand that maxpooling with size=2,stride=2 would decrease the output size to half of its size.

However, if the max-pooling is size=2,stride=1 then it would simply decrease the width and height of the output by 1 only.

However, the darkflow model doesn't seem to decrease the output by 1.

Here is the model structure when I load the example model tiny-yolo-voc.cfg.

Source | Train? | Layer description                | Output size
-------+--------+----------------------------------+---------------
       |        | input                            | (?, 416, 416, 3)
 Init  |  Yep!  | conv 3x3p1_1  +bnorm  leaky      | (?, 416, 416, 16)
 Load  |  Yep!  | maxp 2x2p0_2                     | (?, 208, 208, 16)
 Init  |  Yep!  | conv 3x3p1_1  +bnorm  leaky      | (?, 208, 208, 32)
 Load  |  Yep!  | maxp 2x2p0_2                     | (?, 104, 104, 32)
 Init  |  Yep!  | conv 3x3p1_1  +bnorm  leaky      | (?, 104, 104, 64)
 Load  |  Yep!  | maxp 2x2p0_2                     | (?, 52, 52, 64)
 Init  |  Yep!  | conv 3x3p1_1  +bnorm  leaky      | (?, 52, 52, 128)
 Load  |  Yep!  | maxp 2x2p0_2                     | (?, 26, 26, 128)
 Init  |  Yep!  | conv 3x3p1_1  +bnorm  leaky      | (?, 26, 26, 256)
 Load  |  Yep!  | maxp 2x2p0_2                     | (?, 13, 13, 256)
 Init  |  Yep!  | conv 3x3p1_1  +bnorm  leaky      | (?, 13, 13, 512)
 **Load  |  Yep!  | maxp 2x2p0_1                     | (?, 13, 13, 512)**
 Init  |  Yep!  | conv 3x3p1_1  +bnorm  leaky      | (?, 13, 13, 1024)
 Init  |  Yep!  | conv 3x3p1_1  +bnorm  leaky      | (?, 13, 13, 1024)
 Init  |  Yep!  | conv 1x1p0_1    linear           | (?, 13, 13, 125)
-------+--------+----------------------------------+---------------

The bold text part is causing the confusion. My expectation what (?,12,12,512) but it is not. It retains the same size (13,13)

The corresponding model info from the .cfg file is:

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=1

Why is the output height/width not decreasing by 1?

I wondered about the same input/output size problem you described.

For the stride = 1 / size = 2 maxpool layers the padding option is set to zero by default, therefore one can wonder why the output is still the same size.

I checked all indices and thats what I observed:

The implemention of the forward_maxpool_layer-function "adds" a column and a row to the matrix, increasing its width and height by one.

Lets say we have a 3x3 matrix as follows:

    1    2    3
    4    5    6
    7    8    9

To actually get a 3x3 matrix as an output, something like this is done:

    1    2    3    -FM
    4    5    6    -FM
    7    8    9    -FM
    -FM  -FM  -FM  -FM

    where FM = FLOAT_MAX

They do not literally add a row and a column. If the index is to high, you simply do nothing.

Hope this helped to clarify stuff. Furthermore I am not a native speaker, so sorry for any mistake!


Are feature maps merged or are they passed on as they are? I am unsure about the following parts of the architecture and mechanics of convolution layers in CNNs. Possibly, this is implementation-dependent though.

First question:

Say I have 2 convolution layers with 10 filters each and the dimension of my input tensors is $n \times m \times 1$  (so, grayscale images for example). Passing this input to the first convolution layer results in 10 feature maps (10 matrices of $n \times m$, if we use padding), each produced by a different filter.

Now, what does actually happen when this is passed to the second convolution layer? Are all 10 feature maps passed as one big $m \times n \times 10$ tensor or are the overlapping cells of the 10 feature maps averaged and a $m \times n \times 1$ tensor is passed to the next convolution layer? The former would result in an explosion of feature maps with increasing number of convolution layers and the spacial complexity would be in $\mathcal{O}\left((nm)^k\right)$, where $k$ is the number of chained convolution layers. Averaging the feature maps before passing them to the next layer would keep the complexity linear. So, which is it? Or are both possibilities commonly used?

Second question (with two sub questions):

a) This is a similar question. If I have an input volume of $n \times m \times 3$ (e.g. RGB images) and I have again 2 convolution layers with 10 filters, does each convolution layer have in actuality 30 filters? So 10 sets of 3 filters, one for each channel? Or do I have in fact only 10 filters and the filters are applied to all 3 channels?

b) This is the same question as question (1) but for channels: Once I have convolved a filter (consisting of three channel filters? (a)) over the input tensor I end up with 3 feature maps. One for each channel. What do I do with these? Do I average them component-wise with each other? Or do I keep them separate until I have convolved all 10 filters across the input and THEN average the 10 feature maps of each channel? Or do I average all 30 feature maps of all three channels? Or do I just pass on 30 feature maps to the next convoloution layers which in turn knows which of these feature maps belong to which channel?

Quite a few possibilities... None of the sources I consulted makes this explicit. Maybe because it depends on the individual implementation.

Anyway, would be great if somebody could clear this confusion up a little!

tl;dr
It helps to think that the channels dimension of a convolutional layer works like a fully connected layer (i.e. the layer computes the weighted sum over all channels).
For a single pixel...
Let's consider a single pixel (e.g. the top left pixel). This pixel has $C$ different values, where $C$ are the number of channels. In order to produce the result of a single filter, the layer takes the weighted sum of these $C$ pixels. It does this by actually having $C$ weights, multiplying the pixel values with their corresponding weights and summing them together.
Example
Let's say you have an $n \times m \times 10$ tensor as an input to a convolutional layer with $1$ filter and a $3 \times 3$ kernel. For creating its output the layer has $3 \times 3 \times 10 = 90$ weights, i.e. a different $3 \times 3$ kernel for each of the $10$ input channels. To create its output, the layer performs the convolution operation separately on each of the input channels (each with its corresponding weight matrix) and creates this way $10$ feature maps which are summed together.
Now imagine the layer has $20$ filters instead of $1$. Nothing changes, just that the same procedure is done $20$ times with $20$ different sets of weights So the total number of weights in the layer in this case is $3 \times 3 \times 10 \times 20 = 1800$.
To answer your questions...
(1) You have a grayscale image $n \times m \times 1$ and it passes through the first convolution layer (which has $10$ filters). This layer will perform the convolution operation on the input image $10$ times independently and produce $10$ feature maps, i.e. an output tensor of $n \times m \times 10$. Now this in fed into the second convolution layer, which again has $10$ filters. For each filter the layer will perform the convolution operation on the $10$ input feature maps independently and will sum the corresponding pixels together to form a $n \times m \times 1$ feature map. It will perform this procedure $10$ times and generate an output tensor of $n \times m \times 10$.
(2) Yes, each layer actually has $30$ filters, a total of $10$ for each of the R, G, B channels. The layer just sums the results of the R, G, B channels to produce a single feature map. For the second part, if I got it right it's pretty much what you said but it sums the maps instead of averaging them
Notes
I'd recommend checking Stanford's CS231 notes on convolution layers, which explain it in much detail and also have numerical examples to confirm if you've got it right.
You could also check this answer for more details.


Late Onset Augmentation If I train a U-Net model for image segmentation (e.g. medical images) and start training until it converges and then add augmentation - can i expect similar results as if i train with augmentation from the beginning ?

It would be better to run experiments on this. But, here's answer analytically.
The models will be different.
With augmentation, the network starts to learn to combat the noise too.
With late onset augmentation, the network will start to deviate from its original solution to combat noise.
Comparing this to the ball rolling down the hill, in case of augmentation, the ball will roll but will face a bit of friction (the noise due to augmentation). On the other hand, with late onset augmentation, it would be like the ball reached the bottom of hill, and now it has to face an uphill battle to model the noise. The ball will need to go a bit up the hill against gravity and friction due to noise.
I don't think it would be a good thing to do. But, running experiments is the only way we can confirm.


Is there a technical name for image classification on amount instead of class? Is there a technical term for an image classifier that classifies on a single class but is classifying on an amount like how full a glass of water is rather than different classes?

Training a system to predict real numbers instead of classes is called regression. If you had a training data where the inputs were images of water glasses and the targets were the amount of water in each glass, then this would be regression task from images of water glasses to the amount of water in each glass. You could call it image regression in the same way you would say image classification, but beyond that, I don't think there's a special term for regression where the inputs are images.


After a model has been trained, how do I use it to address the real-world problems? I understand the way we build and train a model, but all of the online courses I've found end with this. I can't find any course explaining the process of utilizing the trained model to address the problem.
Is there any course out there that explains the whole process from data collection, model building, and utilizing the model to solve the real-world problems?

Using a machine learning or AI-powered model once it has been built and tested, is not directly an AI issue, it is just a development issue. As such, you won't find many machine learning tutorials that focus on this part of the work. But they do exist.

In essence it is the same as integrating any other function, which might be in a third-party library:

Package the new function so that it can be called from your production system (this may be the hardest part)
Decide where in the code of your production system to call the new function. In your example case maybe after form is completed describing an incident you could link the top recommended KB articles from the new ticket.
Design and check the associated user experience/UI as appropriate for your development team (in small projects you may skip this step and just implement)
Change your production code to call the packaged function. In any professional development team, this part will have multiple stages, but not really relevant to the question - if you are the ML specialist and delivering your new model to an existing team, you need to talk to them about both the packaging part and the steps involved here.

Often, machine learning models come with a whole bunch of dependencies that the rest of a system does not have. There are a variety of solutions for that, depending on the libraries that you are using, whether you are using a cloud PaaS service etc. You could just build a Docker image to hold all the AI parts and call it passing the input data.

Deploying ML models to production as a job is often a task for Machine Learning Engineer roles. There are courses and articles covering the practical aspects of these steps, if you search for them. Here are a couple:

Deploying Machine Learning Models from Coursera
A blog article on Medium: There are two very different ways to deploy ML models . . .

I have no affiliation with any of the above, have not read articles or taken the courses, and am not able to make any recommendation, even if you told me the technologies you were using for ML and in production currently.

After deploying to production . . .

Your work is not necessarily done. You will want to monitor behaviour of the system. During integration, you should of added logging, or some way to get feedback of performance in the wild.

For instance, if this is an ML system, does the accuracy seen in testing hold up against real life? Is the target population drifting? If the system makes interventions - by e.g. suggesting links, or showing categories to end users - how well are those working in practice? Is the performance and responsiveness of the system fast enough when the service is under load?

If it is more reactive AI system, that takes actions itself, you will similarly want to monitor what it is supposed to optimise, or sample its output for errors and quality control.

All this feedback can go back into a new iteration of design and integration. How to incorporate that will depend a lot on the nature of the system and what you discover, so could be the subject of further questions.


How is exponential moving average computed in deep Q networks? In normal Q-learning, the update rule is an implementation of the exponential moving average, which then converges to the optimal true Q values. However, looking at DQN, how exactly is the exponential moving average implemented in deep networks?

However, looking at DQN, how exactly is the exponential moving average implemented in deep networks?

It is not implemented directly as exponential moving average.
Instead, the ability of neural networks to learn online and incrementally forget older input/output associations is used to achieve the same goal.
If you use the simplest mini-batch stochastic gradient descent methods - i.e. just a simple gradient step $\mathbf{w} \leftarrow \mathbf{w} - \alpha \nabla_{\mathbf{w}}\sum_i(g_i - \hat{q}_i)^2$ where $g_i$ is measured (or bootstrap estimated) discounted return for a single state/action pair and $\hat{q}_i$ is the current estimate, then the learning rate $\alpha$ is analagous to the same factor in exponential moving average approach, and in fact would be the same thing mathematically if you one-hot-encoded the states and only had a single layer in the neural network.
Typical implementations of DQN will have deeper networks, will not one-hot-encode the entire state space, and will typically use some gradient modifier such as momentum or Adam to improve performance. So the match to exponential moving average is not exact. But the behaviour is similar in the most important aspect for RL - the ability to learn online and forget older values as the target distribution of expected returns changes due to changes in policy.


In style transfer, why does the comparison between channels give a good sense of style? I have been learning about Style Transfer recently. Style is defined as

The correlation of activations between channels.

I can't seem to understand why that would be true. Intuitively, style seems to be the patterns that exist in one particular channel/image rather than the patterns between channels. When filters in CNNs have different weights for acting as filters for different channels, why do we even expect 2 channels to be correlated? And further, why do we expect the style to be conveyed by it?
I expected a style function that could compare activations in some layer of a CNN condensed into one channel so that an algorithm can search for which activations occur simultaneously and hold style information.
I understand how we are carrying out the operations with the matrix and defining the loss function, what I don't get is why we are assuming style information lies in correlation between channels in the first place.

When the original Neural Transfer paper was published, your question stayed unanswered for a while.  The reason why the Gram matrix represents artistic style was not entirely clear. A satisfactory (in my opinion) explanation came with the "Demystifying Neural Style Transfer" paper.
The basic idea is that you cannot just directly compare activations for two images. The spatial positions of various features are different for the source and target image, so you should somehow get rid of positional information of the activations and compare their distributions across the whole image. The goal of the style transfer task is thus to make the two distributions as close as possible. One of the possible measures of distance between two distributions $P$ and $Q$ is the Maximum Mean Distance (MMD):
$$\text{MMD}^2(P,Q) = \left\Vert \mathbb{E}_{P}[\phi(X)] - \mathbb{E}_{Q}[\phi(Y)]\right\Vert^2$$
With $\phi(\cdot)$ being a feature function - in our case it would be the NN activations in a particular layer. The next step would be to apply a "kernel trick" to the MMD, representing it through a kernel $k(x,y)$
$$\text{MMD}^2(P,Q) = \mathbb{E}[k(x_i,x_j)] + \mathbb{E}[k(y_i,y_j)]  - 2 \mathbb{E}[k(x_i,y_j)]$$
The Gram matrix of the original style transfer corresponds to the squared dot-product kernel $k(x,y) = (x^Ty)^2$:
$$\text{MMD}^2(P,Q) =  \mathbb{E}[(x^T_ix_j)^2] + \mathbb{E}[(y^T_iy_j)^2]  - 2 \mathbb{E}[(x^T_iy_j)^2]  =  \left\Vert G^x - G^y\right\Vert_F^2 $$
Where $G^x$ and $G^y$ are the Gram matrices for activations and $\Vert\cdot\Vert_F$ is the Frobenius norm. The last equality gives a bit of trouble to those not used to the kernel trick, so I'll expand it (assuming Einstein summation convention):
$$\begin{array}{l}
 \mathbb{E}[(x^T_ix_j)^2] + \mathbb{E}[(y^T_iy_j)^2]  - 2 \mathbb{E}[(x^T_iy_j)^2]  =  \\
 = x_{ik}x_{jk}x_{im}x_{jm} + y_{ik}y_{jk}y_{im}y_{jm} - 2  x_{ik}x_{jk}y_{im}y_{jm} \\
 = (x_{ik}x_{im} - y_{ik}y_{im})(x_{jk}x_{jm} - y_{jk}y_{jm}) \\
 = (G^x_{km} - G^y_{km})(G^x_{km} - G^y_{km}) \\
 = \left\Vert G^x - G^y\right\Vert_F^2
\end{array}
$$
The authors of the 1701.01036 also tried different kernel functions $k(x,y)$ getting more interesting style transfer results.


Each training run for DDQN agent takes 2 days, and still ends up with -13 avg score, but OpenAi baseline DQN needs only an hour to converge to +18? Status:

For a few weeks now, I have been working on a Double DQN agent for the PongDeterministic-v4 environment, which you can find here.

A single training run lasts for about 7-8 million timesteps (about 7000 episodes) and takes me about 2 days, on Google Collab (K80 Tesla GPU and 13 GB RAM). At first, I thought this was normal because I saw a lot of posts talking about how DQNs take a long time to train for Atari games.

Revelation:

But then after cloning the OpenAI baselines repo, I tried running python -m baselines.run --alg=deepq --env=PongNoFrameskip-v4  and this took about 500 episodes and an hour or 2 to converge to a nice score of +18, without breaking a sweat. Now I'm convinced that I'm doing something terribly wrong but I don't know what exactly.

Investigation:

After going through the DQN baseline code by OpenAI, I was able to note a few differences:

I use the PongDeterministic-v4 environment but they use the PongNoFrameskip-v4 environment
I thought a larger replay buffer size was important, so I struggled (with the memory optimization) to ensure it was set to 70000 but they set it to a mere 10000, and still got amazing results.
I am using a normal Double DQN, but they seem to be using a Dueling Double DQN.

Results/Conclusion

I have my doubts about such a huge increase in performance with just these few changes. So I know there is probably something wrong with my existing implementation. Can someone point me in the right direction?

Any sort of help will be appreciated. Thanks!

Dueling architectures create bigger differences in the values of actions in the state space. This is because the state-value V(s) function is estimated separately from the state-action value Q(s, a). A new quantity, the advantage of an action, can then be defined as A(s, a) = Q(s, a) - V(s).

  The Q function, however, measures the the value
  of choosing a particular action when in this state. The advantage
  function subtracts the value of the state from the Q
  function to obtain a relative measure of the importance of
  each action.

  Dueling Network Architectures for Deep Reinforcement Learning

To better direct you, here are 2 resources that could really help you understand why those differences are important.

Speeding up DQN on PyTorch: how to solve Pong in 30 minutes

The main points of the blog are:

Use a larger batch size and play several steps before updating
Play and train in a separate process
Use asyncronous cuda transfers

RL Adventure

This github library has easy to follow jupyter notebooks and links to all of the papers.
It includes:

DQN
Double DQN
Dueling DQN
Prioritized Experience Replay
Noise Networks for Exploration
Distributional RL
Rainbow (That network that deepmind made that had so many things in it they couldn't find a good name)
Distributional RL with Quantile Regression
Hierarchical Deep RL


Is a good evaluation function as good as any of the extensions of alpha-beta pruning? I would like to know if having a really good evaluation function is as good as using any of the extensions of alpha-beta pruning, such as killer moves or quiescence search?

To build on Neil's answer a bit, you're right that the better your evaluation function gets, the less work your optimization function will need to perform. If your evaluation function gets good enough, you won't need to search at all.

This is not just an academic idea though! It's actually fairly widely used, and has been key to solving several games.

The first example I'm aware of is Tesauro's TD-Gammon player, from 1995. Tesauro used the ideas of reinforcement learning and self-play to train a Neural Network to act as an evaluation function. TD-Gammon played with just a 2-move lookahead using the best evaluation function that was found, and was deemed better than most (all?) human expert players at the time.

More recently, AlphaGo Zero used similar techniques to solve Go, but learning both an evaluation function and (separately) a function to randomize over possible moves.


Reduce the effect of excessive zeros I am working on an autoregression problem where I use sequential LSTM. My target is well defined, but I think I am facing a problem with the features. As the features were non-stationary, then I decided to apply the log-returns to each of them. In other words, if $F_t$ is a feature at a certain time $t$, then I apply $$\log(\frac{F_t}{F_{t-1}}).$$

$\Rightarrow$ Why non-stationary data is hard to analyse?

The property makes it easier to analyse, but produce a lot of zeros when $F_t = F_{t-1}$.

As there is lots of zeros, then the predictions tend to be closer to 0. How can I reduce that effect mathematically?

That the features are not stationary is a reason why LSTM is a good approach.  Calculating the ratio of adjacent values in the sequence is unnecessary.  Since the logarithm would remove a skew in the input of numbers close to zero, the acquisition is in question.  Why are the values not often varying?  Are their transient events followed by inactivity?

Aggregating unchanged items in the sequence and adding duration as a feature may add entropy to the input and improve LSTM performance.  This is called decimation, and can be useful in adding entropy (removing redundancy) in data that has regions of redundant oversampling.

If the values without the division are still distributed exponentially, then keep the logarithm but maybe make it a natural log.


What is wrong with the idea that the AI will be capable of omniscience? In the context of artificial intelligence, the singularity refers to the advent of an artificial general intelligence capable of recursive self-improvement, leading to the rapid emergence of artificial superintelligence (ASI), the limits of which are unknown, shortly after technological singularity is achieved. Therefore, these superintelligences would be able to solve problems that we possibly are unable to solve.

According to a poll reported in Future progress in artificial intelligence: A survey of expert opinion (2014)

  The median estimate of respondents was for a one in two chance that highlevel machine intelligence will be developed around 2040-2050

which isn't very far away.

What is wrong with the idea that the AI will be capable of omniscience, given that it could benefit us by solving many problems?

I quite like your outlook, and without getting into the details of how a "singularity" may be effected which is covered in numerous other questions, or how consciousness and "omniscience" come into play because consciousness and omniscience are not requirements, I will instead direct you to two key philosophers:

Phillip K. Dick, for whom the central theme in his famous 1968 book on AI is empathy.  (If you haven't read it, I'm not posting a spoiler, but will only say the plot is driven by the concept of Evolutionary Game Theory which was formalized just 5 years later.)
John Nash, and in particular, the concept of the Nash Equilibrium. (Nash could be said to have mathematically demonstrated that being a "douchebag" is not an optimal strategy.  His proof can be used to explain why nuclear détente actually worked, which was counter to the expectation of Von Neumann.)

So when people go nuts, focusing on the "Skynet" mythos under which machines rise up to destroy us, I have to wonder if they're simply not as smart as Nash or as profound as Dick, which might explain their lack of emphasis on what can be called the "Electric Sheep" paradigm.


What does the Markov assumption say about the history of state sequences? Does the Markov assumption say that the conditional probability of the next state only depends on the current state or does it say that the conditional probability depends on a fixed finite number of previous states?

As far as I understand from the related Wikipedia article, the probability of the next state $s'$ to appear only depends on the current state $s$.

However, in the book "Artificial Intelligence: A Modern Approach" by Russell and Norvig, on page 568, they say: "Markov assumption — that the current state depends on only a finite fixed number of previous states".

To me, the second statement seems contradictory to the first, because it may mean that a state can depend on the history of states as long as the number is fixed a finite. For example, the current state depended on the last state and the state before the last state, which is 2 sequential previous states (a finite number of states).

Is Markov assumption and Markov property the same?

A stochastic process has the Markov property if the probability distribution of future states conditioned on both the present and past states depends only on the present state or, more formally, the following equality holds.

$$
p(s_{t+1} \mid s_{t}, s_{t-1:1}) = p(s_{t+1} \mid s_{t}), \forall t
$$

The hidden Markov model (HMM) is an example of a model where the Markov property is often assumed to hold. In other words, the Markov assumption is made in the case of the HMM.

There are also the variable-order (or higher-order) Markov models, where the future state can depend on the previously $n$ states or, more formally, the following equality holds.

$$
p(s_{t+1} \mid s_{t}, s_{t-1:1}) = p(s_{t+1} \mid s_{t:t-n}), \forall t
$$

In this context, a hidden Markov model is called a first-order Markov model ($n=0$). Therefore, there can also be second-order ($n=1$), third-order ($n=2$), etc., Markov models. In fact, there are also higher-order hidden Markov models.

To conclude, the expressions Markov property and Markov assumption are not exactly interchangeable. The Markov property is an attribute that a stochastic process can be assumed to possess. In that case, the Markov assumption is made. The expression Markov property usually refers to a first-order Markov property, but it can more generally refer to a higher-order Markov property.


How is the variational lower bound for hard attention derived in Show, Attend and Tell How is the jump from line 1 to line 2 done in equation 10 of Show, Attend and Tell?

While we're at it, another thing that might be muddying the waters for me is that I'm not clear on what the sum is over. I know that $s$ is indexed as $s_{t,i}$, but the one-hot trick from line 2 to 3 makes me believe that the sum is over just $i$.

This is Jensen's inequality at work.
First of all, note that the first line can be rewritten as an expectation
$$\sum_{s} p(s \mid \mathbf{a}) \log p(\mathbf{y} \mid s, \mathbf{a}) = \mathbb{E}_{p(s|a)}[\log p(\mathbf{y} \mid s, \mathbf{a})]$$
Then Jensen's inequality gives (Note that a log function is a concave function so gives the opposite inequality to what is normally given when explaining Jensen's inequality with respect to convex functions):
$$\mathbb{E}_{p(s|a)}[\log p(\mathbf{y} \mid s, \mathbf{a})] \leq \log\mathbb{E}_{p(s|a)}[ p(\mathbf{y} \mid s, \mathbf{a})] $$
and then finally you can rewrite the Expectation as a summation.
$$\log\mathbb{E}_{p(s|a)}[ p(\mathbf{y} \mid s, \mathbf{a})] = \log \sum_{s} p(s \mid \mathbf{a}) p(\mathbf{y} \mid s, \mathbf{a})$$


OpenAI Gym: How is "experience" stored? I have been playing around with the Pong-v0 game. After few days, the AI was able to beat it. I stopped the script for a moment and ran it again. For my surprise it seems it started all over again.

Where and how does OpenAI store the experience?
How can I continue the learning procedure in another computer?

This is sort of a tricky question to answer, by results you could mean 2 possible things. History, which is a culmination of actions, observations and rewards. Or the value func/policy which is the thing you are trying to improve.

With that being said, you for most instances of RL will only be dealing with the last few events in the history, or sampling uniform randomly from it according to your route of exploration. So I don't think this is what you are talking about.

As far as storing your value func/policy OpenAI offers a way to post and to a certain extent store your result. This Readme will show you how to do so.

Edit: OpenAI does not offer anything to store experience out of the box, as I noted in my comment you could pretty easily whip up some code that writes JSON to some file and saving that would be a store of experience. Although, it looks like you already found something that does this in tensorflow.

OpenAI likely hasn't included it up to this point because they don't put a ton into this framework. I would encourage you check out SerpentAI on Github, as it has a lot of features missing in OpenAI.

Hope I answered your question!


Are there any active areas of research in machine learning that do not involve neural networks at all? So far, I have not been able to find many papers that do not involve neural networks, so I was hoping I can gain some insight here. Any references would be greatly appreciated.

If you look into the top conferences on machine learning and neural networks, such as NeurIPS, ICLR, and ICML, you will find many papers related to neural networks and deep learning, given that these are still very hot/promising topics. However, occasionally, you will find accepted papers that do not involve neural networks. Here's a small list of them that I've found after a quick search.

Near-Tight Margin-Based Generalization Bounds for Support Vector Machines (ICML, 2020, pdf) by Allan Grønlund et al.
Implicit Regularization of Random Feature Models (ICML, 2020, pdf) by Arthur Jacot et al.
Quantum Expectation-Maximization for Gaussian mixture models (ICML, 2020, pdf) by Alessandro Luongo et al.
From PAC to Instance-Optimal Sample Complexity in the Plackett-Luce Model (ICML, 2020, pdf) by Aadirupa Saha et al.

So, yes, research in machine learning is not exclusively devoted to neural networks and deep learning. You can probably find more papers here or in the proceedings of similar conferences or journals.


Alpha zero before move 8 The Alpha zero paper says that the The first set of features are repeated for each position in a T = 8-step history. So what happens before the first 8 moves? Do they just repeat the starting position?

On page 13, right under Table S1 in the linked paper, this is explained (emphasis in bold at the end mine):

  Each set of planes represents the board position at a time-step $t - T + 1, \dots, t$, and is set to zero for time-steps less than $1$.

I suspect the solution they write there would indeed work better than just repeating the starting position up to 8 times. Intuitively, you'll want the Neural Network to learn to primarily focus on the current game state. If the starting position is repeated a bunch of times in those planes, the Neural Network cannot distinguish between any of them in the learning process in the first few steps, and may start relying on them all equally. Only in later time steps will it "figure out" that they're sometimes not equal, and that the last one is probably the most informative one. If the "useless planes" during the first few steps are all-zero, they can be much more easily ignored in the start of the learning process.

Note that I suspect the difference really won't matter a whole lot at all, I suspect there'd just be a tiny difference in learning speed based on the intuition described above.


Convolutional Layers on a hexagonal grid in Keras Keras' convolutional and deconvolutional layers are designed for square grids. Is there was a way to adapt them for use in hexagonal grids?

For example, if we were using axial coordinates, the input of the kernel of radius 1 centered at (x,y) should be:

[(x-1,y), (x-1,y+1), (x,y-1), (x,y+1), (x+1,y-1), (x+1, y)]

One option is to fudge it with a 3 by 3 box, but then you are using cells at different distances.

Some ideas:

Modify Kera's convolutional layer code to use those inputs instead of the default inputs. The problem is that Kera calls its backend instead of implementing it itself, which means we need to modify the backend too.
Use a 3 by 3 box, but set the weights at (x-1,y-1) and (x+1,y+1) to zero. Unfortunately, I do not know how to permanently set weights to a given value in Kera.
Use cube coordinates instead of Axial coordinates. In this case, a 3 by 3 by 3 box will only contain the central hex's neighbors and inputs set to 0. The problem is that it makes the input array much bigger. Even more problematic, some coordinates that correspond to non-hexes (such as (1,0,0)) will be assigned non-zero outputs (since (0,0,0) falls within its 3 by 3 by 3 box).

Are there any better solutions?

I had a similar problem with a 2D convolution on a hexagonal grid while working on a diffusion problem and stumbled upon this question. Rather than using cube coordinates, you could use doubled coordinates, which I found much easier to save in a 2D array.

An example kernel that only changes the direct neighbours and the cell itself would be this

kernel = np.array([[0,   0.1, 0,   0.1, 0  ],
                   [0.1, 0,   0.4, 0,   0.1],
                   [0,   0.1, 0,   0.1, 0  ]])

I don't know much about Keras, but I assume that this is possible there as well.


Is AI entirely a part of Computer Science? Both AI and Computer Science are Sciences, as I understood from Wikipedia, Computer Science is everything that has any relation to computers. And AI is commonly defined as

  Study of machines that take the prerogative of humans (creating musical pieces e.t.c

But recently, when I was reading, I read this sentence : "In Computer Science, AI is [...]"

So my question is really : Is there a part of AI studies that do not refer to Computer Science?

AI is an amalgamation of many fields, Computer Science plays a major role in imparting "Intelligence" to the machine. Following is a quote from the best selling AI book Artificial Intelligence: A Modern Approach by
Stuart J. Russell and Peter Norvig.

Artificial Intelligence (AI) is a big field, and this is a big book.
We have tried to explore the full breadth of the field, which
encompasses logic, probability, and continuous mathematics;
perception, reasoning, learning, and action; and everything from
microelectronic devices to robotic planetary explorers.

So, the answer to your question is yes, there are other fields that AI depends including mathematics (to optimize AI algorithms), electronic components (sensors, microprocessors, etc.), mechanical actuators (hydraulic, pneumatic, electric, etc.)
I highly recommend the book if you are looking for a starting point.


Are there dynamic neural networks? Are there neural networks that can decide to add/delete neurons (or change the neuron models/activation functions or change the assigned meaning for neurons), links or even complete layers during execution time?

I guess that such neural networks overcome the usual separation of learning/inference phases and they continuously live their lives in which learning and self-improving occurs alongside performing inference and actual decision making for which these neural networks were built. Effectively, it could be a neural network that acts as a Gödel machine.

I have found the term dynamic neural network but it is connected to adding some delay functions and nothing more.

Of course, such self-improving networks completely redefine the learning strategy, possibly, single shot gradient methods can not be applicable to them.

My question is connected to the neural-symbolic integration, e.g. Neural-Symbolic Cognitive Reasoning by Artur S. D'Avila Garcez, 2009. Usually this approach assigns individual neurons to the variables (or groups of neurons to the formula/rule) in the set of formulas in some knowledge base. Of course, if knowledge base expands (e.g. from sensor readings or from inner nonmonotonic inference) then new variables should be added and hence the neural network should be expanded (or contracted) as well.

This article on Dynamically Expandable Neural Networks (DEN) (by Harshvardhan Gupta) is based on this paper Lifelong Learning with Dynamically Expandable Networks (by Jeongtae Lee, Jaehong Yoon, Eunho Yang, Sung Ju Hwang)

This presents 3 solutions to increase the capacity of the network if needed retaining whatever useful information from the old model and train the new model:

Selective retraining
Dynamic Network Expansion
Network Split/Duplication

To me, it seems that such neural network is dynamic and improving. As such, they answer partially your question. If they don't sorry about that.


If vanishing gradients are NOT the problem that ResNets solve, then what is the explanation behind ResNet success? I often see blog posts or questions on here starting with the premise that ResNets solve the vanishing gradient problem.
The original 2015 paper contains the following passage in section 4.1:

We argue that this optimization difficulty is unlikely to
be caused by vanishing gradients. These plain networks are
trained with BN, which ensures forward propagated
signals to have non-zero variances. We also verify that the
backward propagated gradients exhibit healthy norms with
BN. So neither forward nor backward signals vanish. In
fact, the 34-layer plain net is still able to achieve competitive accuracy, suggesting that the solver works
to some extent.

So what's happened since then? I feel like either it became a misconception that ResNets solve the vanishing gradient problem (because it does indeed feel like a sensible explanation that one would readily accept and continue to propagate), or some paper has since proven that this is indeed the case.
I'm starting with the initial knowledge that it's "easier" to learn the residual mapping for a convolutional block than it is to learn the whole mapping. So my question is on the level of: why is it "easier"? And why does the "plain network" do such a good job but then struggle to close the gap to the performance of ResNet. Supposedly if the plain network has already learned reasonably good mappings, then all it has left to learn to close the gap is "residual". But it just isn't able to.

They explained in the paper why they introduce residual blocks. They argue that it's easier to learn residual functions $F(x) = H(x) - x$ and then add them to the original representation $x$ to get hidden representation $H(x) = F(x) + x$ than it is to learn hidden representation $H(x)$ directly from original representation. That's the main reason and empirical results show that they might be right. Better gradient propagation might be an additional bonus but that's not why they originally introduced the idea.
"Normal" networks work too but, at some point, they become too deep and start working worse than shallower versions (they empirically showed that in the paper). Again, they argue that the reason for that might be that at deeper layers hidden representations become approximately similar $H_n \approx H_{n+1}$ because representation is already well learned and you only need some slight adjustments.  That would mean that transformation for deeper layers is similar to identity transformation and that ordinary layers might have trouble learning that, while for residual blocks it would be easy to learn slight modification and add that to the already existing representation from the previous layer.


In deep learning, do we learn a continuous distribution based on the training dataset? At least at some level, maybe not end-to-end always, but deep learning always learns a function, essentially a mapping from a domain to a range. The domain and range, at least in most cases, would be multi-variate.

So, when a model learns a mapping, considering every point in the domain-space has a mapping, does it try to learn a continuous distribution based on the training-set and its corresponding mappings, and map unseen examples from this learned distribution? Could this be said about all predictive algorithms?

If yes, then could binary classification be compared to having a hyper-plane (as in support vector classification) in a particular kernel-space, and could the idea of classification problems using hyper-planes be extended in general to any deep learning problem learning a mapping?

It would also explain why deep learning needs a lot of data and why it works better than other learning algorithms for simple problems.

Well, there are some questions here...

Does it (Deep Learning) try to learn a continuous distribution based
on the training-set and its corresponding mappings, and map unseen
examples from this learned distribution?

Yes. Talking about Deep Artificial Neural Networks, they try to learn continuous distribution using continuous activation functions in each neuron. Therefore, the output is also a continuous function to represent a continuous probability distribution. The issue with the unseen examples is the need for similar examples in the training set; otherwise, the weights and bias of the network will not be tuned in the regions of space around the unseen example. Imagine a Neural Network learning a function y = x, if we only present values between 0 and 10 during training, we should expect it to only make good predictions for y for values of x ranging from 0 to 10. It doesn't mean that it won't predict for other values, but the predictions will not be so accurate or nowhere close to the expectations. That is because the network is not trying to guess what was the function used to generate y, but it is simply trying to adjust its parameters to make its internal functions generate the expected y for the given x. That is why Deep Neural Networks require a lot of data. In a unidimensional space is easier to provide examples that cover the subset of the domain we want our network to learn. When we use multidimensional space, we need a lot more examples to have a good representation of the hyperspace used as domain.

Could this (map unseen examples) be said about all predictive algorithms?

Yes, it should. Otherwise, the algorithm would not be able to generalize well. A good predictive algorithm is the one that can predict unseen examples using fewer training samples.

Could the idea of classification problems using hyper-planes be
extended in general to any Deep Learning problem learning a mapping?

In the case of Deep Neural Networks, the result is more like, for a given input value, return the probability of it belonging to a class. For binary classification, the network will have a single output. The sigmoid function modulates this output to ranges between 0 and 1. We can interpret the output as the probability of belonging to one out of two possible classes. To know the probability for the other class, we subtract it from 1. For three or more classes, we will need three or more outputs ranging from 0 to 1, and each output is the probability of belonging to one of the classes. In this case, the outputs are also normalized by a softmax function, that guarantees that the sum of all outputs is equal to 1, as a probability distribution.

Would also explain why Deep Learning needs a lot of data and why it
works better mostly than other Learning algorithms for simple
problems.

Already partially explained... The need for a lot of data is to have a good representation of the hyperplane used as the domain.
The Deep Neural Networks work well because of their power to represent different models. They are a very 'flexible' functions that can be bent to approximate the relation existent between the data in the training set and the expected target. Simpler algorithms, as linear models, for instance, have less representation power, they are limited to a smaller set of models. Even though many models can be linearly approximated (because the input and output almost follow a linear relation), the neural network will be able to learn the nuances of the dataset better. This can also be the curse of neural networks, because they may try to learn every detail of the training set that wasn't really relevant and true for other cases and this concept is called overtraining... but is a discussion for another topic.


What is the difference between artificial intelligence and artificial neural networks? I have made several neural networks by using Brain.Js and TensorFlow.js.

What is the difference between artificial intelligence and artificial neural networks?

Artificial intelligence is a broad field, ANN is a specific technique with in that field.


How to mathematically describe the convolution operation (with a Gaussian kernel)? I have to build a model where I pre-process the data with a Gaussian kernel. The data are an $n\times n$ matrix (i.e one channel), but not an image, thus I can't refer to this matrix as an image and to its elements as pixels. The Gaussian kernel is built by the following function (more i.e. here)
$$\begin{equation}
\begin{aligned}
g(x,y,\sigma) =
\dfrac{1}{2\pi\sigma^2} e^{\dfrac{-(x^2+y^2)}{2\sigma^2}}.
\end{aligned}
\end{equation}$$
This kernel is moving one by one element and doing convolution. In my case, most of the elements are zero, the matrix is sparse.
How can I describe/understand the process of convolving the original data with a Gaussian kernel?
I have been looking for some articles, but I am unable to find any mathematical explanations, only explanation in words or pseudo-code.

Mathematically, the convolution is an operation that takes two functions, $f$ and $g$, and produces a third function, $h$. Concisely, we can denote the convolution operation as follows
$$f \circledast g = h$$
In the context of computer vision and, in particular, image processing, the convolution is widely used to apply a so-called kernel (aka filter) to an input (typically, an image, but this does not have to be the case). The input (e.g. an image), the kernel, and the output of the convolution, in this context, is usually a matrix or a tensor. In image processing, the convolution is typically used to e.g. blur images or maybe to remove noise.
However, in the beginning, I said that the convolution is an operation that takes two functions (and not matrices) and produces a third one, so these two explanations of the convolution do not seem to be consistent, right?
The answer to this question is that the two explanations are consistent with each other. More precisely, if you have a function $f : X \rightarrow Y$ (assuming that $X$ is discrete/countable), you can represent it in a vector form as follows $\mathbf{f} = [y_1, y_2, \dots, y_n]$, i.e. $\mathbf{f}$ is a vector that contains all outputs of the function $f$ (for all possible inputs).
In image processing, an image and a kernel can also be thought of as a function with a discrete domain (i.e. the pixels), so the matrices that represent the image or the kernel are just the vector forms of the corresponding functions. See this answer for more details about representing an image as a function.
Once you understand that the convolution in image processing is really the convolution operation as defined in mathematics, then you can simply look up the mathematical definition of the convolution operation.
In the discrete case (i.e. you can think of the function as vectors, as explained above), the convolution is defined as
$${\displaystyle h[n] =  (f  \circledast g)[n]=\sum _{m=-M}^{M}f[n-m]g[m].} \tag{1}\label{1}$$
You can read equation $1$ as follows

$f  \circledast g$ is the convolution of the input function (or matrix) $f$ and the kernel $g$
$(f  \circledast g)[n]$ is the output of the convolution $f  \circledast g$ at index (or input position) $n$ (so you need to apply equation \ref{1} for all $n$, if you want to have $h$ and not just $h[n]$)
So, the result of the convolution at $n$, $h[n]$, is defined as $\sum _{m=-M}^{M}f[n-m]g[m]$, a sum that goes from $m = -M$ to $m = M$. Here $M$ may be half of the length of the kernel matrix. For example, if you use the following Gaussian kernel, then $M = 2$ (and I assume that the center of the kernel is at coordinate $(0, 0)$).

$$
\mathbf{g} =
\frac{1}{273}
\begin{bmatrix}
1 & 4 & 7 & 4 & 1 \\
4 & 16 & 26 & 16 & 4 \\
7 & 26 & 41 & 26 & 7 \\
4 & 16 & 26 & 16 & 4 \\
1 & 4 & 7 & 4 & 1
\end{bmatrix}
\label{2}\tag{2}
$$
Here are some notes:

The kernel \ref{2} is symmetric around the $x$ and $y$ axes: this actually implies that the convolution is equal to the cross-correlation, so you don't even have to worry about their equivalence or not (in case you have ever worried about it, which would have happened only if you already came across the cross-correlation). See this question for more info.

The kernel \ref{2} is the vector form of the function form of the 2d Gaussian kernel (the one in your question): more precisely, an integer-valued approximation of the 2D Gaussian kernel when $\sigma = 1$ (as stated in your slides).

The convolution can be implemented as matrix multiplication. This may not be useful now, but it's something useful to know if you want to implement it. See this question for more info.

Question for you: what is the result of the application of this Gaussian kernel to any input? What does this kernel intuitively do? Once you fully understand the convolution, you can answer this question.


Can we teach an artificial intelligence through sentences? Could we teach an AI with sentences such as "ants are small" and "the sky is blue"? Is there any research work that attempts to do this?

I think that what you're really asking about is the question of knowledge representation.  Regardless of how you train your AI, one of the most fundamental questions is how do you represent "knowledge" and especially when it exists at different levels of abstraction, may be mutually recursive, etc.  Along with that goes the question of belief revision which deals with how you update existing beliefs/knowledge in the light of new information.

Both of these areas are still subject to plenty of active research and neither has entirely settled answers to the core questions.  But progress has been made in both areas.

Personally I suspect that something like semantic networks or conceptual graphs will be the best answer to the KR problem.  Dealing with belief revision seems even fuzzier to me, although there are known strategies (like the AGM postulates) that work to a point. Something like Bayesian Belief Networks may also prove useful.


What part of the game is the value network trained to predict a winner on? The Alpha Zero (as well as AlphaGo Zero) papers say they trained the value head of the network by "minimizing the error between the predicted winner and the game winner" throughout its many self-play games. As far as I could tell, further information was not given.

To my understanding, this is basically a supervised learning problem, where, from the self-play, we have games associated with their winners, and the network is being trained to map game states to the likelihood of winning. My understanding leads me to the following question:

What part of the game is the network trained to predict a winner on?

Obviously, after only five moves, the winner is not yet clear, and trying to predict a winner after five moves based on the game's eventual winner would learn a meaningless function. As a game progresses, it goes from tied in the initial position to won at the end.

How is the network trained to understand that if all it is told is who eventually won?

To my understanding, this is basically a supervised learning problem, where from the self play we have games associated with their winners, and the network is being trained to map game states to likelihood of winning.

Yes, although the data for this supervised learning problem was provided by self-play. As AlphaZero learned, the board evaluations of the same positions would need to change, so this is a non-stationary problem, requiring that the ML forgot the training for older examples over time.

  What part of the game is the network trained to predict a winner on?

Potentially all of it, including the starting empty board. I am not sure if the empty board was evaluated in this way, but it is not only feasible, but can be done accurately in practice for simpler games (Tic Tac Toe and Connect 4 for example), given known player policies.

  Obviously after only five moves, the winner is not yet clear, and trying to predict a winner after five moves based on the game's eventual winner would learn a meaningless function.

Not at all. This is purely a matter of complexity and difficultly. In practice at such an early stage, the value network will output something non-committal, such as $p=0.51$ win chance for player 1. And it will have learned to do this, because in its experience during self-play similar positions at the start of the game lead to almost equal numbers of player 1 and player 2 winning.

The function is not meaningless either, it can be used to assess results of look-ahead searches without needing to play to the end of the game. It completely replaces position-evaluation heuristics as used in more traditional game tree searches. In practice, very early position data in something as complex as chess or go is not going to be as useful as later position evaluations, due to ambivalent predictions. However, for consistency it can still be learned and used in the game algorithms.

  How is the network trained to understand that, if all it is told is who eventually won?

If a supervised learning technique is given the same input data $X$ that on different examples predicts the labels $A, B, B, B, A, A, B, B$, then it should learn $p(B|X) = 0.625$. That would minimise a cross-entropy loss function, and is what is going on here.


How to prevent image recognition of my dataset with neural networks and make it hard to train them? Suppose I have a private set of images containing some objects.
How do i

Make it very hard for the neural networks such as ImageNet to recognize these objects, while allowing humans to do it at the same time?

Suppose I label these private images - a picture of a cat with a label "cat" - how do I make it hard for the attacker to train his neural network on my labels? Is it possible to somehow fool a neural network so that they couldn't easily train it to recognize it?

Like random transforms etc, so that they couldn't use a neural network to recognize these objects, or even train it on my dataset if they had labels.

If the model is trained and held constant, then there are so-called adversarial attacks to modify images such that the model classifies them incorrectly (see Attacking Machine Learning
with Adversarial Examples).
However, if you want to make images that are untrainable, you are probably out of luck. Deep neural networks can learn to recognize even random images with random labels (see Understanding deep learning requires rethinking generalization), though if there's no rhyme or reason to the randomness, they won't generalize in meaningful ways.