How does Microsoft use AI to make Windows 10 updates smoother According to this news, Microsoft is using AI to make Windows 10 updates smoother. So I was curious and went further to search and came across this website, which describes:

  Artificial Intelligence (AI) continues to be a key area of investment
  for Microsoft, and we’re pleased to announce that for the first time
  we’ve leveraged AI at scale to greatly improve the quality and
  reliability of the Windows 10 April 2018 Update rollout.  Our AI
  approach intelligently selects devices that our feedback data indicate
  would have a great update experience and offers the April 2018 Update
  to these devices first.  As our rollout progresses, we continuously
  collect update experience data and retrain our models to learn which
  devices will have a positive update experience, and where we may need
  to wait until we have higher confidence in a great experience.  Our
  overall rollout objective is for a safe and reliable update, which
  means we only go as fast as is safe.

  Our AI/Machine Learning approach started with a pilot program during
  the Windows 10 Fall Creators Update rollout.  We studied
  characteristics of devices that data indicated had a great update
  experience and trained our model to spot and target those devices.  In
  our limited trial during the Fall Creators Update rollout, we
  consistently saw a higher rate of positive update experiences for
  devices identified using the AI model, with fewer rollbacks,
  uninstalls, reliability issues, and negative user feedback. For the
  April 2018 Update rollout, we substantially expanded the scale of AI
  by developing a robust AI machine learning model to teach the system
  how to identify the best target devices based on our extensive
  listening systems.

To me, it sounds like simple if-else statements would have implemented the whole thing without touching the AI; they mentioned that positive experiences include fewer rollbacks, uninstalls, and so on, so we may use these as a criterion of a positive experience.

I am just wondering if the word 'AI' is being misused, or can be misleading in this context? Could anyone point me out on this or give any insight on how AI can be used in this context? In my experience, I have only seen AI mostly being used in speech recognition, image recognition and other sort-of classifying problems, with a training and consequently a computer can "learn" from the data, not like an if-else statement. Today, AI seems to be everything that is considered "smart"?

Hint :  According to antlersoft's comment which says

  ".....Microsoft  may well have used a neural network classifier to
  scan through it's telemetry  to find good  update candidates, it's
  clearly a classification problem.... "

This could be right,however,here questions (information) concerning the company's internal research and development of their software products is classified.

Any machine learning engineer; is not professionally  allowed to discuss nor give out information pertaining the company's product i.e. source codes,plan/design or even the flow of the conceptual idea on how the system is implemented,and this is inline with what the question is all about.

Therefore,if you're passionate about how Microsoft implements artificial intelligence in previous operating system software source codes,then by all means you have to check out its;

BetaArchive Discussion and Collection of Betas and Abandonware;

If you're a serious machine learning engineer,this community can help you as well,it releases them for developers and those who are passionately about open source project contribution.

The archive holds terabytes of files,which you can download for free.
So select which windows files have some machine learning algorithms in!
I wanted to give you a little bit of a glimpse,if you're a machine learning engineer.

For your information; answers to this question, will tend to be almost entirely based on opinions,rather than facts concerning Microsoft's software products.


Which matrix represents the similarity between words when using SVD? Two words can be similar if they co-occur "a lot" together. They can also be similar if they have similar vectors. This similarity can be captured using cosine similarity. Let $A$ be a $n \times n$ matrix counting how often $w_i$ occurs with $w_k$ for $i,k = 1, \dots, n$. Since computing the cosine similarity between $w_i$ and $w_k$ might be expensive, we approximate $A$ using truncated SVD with $k$ components as: $$A \approx W_k \Sigma W^{T}_{k} = CD$$

where $$C = W_{k} \Sigma \\ D = W^{T}_{k}$$

Where are the cosine similarities between the words $w_i$ and $w_k$ captured? In the $C$ matrix or the $D$ matrix?

You can find some material here and here but the idea (at least in this case) is the following: consider the full SVD decomposition of the symmetric matrix $A = W \Delta W^T$. We want to calculate the cosine similarity between the $i$-th column (aka word) $a_i$ and the $j$-th column $a_j$ of $A$. Then $a_k = A e_k$, where $e_k$ is the $k$-th vector of the canonical basis of $\mathbb{R}$. Let's call $\cos(a_i,a_j)$ the cosine between $a_i,a_j$. Then
$$\cos(a_i,a_j) = \cos(Ae_i,Ae_j) = \cos(W \Delta W^T e_i,W \Delta W^T e_j) = \cos(\Delta W^T e_i,\Delta W^T e_j)$$
where the last equality holds because $W$ is an orthogonal matrix (and so $W$ is conformal, i.e. it preserves angles). So you can calculate the cosine similarity between the columns of $\Delta W^T$. A $k$-truncated SVD gives a well-enough approximation. In general, columns of $W \Delta$ and rows of $W$ have different meanings!


How can I update my Q-table in Python? I want to implement this function on a voice searching application:
$$
Q(S, A) \leftarrow Q(S, A)+\alpha\left(R+\gamma Q\left(S^{\prime}, A^{\prime}\right)-Q(S, A)\right)
$$
And also restricted to use epsilon-greedy policy based on a given Q-function and epsilon. I simply need a $\epsilon$-greedy policy for updating my q-table.

Just try returning a function that takes the state as an input and returns the probabilities for each action in the form of a numpy array of length of the action space (set of possible actions). Here, is one attempt:
def EpsilonGreedyPolicy(Q, epsilon, no_of_actions):

    def policy(state):
        probabilities = np.ones(no_of_actions, dtype = float) *
                    epsilon / num_actions
        best_action = np.argmax(Q[state])
        probabilities[best_action] += (1.0 - epsilon)
        return probabilities

    return policy


How can I develop a genetic algorithm with a constraint on the sum of alleles? I'm working on a genetic algorithm with a constraint on the sum of the alleles, e.g. if we use regular binary coding and a chromosome is 5-bits long I'd like to constrain it so that the sum of the bits has to be 3 or less (011100 is valid but 011110 is not). Moreover, the fitness function is such that invalid chromosomes cannot be evaluated.

Any ideas on how this problem could be approached?

I've started looking into the direction of messy GAs (since those can be over-specified) but I'm not sure if there's anything there.

There are multiple ways to handle 'illegal' individuals, each one with pros and cons:

Abortive methods: The individuals that violate constraints are eliminated as soon as discovered (i.e. after crossover or mutation) and new individuals are generated in order to keep the population stable. This usually implies a slower creation of new generations, as some individuals are discarded.
Contraceptive methods: The crossover and mutation are written in such a way to make it impossible for a newly generated individual to violate any constraints. This way to act is usually more efficient, as no individuals are discarded, but it might not be possible.
Penalization function: The fitness function gives a huge penalization to the individuals that violate constraints (usually proportionally to the constraints it violates). In this way they usually do not get to reproduce and their genes are eventually lost. You'd go this way if illegal individuals are very rare and there is no possibility for them to take over the full population (causing the algorithm to fail).

By experience, try to act on the crossover algorithm first (contraceptive way), it is the best way to exclude constraints violations. If this is not possible, pick one of the other two methods, depending on how often the illegal individuals are generated (not so often -> penalization, very ofter -> abortive)  and on how easy it is to penalize individuals that violate constraints (easy -> penalization, not easy -> abortive).

A contraceptive way to handle your example is writing the crossover as following:

bool[] crossover(bool[] p1, bool[] p2)
{
    // classical binary crossover
    bool[] child = p2.copy()
    child[rndSection] = p1[rndSection]

    // contraceptive part
    while (CountTrue(child) > 3) child[RandomIndexOfTrue(child) = false

    return child
}

But this is a fairly complex problem and each scenario has its own specifications.


How much is currently invested in artificial general intelligence research and development? How much is currently invested in artificial general intelligence research and development worldwide?
Feel free to add company or VC names, but this is not the point. The point is to get an idea of the economics around artificial general intelligence.

In the last years, there have been big investments in AI technologies. For an overview, maybe take a look at this article Artificial Intelligence: Investment Trends and Selected Industry Uses (2019).
A few companies that have the long-term goal of creating an AGI, although, currently, they mainly do research on specific problems (e.g. video games) or AI techniques (e.g. reinforcement learning), have received many funds. I will only list a few (maybe the most well-known ones) of these companies below, but there are probably many other companies that have this long-term goal and have been funded by other companies or people.
DeepMind
In their site, they write

Like the Hubble telescope that helps us see deeper into space, we aim to build advanced AI - sometimes known as Artificial General Intelligence (AGI) - to expand our knowledge and find new answers. By solving this, we believe we could help people solve thousands of problems.

DeepMind was acquired by Google in 2015 for $500 million, given its success in playing games at a superhuman performance, which is a promising step towards the development of more AI techniques and maybe AGI.
The Wikipedia article on DeepMind contains some information about people or companies that have invested in DeepMind, which includes companies Horizons Ventures and Founders Fund, and people  Scott Banister, Peter Thiel, Elon Musk, and Jaan Tallinn, although I cannot give you the exact numbers in terms of capital. In any case, serious investments have been done in DeepMind, which is definitely one of the promising companies that could develop good insights into the development of AGI systems.
OpenAI
Another company that has a similar goal and is doing research on similar topics (such as reinforcement learning or natural language processing) is OpenAI, which also has the long-term goal of creating AGI systems, as they write in their website

OpenAI’s mission is to ensure that artificial general intelligence (AGI) — by which we mean highly autonomous systems that outperform humans at most economically valuable work — benefits all of humanity. We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.

Their investors include Microsoft, Reid Hoffman's charitable foundation, and Khosla Ventures.
Vicarious
They write in their site

Artificial general intelligence is the finish line on our journey toward progressively more capable robots. Our approach leverages deep expertise in neuroscience and is shaped by a decade of research.

They are apparently backed by more than 150 million dollars from people like Jeff Bezos, Elon Musk and Mark Zuckerberg and companies like Samsung.


Do Gödel's theorems imply that intelligence systems may end up in some undecidable situation (that may make them take a wrong decision)? So far I understand - I know very little on the topic - the core of AI boils down to design algorithms that shall provide a TRUE/FALSE answer to a given statement. Nevertheless, I am aware of the limitations provided by the Gödel's incomplete theorems but I am also aware that there have been long debates such as the Lucas and Penrose arguments with all the consequent objections during the past 60 years.

The conclusion is, in my understanding, that to create AI systems we must accept incompleteness or inconsistency.

Does that mean that intelligence systems (including artificial ones), like humans, may end up in some undecidable situation that may lead to take a wrong decision?

If this may be acceptable in some application (for example, if every once in a while a spam email ends up in the inbox folder - or vice versa - despite an AI-based anti-spam filter) in some other application it may not. I am referring to real-time critical applications when a "wrong" action from a machine may harm people.

Does that mean that AI will never be employed for real-time critical applications?

Would in that case more safe to use deterministic methods that do not leave room for any kind of undecidability?

Your initial statement on the core of AI is rather limited. In general, AI is concerned with modeling human behaviour either by imitation (soft AI) or by replicating the way human cognition works (hard AI). So far there have been some successes with soft AI, as computers can perform tasks that required some "intelligence", though the degree of this intelligence is questionable. This is partly due to the fact that even we as humans don't really have a clear idea what it means for a computer to "understand" something.

But your conclusion is correct: if we build an AI system with human characteristics, then it will make mistakes, just as humans make mistakes. And any system designed by humans (or machines!) will make mistakes. However, not being able to deal with an imperfect world is not really relevant to AI alone: even systems that do not use AI methods will have to face that, and whether a system is suitable for real-time critical applications has got nothing to do with whether it is based on AI or not.

UPDATE: There seem to be two distinct issues at play here: decidability and real-time processing.

Real-time computing (RTC): This is not really related to AI. Even ordinary programmes written in Java are not really safe for RTC, as they could start a garbage collection cycle at any time which pauses execution of the program. Just imagine a reactor core starts overheating just as your controller runs out of memory and garbage collection kicks in, halting the program for a few minutes. If you implement AI methods in RTC-safe systems, that should not be an issue.
Decidability: Your reasoning is that AI systems attempt to mirror human cognition, thus incorporating the ability to make mistakes. This is a more philosophical issue — if a human can control a system, then an AI system with the same capabilities should be able to do it too. This assumes that we are able to replicate human behaviour (which we are not). There are AI methods which are deterministic, so would come to the same conclusions given identical environments. So I would say that they would not perform worse than non-AI methods. It partly depends what you want to call AI; the distinction between traditional AI and statistical methods keeps getting blurred at present.

To conclude: No, AI methods should be suitable, as they can also be deterministic. It depends on the actual application and method if they are. And, of course, on what you count as AI.


A neural network to learn the connection between two totally different type of images I have a dataset of two different type of images. Say, I have images of a person and his all 10 fingerprints. I want to create a relation between them to predict one from another. How I can do that and which architecture is suitable for this problem or similar type of problem.

I would try a pair of separate deep image embeddings with a contrastive loss. The idea is similar to the Siamese network architecture. In Siamese networks the pairs of images are of the same type - so both input images are fed through the copy of the same network. In your case the images are of different kinds, so I would just have separate nets for person images and fingerprints.


How do we choose the activation function for each hidden node? I am new to neural networks. I would like to use them as a fitting or forecasting method.

A simple NN model that does not contain hidden layers, that is, the input nodes are directly connected to the outputs nodes, represents a linear model.  Nonlinearity begins to appear in an ANN model when we have hidden nodes, in which a nonlinear function is assigned to the hidden nodes, and using minimization their weights are determined.

How do we choose the non-linear activation function that should be assigned to each hidden neuron?

To know the form of your non-linear function,  firstly you should define the type of problem you are dealing with such as an image classification task. Secondly, pick the activation functions based on your task such as sigmoid, Tanh, ReLu, LeadyRelu, Softmax etc. Overall, your ANN performance mainly depends on the number of hidden layers (hidden units), selection of activation functions,  weight decay, momentum and dropout etc.


Fuzzy Logic Controller: Choosing Membership Function In classical set theory there is two options for an element. It is either a member of a set, or not. But in fuzzy set theory there are membership functions to define "rate" of an element being a member of a set. In other words, classical logic says it is all black or white, but fuzzy logic offers that there is also grey which has shades between white and black.

Matlab Simulink Library is very easy to design and helpful in practice. And it has good examples on its own like deciding about tip for a dinner looking at service and food quality. In the figure below some various membership functions from Matlab's library are shown:

My question: How do we decide about choosing membership functions while designing a fuzzy controller system? I mean in general, not only in Matlab Simulink. I have seen Triangular and Gaussian functions are used mostly in practise, but how can we decide which function will give a better result for decision making? Do we need to train a neural network to decide which function is better depending on problem and its rules? What are other solutions?

First I need to note that there is no prescribed/best way to choose the shape of membership function in fuzzy systems, that's the fuzziness in it. One could argue that the best way is to ask an expert in the field where you are going to apply your fuzzy solution but those are not always available.

With that said, fuzzy membership functions are used to describe the distribution of probabilities in real world for the variable you are trying to use in your fuzzy controller. That means you go out in the real world, you look at the system you are trying to control, you try your best to understand how it works and reacts to different outside changes and based on your findings you choose the shape that best fits. Or if you want you may call this process heuristic choice (something like that, I do not like theory very much).

On top of that you need to realize one important thing, the shape of membership function does not have big impact on the resulting controller behavior. The most influencing parts are the Fuzzy Rules and their inference methods you use in your controller, but that is a different topic. So no matter what you choose it will not make a big difference.

Gaussian functions are most commonly used because of the character of the world we are living in. Many people argue that everything in the world has Gaussian distribution (everyone is entitled to their opinion). And the triangular functions are used because they are the simplest alternative that is somewhat similar to Gaussian function.

But if you absolutely need to chose the best one for your particular problem. There always are Simulation tools that exists precisely for this purpose. One of them is Matlab Simulink as you mentioned, but there are others if you don't like the price of the Matlab.

My suggestion is, go with your gut, test it in safe environment and if it works then deploy it to the real world.


How many nodes/hidden layers are required to solve a classification problem where the boundary is a sinusoidal function? A single neuron is capable of forming a decision boundary between linearly seperable data. Is there any intuition as to how many, and in what configuration, would be necessary to correctly approximate a sinusoidal decision boundary?

Thanks

It depends on the accuracy you want. If you had 1 neuron, it could discern things across a line, if you have 2, you could solve things across 2 lines, etc. As you increase the number of neurons, you are increasing the number of discernible areas. As you increase the number of lines you can use to break up the input space, the lines can be placed to approximate any curve (sinusoidal) As the number of neurons approaches infinity, the accuracy of categorizing different inputs across this curve increases.

Interestingly enough, if one graphed "Number of Neurons (x) vs Accuracy (y)", it would look sinusoidal.


Can Q-learning be used in a POMDP? Can Q-learning (and SARSA) be directly used in a Partially Observable Markov Decision Process (POMDP)? If not, why not? My intuition is that the policies learned will be terrible because of partial observability. Are there ways to transform these algorithms so that they can be easily used in a POMDP?

The usual (as presented in Reinforcement Learning: An Introduction) $Q$-learning and SARSA algorithms use (and update) a function of a state $s$ and action $a$, $Q(s, a)$. These algorithms assume that the current state $s$ is known. However, in POMDP, at each time step, the agent does not know the current state, but it maintains a "belief" (which, mathematically, is represented as a probability distribution) in what the current state might be, so it cannot maintain (an approximation of) the function $Q(s, a)$. Hence, the usual Q-learning and SARSA algorithms shouldn't be directly applicable to a POMDP.

However, $Q$-learning is often used in the contexts where observations emitted by the environment (or transformations of the raw observations) are used to build the current state (which is assumed to be Markov, even if it is not). For example, in the original DQN, the action taken at the current step and the raw observation and the reward emitted by the environment (after this action is taken) are combined to produce the current (Markov) state. It might not be the case that the way they combine the action, the reward and the observation is sufficient to fully describe the current state (which might not even be Markov).

In this report, Deep Reinforcement Learning with POMDPs, the author attempts to use Q-learning in a POMDP setting. He suggests to represent a function, either $Q(b, a)$ or $Q(h, a)$, where $b$ is the "belief" over the states and $h$ the history of previously executed actions, using neural networks. So, the resulting parameterized functions would be denoted by $Q(b, a; \theta)$ or $Q(h, a; \theta)$, where $\theta$ is a vector representing the parameters of the corresponding neural network. Essentially, the author uses a DQN (with an experience replay buffer and target network), but the results are not great: $Q$ values converge, but policies do not and they are not robust (in that they are sensitive to small perturbations).


How to deal with the terminal state in SARSA in a multi-agent setting? I'm training a SARSA agent to update a Q function, but I'm confused about how you handle the final state. In this case, when the game ends and there is no $S'$.
For example, the agent performed an action based on the state $S$, and, because of that, the agent won or lost and there is no $S'$ to transition to.
So, how do you update the Q function with the very last reward in that scenario, given that the state hasn't actually changed? That case $S'$ would equal $S$ even though an action was performed and the agent received a reward (they ultimately won or lost, so quite important update to make!).
Do I add extra inputs to the state "agent won" and "game finished", and that's the difference between $S$ and $S'$ for the final Q update?
To make clear, this is in reference to a multi-agent/player system.  So, the final action the agent takes could have a cost/reward associated with it, but the subsequent actions other agents then take could further determine a greater gain or loss for this agent and whether it wins or loses. So, the final state and chosen action, in effect, could generate different rewards without the agent taking further actions.

The SARSA update rule looks like:
$$Q(S, A) \gets Q(S, A) + \alpha \left[ R + \gamma Q(S', A') \right].$$
Very similar, the $Q$-learning update rule looks like:
$$Q(S, A) \gets Q(S, A) + \alpha \left[ R + \gamma \max_{A'} Q(S', A') \right].$$
Both of these update rules are formulated for single-agent Markov Decision Processes. Sometimes you can make them work reasonably ok in multi-agent settings, but it is crucial to remember that these update rules should still always be implemented "from the perspective" of a single learning agent, who is oblivious to the presence of other agents and pretends them to be a part of the environment.
What this means is that the states $S$ and $S'$ that you provide in update rules really must both be states in which the learning agent is allowed to make the next move (with the exception being that $S'$ is permitted to be a terminal game state.
So, suppose that you have three subsequent states $S_1$, $S_2$, and $S_3$, where the learning agent gets to select actions in states $S_1$ and $S_3$, and the opponent gets to select an action in state $S_2$. In the update rule, you should completely ignore $S_2$. This means that you should take $S = S_1$, and $S' = S_3$.
Following the reasoning I described above literally may indeed lead to a tricky situation with rewards from transitioning into terminal states, since technically every episode there will be only one agent that directly causes the transition into a terminal state. This issue (plus also some of my explanation above being repeated) is discussed in the "How to see terminal reward in self-play reinforcement learning?" question on this site.


How can I determine whether a video's frame is realistic (was recorded by a camera) or contains computer-generated graphics? Given a video, I'm trying to classify whether it is a graphical (computer-generated) or realistic scene. For instance, if it contains computer-generated graphics, credit, moving bugs, blue screen, etc. it will be computer-generated graphics, and if it is a realistic scene captured by camera, it will be a realistic scene.
How can we achieve that with AI? Do we have any working solutions available?
Some examples of graphical scenes:

As per your requirements, I would suggest that you start with any simple CNN network.

CNNs take advantage of the hierarchical pattern in data and assemble
more complex patterns using smaller and simpler patterns. Therefore,
on the scale of connectedness and complexity, CNNs are on the lower
extreme.

Here is a Keras example:
model = models.Sequential()
model.add(layers.Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=image_shape))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, kernel_size=(3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, kernel_size=(3, 3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
# output layer
model.add(layers.Dense(1))

where image_shape is the resolution and number of channels of images (e.g. 128x128x3 for RGB images). I also suggest you downscaling the images to a lower resolution. You will also have to crop the images as they must all be the same image_shape.
Also take a look at the MaxPooling2D  and BatchNormalization layers.
Since you only have real and CGI images, this becomes a binary classification problem. Therefore you can have a single output (0 - CGI, 1 - real). Such problems can be solved with BinaryCrossentropy loss.
model.compile(loss=losses.BinaryCrossentropy(from_loits=True), optimizer='adam')

Finally, you can fit your model
history = model.fit(train_images, train_labels, epochs=1000, validation_data=(test_images, test_labels))

You can find a complete example here.
Please note that depending on your data, the model can become biased if your dataset is unbalanced. That is, if all of your CGI images have text, and only a small fraction of the real images also have text, they might be misclassified. Therefore, I recommend that you visualize your model to better understand what it has learned. Here is an example of such a problem we faced at our university.
There are also more advanced CNN architectures such as ResNet, VGG or YOLO. You can also extend your model with time series (i.e. video) using LSTM or GRU architecture.


Is Lisp still worth learning today in the particular context of Machine learning? Lisp was originally created as a practical mathematical notation for computer programs, influenced by the notation of Alonzo Church's lambda calculus. It quickly became the favored programming language for artificial intelligence (AI) research, according to Wikipedia.

If Lisp is still used in AI, then is it worthy of learning it, particularly in the context of machine learning and deep learning?

LISP was popular because back in the old days of AI because of the functional syntax, which worked well with the GOFAI paradigm of the time.

Nowadays most researchers have given up on the classical computational theory of mind (read: language of thought), and thus also the GOFAI paradigm that it associates with.

LISP is not what you want to learn if you want to do neural network stuff, but the philosophical background is still important to know.


How can max-pooling be applied to find features in words? I'm reading about max-pooling in a dynamic CNN paper. I can see how it can help find features in images, given that the pixel with the highest density gets pooled, but how does it help to find features in words?

In an image you are pooling usually over some (n x n) set of positions which lets you maintain spatial correlation but on the other hand most 1D CNNs used for language modeling pool over the temporal axis completely annihilating all form of temporal correlation within the resulting feature vector. I will take it this difference is what confuses you.

For simplicity sake Ill keep my explanation to max-pooling but it can extend to any of the others.

MaxPooling as the name suggests just takes a maximum value from a set of candidates. In images you commonly see maxpooling over neighborhoods of pixels. This is because if we assume high activation is correlated to some feature being more prevalent (this assumption is not inherent, its more learned than anything else by construction of the architecture/training), then doing this we keep the most prevalent features, maintain spatial correlation, and reduce the dimensionality.

When using 1D CNNs for LMs rather than RNNs the idea is that each kernel is searching for a feature, and so you use an exorbitant number of the kernels at various sizes to search for a lot of features. Pooling over the temporal dimension then just tells you at each neuron how prevalent its corresponding feature is in that time series input. That is why your output is usually a feature vector of size equal to the number of kernels you used.

A point I want to emphasize is that pooling takes multiple inputs and returns one based on some condition/function but that is vague, and it's usage in different scenarios can be for different goals (to an extent).


What are the popular approaches to estimating the Q-function? I need the q-value for my RL training, there are some approaches:

Brute-force the action sequence (this won't work for long sequence)
Use a classic algorithm to optimise and estimate (this ain't much AI)
Create Monte Carlo samples and train an approximator network for calculating q-value

I find the Monte Carlo method above rather widely applicable to different problems, and the more computing power, the more precise it is. Any other methods for calculating q-value?

There's are some solutions to calculating q-values; find the exact values:

Brute-force the action sequence to find max (not pratical)
Do recursion on Bellman equation to get max (the same like action sequence brute-force, not pratical)

Estimate the q-values:

Based on different problems to solve, apply some classic algorithms or human logics to estimate; during estimation, some heuristic tactics may be used
Do a lot of randomisation to find max, including Monte Carlo tree search

Gradually optimise the q-values:
(1) Do optimisation based on Bellman equation (Q-learning):
$$ q_t(s_t,a_t) = q_t(s_t,a_t) + \alpha(r + \gamma\times\max(q_{t+1}(s_{t+1},a_{t+1})) - q_t(s_t,a_t))$$
Bellman equation is true when the temporal difference (the part multiplied with $\alpha$) reaches zero, which means the max of time t+1 reaches exact value.
(2) Do optimisation based on Bellman equation (Q-network), fit the neural network to expected value:
$$r + \gamma\times\max(q_{t+1}(s_{t+1},a_{t+1}))$$


Maximizing or Minimizing in Trust Region Policy Optimization? I happened to discover that the v1 (19 Feb 2015) and the v5 (20 Apr 2017) versions of TRPO papers have two different conclusions. The Equation (15) in v1 is $\min_\theta$ while the Equation (14) in v2 is $\max_\theta$. So, I'm a little bit confused about which one to choose.

BTW, I found that in the High-Dimensional Continuous Control Using Generalized Advantage Estimation, the Equation (31) uses $\min_\theta$.

The differences you have observed between the two different versions of the TRPO paper are due to different formalizations of the problem and the objective.

In the first version of the paper you linked, they start out in Section 2 by defining Markov Decision Processes (MDPs) as tuples that, among other things, have a cost function $c : \mathcal{S} \rightarrow \mathbb{R}$. They define $\eta(\pi)$ as the expected discounted cost of a policy $\pi$, and subsequently also define state-action value functions $Q_{\pi}(s_t, a_t)$, value functions $V_{\pi}(s_t)$, and advantage functions $A_{\pi}(s, a)$ in terms of costs. Ultimately, in Equation 15, they write the following:

\begin{aligned}
\underset{\theta}{\text{minimize }} & \mathbb{E}_{s \sim \rho_{\theta_{\text{old}}}, a \sim q} \left[ \frac{\pi_{\theta}(a \vert s)}{q(a \vert s)} Q_{\theta_{\text{old}}}(s, a) \right] \\
\text{subject to } & \mathbb{E}_{s \sim \rho_{\theta_{\text{old}}}} \left[ D_{KL}(\pi_{\theta_{\text{old}}}(\cdot \vert s) ~ \vert \vert ~ \pi_{\theta}(\cdot \vert s)) \right] \leq \delta
\end{aligned}

Now, there's a lot going on there, but we can very informally "simplify" it to only the parts that are relevant for this question, as follows:

$$\underset{\theta}{\text{minimize }} \mathbb{E} \left[ Q(s, a) \right]$$

When we look at just that, we see that we're pretty much trying to minimize $Q$-values, which are costs; that makes sense, typically costs are things we want to minimize.

In the second version of the paper you linked, they have changed the Preliminaries in Section 2. Now they no longer have a cost function $c$ in their definition of an MDP; they have replaced it by a reward function $r : \mathcal{S} \rightarrow \mathbb{R}$. Then they move on to define $\eta(\pi)$ as the expected discounted reward (rather than expected discounted cost), and also define $Q$, $V$ and $A$ in terms of rewards rather than costs. This now all matches the standard, common terminology in Reinforcement Learning.

Ultimately, Equation 14 looks identical to what we saw above, it's again about an expectation of $Q$-values. But, now $Q$-values are rewards rather than costs. Rewards are generally things we want to maximize, rather than minimize, so that's why the objective swapped around.


Solving Crossword Puzzles I have to model an AI that should be able to understand clues and find the answer from a specified word database. I came across several papers that solves the problem with training neural networks or processing the clues with training several machine learning architectures on clues that exist on several databases. However, they all seem overkill for the course currently I'm taking.

So, what would be the most simple approach to solve that problem?

Artificial neural network to solve Crossword puzzles, this model might help you to get a start on evolving your own.

https://github.com/fh295/DefGen2

A brute force of word combinations would solve the matrix, but might not get the context right and might be the simplest model if you are not in for a neural network.


Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function? Here is David Silver's lecture on that. Look at 9:30 to 10:30.
He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function $Q$ is used.

But then state-values are already calculated (via first-visit or every-visit). So, why aren't these values used?

Secondly, even if we were to use $Q$, we have $Q^{\pi}(s,a) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s')$, so we still need to know the transition model, which is unknown.

What am I missing here?

In Model Based Reinforcement learning, state and state-action values for all states can be calculated based on the bellman equations. The equations are taken from Andrew Ng's Algorithms for Inverse Reinforcement Learning $$V^{\pi}(s) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s') \\ Q^{\pi}(s,a) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s')$$
In this setting, $Q^{\pi}$ can be obtained from $V^{\pi}$ because we have access to the transition model $P(s'|s,a)$. The $Q^{\pi}$ values allow us to carry out a step in $\textbf{policy improvement}$ as in policy iteration.
To answer the first bullet point, the first visit or every state visit policy evaluation in the model free setting for $\textbf{state values}$ is not helpful in determining how to carry out model free control because we cannot compute $Q^{\pi}(s,a)$ from $V^{\pi}$ in the model free case.
The update for SARSA in model free control is $$Q(s,a) \rightarrow Q(s,a) + \alpha (r(s) + \gamma Q(s',a') - Q(s,a))$$
Even though we do not know the transition model, we are essentially $\textbf{sampling}$ from $P(s'|s,a)$ by allowing the environment to provide us the possible next states $s'$ that we may end up in. The following update for SARSA is equivalent to computing $$Q^{\pi}(s,a) = R(s) + \gamma E_{s' \sim P(s'|s,a)}[Q^{\pi}(s',a')]$$
Essentially this should give the same $Q^{\pi}(s,a)$ values when we have the ground truth $P(s'|s,a)$ values for the model free case.


Is this page of Goodfellow's Deep Learning Textbook missing text? The textbook Deep Learning by Goodfellow, Bengio, and Courville can be viewed in individual html chapters here.

I'm currently reading Chapter 15 on Representation Learning and saw this for algorithm 15.1 on page 530:

There no condition after end for and similarly no end condition for end if. Is that intentional? If so, what is the correct way to interpret it? I haven't encountered algorithm syntax like this in the past.

There is no industry standard for psuedocode as it is read by humans (for the moment ;) and the author can decide how best to represent the concepts of the algorithm with the aim of balancing precision and understandability. In this case there is a

FOR ... END FOR

block which is iterated over k for 1 to m. And an

IF ... END IF

block that is run if fine-tuning evaluates to true.


Is Artificial Intelligence restricted to electrical based technology? According to Wikipedia:

  AI is intelligence exhibited by machines.

I have been wondering if with the recent biological advancements, is there already a non-electrical-based "machine" that is programmed by humans in order to be able to behave like a:

  flexible rational agent that perceives its environment and takes actions that maximize its chance of success at some goal

I was specifically thinking of viruses and bacteria. Have these been programmed by humans in order to behave as a flexible rational agent (i.e. an AI entity)?

Are there are other organisms that have already been used for this purpose?

Not yet. Synthetic virology / Synthetic life are still in their infancy.
We can now synthesize simple bacteria (see Craig Venter's fascinating TED talk and also an article about his recent work) but definitely nothing that may be called 'rational' in human standards.


Feature extraction timeseries, model compatibility I've got a timeseries with sensor data (e.g. accelerometer and gyroscope). I now want to extract the activity out of it (e.g. walking, standing, driving, ...). I Followed this Jupyter Notebook. But there are some issues left.

Why do they only pick 500 rows?
What's the point of re-arranging the rows/columns?
When they build their decicion tree learner with the train data, they build it upon extracted features. But how can we then use this tree for new sensor data? Should we extract the features of the new data and pass it as input for the tree? But new sensor data might not have as many features as the train data. Eg: (ValueError: Number of features of the model must match the input. Model n_features is 321 and input n_features is 312)

there's a lot to un-pack in this question.

Why do they only pick 500 rows?

my guess: in order to keep the example running quickly. tsfresh usually takes a while to calculate its features.  note that when they evaluated their model, they took the last 500 samples.

What's the point of re-arranging the rows/columns?

answer: the data frame format that tsfresh requires in order to calculate features is that format. it is a bit of a pain...especially when you need to keep track of an id-column for other data.

When they build their decicion tree learner with the train data, they build it upon extracted features.

answer: yes

But how can we then use this tree for new sensor data?
Should we extract the features of the new data and pass it as input for the tree?

answer: yes

But new sensor data might not have as many features as the train data. Eg: (ValueError: Number of features of the model must match the input. Model n_features is 321 and input n_features is 312)

answer: yes it will. I don't know where your copy-pasted error message came from. When you generate a set of features using tsfresh, you can do it in a couple of different ways. you can generate all of them or you can generate a subset of them---they generated a subset...but then subsetted it once again(using their importance stuff...whether you use the importance methods or other methods to select features you will end up with a bunch of features you need to calculate). If you follow the procedure for generating features based on a pre-determined subset (relevant_features in their case), you will end up with the same # of features.  This needs to be stressed...don't generate tsfeatures that you don't need! as it will take foooorever.


Are POS tagging, Chunking, Disambiguation, etc. subtasks of annotation? I wonder about the legitimacy of using the terms "POS tagging", "Chunking", "Disambiguation" and "Categorization" to describe an activity that doesn't include writing code and database queries, or interacting with the NLP algorithm and database directly.

More specifically, let's suppose I use the following tools:

an "Annotator" for analyzing the input text (e.g. sentences copypasted from online newspapers) and choose and save proper values as regards to "POS" of tokens and words, "Words"(entities and collocations) and "Chunk". Tokens are already detected by default. I have to decide which words are entities and/or collocations or not and their typology, though. May the performed tasks be called "POS tagging", "Chunking" and "support to categorization"?
A knowledge base, for searching and choosing the proper synsets of the lemmas and assigning them to the words analyzed in the previous Annotation tool. May such a task be called "Disambiguation"?
A graphical user interface which shows how the NLP analyzes by default the input texts as regards to Lemmas, POS, Chunks, Senses, entities, domains, main concepts, dependency tree, in order to make analyses consistent with it.

If I want to define these activities in a few words, "Machine Learning annotation" may be the most correct.

But what if I want to be more specific? I don't know whether or not the terms "POS tagging", "Chunking", "Disambiguation" and "Support to categorization" may be appropriate for they generally come within "programming contexts", as far as I know. In other terms, do they involve writing algorithms and programming or are they / may they be referred to the "less-technical" activities described above?

The procedures you mention don't need to involve writing code. There are now many ready-made tools available which implement various algorithms.

POS-Tagging, Chunking, and Semantic Tagging/Disambiguation are knowledge-based procedures which can all be seen as classification/clustering tasks: POS-Tagging and Disambiguation are classifications, in that they propose a label (a POS- or semantic tag) that is assigned to a token. Chunking is a clustering algorithm, as it finds coherent groups in sequences of tokens. You could also view it as segmentation, as it splits a sentence into parts.

In principle they don't have anything to do with machine learning; the algorithm you are using (ML or not) is an implementation detail. Early Taggers were mostly rule-based, but with increasing availability of annotated data it became more feasible to use ML algorithms. This, however, has got nothing to do with how you classify the procedures.

In general I would call the activity you describe as 'annotation', as you enrich the source text by adding descriptive categories to tokens and token sequences.

So, no, they do not have to involve programming or implementing algorithms, and you don't need to be technical to perform them. Though some knowledge of linguistic concepts would help.


What is the use of softmax function in a CNN? What is the use of softmax function? Why was it used at the end of fully connected layer in convolution neural network?

The main purpose of the softmax function is to transform the (unnormalised) output of $K$ units (which is e.g. represented as a vector of $K$ elements) of a fully-connected layer to a probability distribution (a normalised output), which is often represented as a vector of $K$ elements, each of which is between $0$ and $1$ (a probability) and the sum of all these elements is $1$ (a probability distribution).

In the case of a classification task, the $i$th element of the vector produced by the softmax function corresponds to the probability of the input of the network of belonging to the $i$th class (e.g. a dog).


Why is gradient ascent necessary when training Actor Critic agents? I have read a lot on Actor Critic and I'm not convinced that there is a qualitative difference doing direct gradient updates on the network and slightly adjusting a soft-max output in the direction of the advantage function and doing gradient descent on the error.

Can anyone explain why updating the gradient directly is necessary?

It might seem to give the same update direction but would it converge to desirable policy parameters?

Actor-Critic is proposed alongside the policy gradient theorem in Sutton 1999. It is shown to maximize the state-value function. If you are able to show that the technique of yours is, in fact, maximizing some desirable objective function, you could propose it with some soundness as well.


Is 1mb an acceptable memory size for images being trained in a CNN? I am using Tensorflow CNN to build an image classification/prediction model. Currently all the images in the dataset are each about 1mb in size.

Most examples out there use very small images.

The image size seems large, but I not too sure.

Any thoughts on the feasibility of 1mb images? If not what can I do to compress programmatically?

1 MB by images is too much. It means that you have a lot of pixels to compute in the inputs, and your images have more features that are not very useful for the classification (we human don't need high resolution images for recognize objects, it's the same for model)

It means that maybe you need also a more deeper network, and then, more computation.

In your particular case, you have a dataset of trees, so images are somewhat similar and then, maybe the classification is harder. So, images need enough information in pixels to have a good prediction.

You should resize your whole datsets to more common size because 1mb are too much (I don't know the resolution, but I think it could be around thousand for both dimension).
Imagenet have images of size 224x224, and there are 1000 classes, which some of them are close. For examples, there are many dogs classes (boxer, american terrier, ...), cats classes (tiger, egyptian, ....), thus I think size of a few hundred would be enough.
In your comment you said you will try with 336x252, I think it can be a good start.
Of course you will need some experimentation in this regard. Maybe you can try to train some models in a few epoches with different image sizes, to see the best one, and keep it to train the model further !


How can I build an AI with NLP that read stories I want to do an NLP project but I don't know if it's doable or not as I have no experience or knowledge in NLP or ML yet.

The idea is as follows: Let's say we have a story (in the text) that has 10 characters. Can we define them, their characteristics, whole sentences they said, and then analyze emotions within those sentences.

After that, is it possible to generate an audio version of the story where: the text, in general, is narrated by one voice, each individual character's sentences are read in a different voice generated specifically for that character. Finally is it possible to make the tones of the characters voices change depending on the emotions detected in their sentences?

This is quite an ambitious project, and IMHO well beyond the scope of what a single individual can do (within a reasonable time span) at present.

You need to first analyse the story text to identify the characters. This can already be quite a tricky task, as pronouns and other reference expressions are generally used to make a text less monotonous. If a character is referred to by name, say Jane, then you can assume that a follow-up the young woman will refer to her and not a male character mentioned in the same paragraph. But what about the young scientist? Such expressions can be very opaque, and you'd need a lot of world-knowledge to decode them correctly, as they can refer to any distinctive attribute of the character.

Identifying speech is a bit easier, unless you're talking about indirect speech. Jane was thinking aloud. She wasn't going to be able to do that. It was too hard. -- is that speech or not? Compare to Jane was thinking aloud: "I am not going to be able to do that. It is too hard.", which is the direct speech equivalent. Also, unless you're dealing with a play, most of the text will probably not be speech. For the audio version you will probably only want to deal with direct speech, which is usually (but not always) indicated by quote marks.

Analysing emotions seems to be comparatively easy if you have reached this stage, though if it is just based on keywords in the speech it probably won't be very accurate. If you can assign any descriptive statements to characters, that might be more successful, though by no means trivial.

Generating the text as audio should be straight forward. Most operating systems nowadays have speech synthesis integrated, and you can generally choose different voices, so if your text is marked up properly with which voice should speak which part it would be trivial.

To summarise: The NLP part is the hardest bit of it. As has been mentioned in the comments already, I don't think it's a problem that machine learning can help with, and I would stick to traditional methods of parsing the text into a structural representation and then applying rules to identify the bits you are interested in. The recognition of emotion might be a subtask that is suitable for ML, but in the past I have only applied pattern matching to similar tasks, so I can't really say much about that.

From my own experience in text analysis I would think that you might be able to get decent results with a few simple heuristics, but those will likely fail when it becomes a bit more complicated. A lot hinges on the type of story: children's fairy tales might be easier than War and Peace in that respect.


What is the difference between same convolution and full convolution in terms of feature map size? In valid convolution, the size of the output shrinks at each layer. So after some point of time additional layers cannot meaningfully performs convolution. For this reason, same convolution is introduced, where where the size of the output remains intact. This is achieved by padding with enough number of zeroes at the borders of input image.
What happens to the size of output feature map in case of full convolution?
If it remains intact then what is the difference between same convolution and full convolution?

What happens to the size of output feature map in case of full convolution?

It increases.

First one is valid padding: the blue square is not padded, so the green square is smaller.  Third one is same padding: the blue square is padded just enough so that the green square is the same size.  Fourth one is full padding: the blue square is padded as much as possible for that size of filter, so the green square is larger.
From here.


Is existence and uniqueness of state-value function at $\gamma < 1$ theoretical? Consider the following statement from 4.1 Policy Evaluation of the first edition of Sutton and Barto's book.

The existence and uniqueness of $V^{\pi}$ are guaranteed as long as
either $\gamma  < 1$ or eventual termination is guaranteed from all
states under the policy $\pi$.

I have doubt on the first condition $\gamma < 1$. If $\gamma < 1$ then it makes our task easy in a way that the $\gamma^k$ becomes zero for sufficiently higher $k$ and it is totally based on the hardware architecture. But, in theory, $\gamma^k$ can never be zero. It may approach zero.
In this context, how can the condition $\gamma < 1$ assure the existence and uniqueness of $V^{\pi}$ theoretically?

In essence, your question is about convergence of infinite series. The mathematical discipline that studies such series is hundreds (if not thousands) years old an has nothing to do with "hardware architecture".
A basic example of an infinite series is the geometric series:
$$ S = 1 + \gamma +  \gamma^2 + \gamma^3 + \dots$$
Note that the series is infinite - no one says that $\gamma^k$ "becomes zero at sufficiently large $k$". To compute the infinite sum we represent it as a limit of partial sums:
$$ S = \lim_{k\to\infty} ( 1 + \gamma +  \gamma^2 + \gamma^3 + \dots + \gamma^k)$$
The partial sum is called the geometric progression and has a well-known expression for it:
$$ 1 + \gamma +  \gamma^2 + \gamma^3 + \dots + \gamma^k = \frac{1-\gamma^{k-1}}{1-\gamma}$$
As a result, the sum of our geometric series is:
$$ S = \lim_{k\to\infty} \frac{1-\gamma^{k-1}}{1-\gamma} = \frac{1}{1-\gamma}\quad\text{if}\;|\gamma| < 1$$
In Reinforcement Learning we are dealing with similar infinite series. In partcular, when we are talking about returns:
$$R_t = r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + \dots = \sum_{k=0}^\infty\gamma^kr_{t+k+1}$$
The convergence of this is acutally discussed in the  Sutton and Barto's book:

If $\gamma < 1$, the infinite sum has a finite value as long as the reward sequence $\{r_k\}$ is bounded.

I guess, the simplest way to prove that would be to use the direct comparison test. Having $\{r_k\}$ bounded means that there exist such number $C$ so that $|r_k| \leq C$ for all $k$. So we've got the dominating series for the return :
$$ \left|\gamma^kr_{t+k+1}\right| \leq C |\gamma^k|$$
And, since the $\gamma^k$ series absolutely converges for $0 \leq \gamma < 1$, so does $\gamma^kr_{t+k+1}$ series.


What is the significance of weights in a feedforward neural network? In a feedforward neural network the inputs are fed directly to the outputs via a series of weights.

What purpose do the weights serve and how are they significant in this neural network?

You described a single-layer feedforward network. They can have multiple layers. The significance of the weights is that they make a linear transformation from the output of the previous layer and hand it to the node they are going to. To say it more simplistically, they specify how important (and in what way: negative or positive) is the activation of node they are coming from to activating the node they are going to.

In your example, since there is only one layer (a row of input nodes and a row of output nodes) it is easy to explain what each node represents. However in multi-layer feedforward networks they can become abstract representations which makes it difficult to explain them and therefore explain what the weights that come to them or go out of them represent.

Another way of thinking about it is that they describe hyperplanes in the space of the output of the previous node layer. If each output from the previous layer represents a point in space, a hyperplane decides which part of the space should give a positive value to the plane's corresponding node in the next layer and which part should give a negative input to it. It actually cuts that space into two halves. If you consider the input space of a multi-layer feedforward network, the weights of the first layer parametrize hyperplanes, however in the next layers they can represent non-linear surfaces in the input space.


How do I determine the most appropriate classifier for a certain problem? Consider a Bayesian classifier used in spam e-mail filtering. It converts an e-mail to a vector, most of the time using the bag-of-words method. Although it learns first before getting employed, it can be made to work as an online system, i.e. it can be used to filter and learn from examples even after deployment.

Now, on the other hand, now comes the perceptron. It calculates a mean vector of spam and not spam, and then classifies them into the appropriate categories. The model adjusts the mean vectors each time it makes mistakes.

Now, comes neural nets, they too are capable of taking a vector-like bag of words or image pixels of dogs and cats and classify them into yes or no.

So, while designing and implementing them into the system, how to determine which one of the methods (Bayesian classifier, perceptron or neural network) is the most appropriate for a given situation or task? One factor to consider is the time complexity (or speed), but what are other factors, and how to rank them?

This is one of the main skills that separates someone with a deep understanding of, and experience in, machine learning learning, from a neophyte. There are several approaches:

Try several methods, perhaps with automated hyperparameter optimization, and see if there's a big difference in typical model quality. This is pretty common if you don't have a lot of experience, but also something experts may try in a more targeted way.
Visualize the shape of your problem, perhaps by using a dimensionality reduction technique like PCA or tSNE, or maybe an auto-encoder. If you compress the data to 2d, are there clear linear patterns? Maybe try a linear model like logistic regression. Are there several distinct groups? How many lines would you need to draw to separate them? If it's a lot, maybe you're going to need a very non-linear model. If it's just a few, maybe a small multi-layer perceptron can help. If there are spiral bands or circular shapes, maybe and SVM with a non-linear kernel. Knowing how to translate the visualizations into intuition about the kinds of model that can help is an advanced skill. You need to understand what shapes of patterns each kind of model can learn to fit, and how these do or do not translate into a higher dimensional space.
Read the literature. If you're working in computer vision, you should try a CNN. Why? Well, everyone else is using them. They work great on most computer vision problems. They hold most competition records. It'd be silly not to try them. If you're working on spam classification though, CNNs are a bad choice. People use compression classifiers, Bayesian models, and sometimes multi-stage models to build complex features. If you look at the recent literature in your area, you can tell what to use. Reading and understanding this literature well enough to interpret it usually requires more advanced scientific training in ML and/or the application area.

As a final note, the No Free Lunch theorems for ML tell us that this is always going to be an art (at least, that's one interpretation experts argue for), and not a science, so the more practice you get with it, the better you'll become.


How do I calculate the policy in the Proximal Policy Optimization algorithm? I recently watched the video on Proximal Policy Optimization (PPO). Now, I want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are calculated.

In the paper Proximal Policy Optimization Algorithms (at page 5), the pseudocode of the PPO algorithm is shown:

It says to run $\pi_{\theta_{\text{old}}}$, compute advantage estimates and optimize the objective. But how can we calculate $\pi_\theta$ for the objective ratio, since we have not updated the $\pi_{\theta_{\text{old}}}$ yet?

You're right, the first time you run it the two policies ($\pi_{\theta old}$ and $\pi_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\theta)={\pi_\theta(a|s)\over\pi_{\theta old}(a|s)}$) by the advantage (so $loss=-r_t(\theta)A_t$).

However, with PPO you run multiple epochs of training on the same data. So after your first update you do the whole thing again (without exploring the environment any more) and this time $\pi_{\theta old}$ is different to $\pi_\theta$.

Here's a great explanation of the algorithm: https://stackoverflow.com/questions/46422845/what-is-the-way-to-understand-proximal-policy-optimization-algorithm-in-rl


Can A3C update the policy / critic on a local machine without needing to copy? To make A2C into A3C you make it asynchronous. From what I understand the 'correct' way to do that is to thread off workers with a copy of the policy and critic, and then return the state/action/reward tuples to the main thread, which then performs the gradients updates on the main policy and critic, and then repeat the process.

I understand why the copying would be necessary in a distributed environment, but if I were to always run it locally then could I just perform the updates on a global variable of the policy and critic, i.e. avoid the need for copying? Provided the concurrency of the updates was handled correctly, would that be fine?

You might want to check out this paper relating to Phizaz's comment: Asynchronous Methods for Deep Reinforcement Learning (specifically search for Hogwild).


Is the Mask Needed for Masked Self-Attention During Inference with GPT-2 My understanding is that masked self-attention is necessary during training of GPT-2, as otherwise it would be able to directly see the correct next output at each iteration. My question is whether the attention mask is necessary, or even possible, during inference. As GPT-2 will only be producing one token at a time, it doesn't make sense to mask out future tokens that haven't been inferred yet.

As a follow up, does this mean that during inference GPT-2 has a "sample" dimension that is always equal to the time iteration count + the prompt length, i.e. during the first iteration, if the prompt was just the "START" token, then only a vector of length 768 (or whatever the embedding size is) will flow through the network?

Answer to Q1) If sampling for next token do you need to apply mask during inference?

Yes you do! The models ability to transfer information across positions was trained in this manner, and changing it up will have unpredictable consequences. Let my try to give an example:

Tokens: 1:sally, 2:sold, 3:seashells, 4:on, 5:the, 6:____
In the above you are trying to predict 6 from {1:5}

Denote $n^{(m)}$ as the set of tokens the $n^{th}$ positional embedding has info from at the $m^{th}$ layer.

In both cases we see that $n^{(0)} = \{n\} \ \  \forall n$. Now though with a mask we get $n^{(i)} = \{k\}_{k\leq n} \ \  \forall n \ \ s.t. \ \ i \geq 1$ but without we see $n^{(i)} = \{k\}_{k \in [1:N]} \ \  \forall n$. This difference means at the final layer the mebeddings going in will differ completely, and unless we train for such an approach it will cause  error

Answer to Q2) What is the sample dimension?

It took me a couple reads to understand what youre asking for but I think I understand. The sample at each step is drawn from a distribution where its logits are linearly associated to a single embedding of dimension $d_{(model)}$ therefore that is our upper bound: $dim(sample) \leq d_{(model)}$ which in the example you gave is 768.


What is the difference between an observation and a state in reinforcement learning? I'm studying reinforcement learning. It seems that "state" and "observation" mean exactly the same thing. They both capture the current state of the game.

Is there a difference between the two terms? Is the observation maybe the state after the action has been taken?

Sometimes observation and state overlap completely, which is convenient. However, there is no reason to expect it in all cases, and that's where interesting problems occur.

Reinforcement learning theory is based on Markov Decision Processes. This leads to a formal definition of state. Most importantly, the state must have the Markov property. Which means that for RL to work according to theory, that knowing the state means that you know everything knowable that could determine the response of the environment to a specific action. Everything that remains must be purely stochastic and unknowable in principle until after the action is resolved.

Systems like deterministic or probability-driven games, and computer-controlled simulations can be designed to have easily observable states that have this property. Games with this trait are often called "games of perfect information", although you may have unknown information, provided it is revealed in a purely stochastic manner.

In practice, real world interactions contain far too much detail for any observation to be a true state with the Markov property. For instance, consider the inverted pendulum environment, a classic RL toy problem. A real inverted pendulum would behave differently depending on its temperature, which could vary along its length. The joint and actuators might be sticky. Rotations and movement will alter temperature and friction, etc. However, a RL agent will typically only consider current motion and position of the trolley and pendulum. In this case, the observation of 4 traits is usually good enough, and a state based on this almost has the Markov property.

There are also problems where observations are not enough to make usable state data for a RL system. The Deep Mind Atari DQN paper had examples of a couple of these. The first example is that a single frame lost data about motion. This could be addressed by taking four consecutive frames and combining them to make a single state. It could be argued that each frame is an observation, and that four observations had to be combined in order to construct a more useful state (although this could be put aside as just semantics).

The second example in Atari DQN is that the pixel observations did not include data that the game was tracking but that was not visible on screen. Games with large scrolling maps are a weakness of the Atari-playing DQN, because its state has no memory of screens other than the four used for movement. An example of such a game, where Deep Mind's player did much worse than a human player is Montezuma's Revenge, where to progress it is necessary to remember some off-screen locations.

There are ways to address knowledge that there is unobserved but relevant state in a problem. The general framework for describing the problem is Partially Observable Markov Decision Processes (POMDPs). Workable solutions include adding explicit memory or "belief state" to the state representation, or using a system such as RNN in order to internalise the learning of a state representation driven by a sequence of observations.


Which ANN can solve for y = x * x + b? I am new to ANN. I am trying out several 'simple' algorithms to see what ANN can (or cannot) be used for and how. I played around with Conv2d once and had it recognize images succesfully. Now I am looking into trend line analyses. I have succeeded in training a network where it solved for linear equations. Now I am trying to see if it can be trained to solve for y in the formula y = b + x * x.

No matter what parameters I change, or number of Dense layers, I get high values for loss and val_loss, and the predictions are incorrect.

Is it possible to solve for this equation, and with what network? If it is not possible, why not? I am not looking to solve a practical problem, rather build up understanding and intuition about ANNs.

See the code I tried with below

#region Imports
from __future__ import absolute_import, division, print_function, unicode_literals
import math
import numpy as np
import tensorflow as tf
from tensorflow.keras import models, optimizers
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Activation, Dropout, Flatten, Dense, Lambda
import tensorflow.keras.backend as K
#endregion

#region Constants
learningRate = 0.01
epochs: int = 1000
batch_size = None
trainingValidationFactor = 0.75
nrOfSamples = 100
activation = None
#endregion

#region Function definitions
def CreateNetwork(inputDimension):
  model = Sequential()
  model.add(Dense(2, input_dim=2, activation=activation))
  model.add(Dense(64, use_bias=True, activation=activation))
  model.add(Dense(32, use_bias=True, activation=activation))
  model.add(Dense(1))
  adam = optimizers.Adam(learning_rate=learningRate)
  # sgd = optimizers.SGD(lr=learningRate, decay=1e-6, momentum=0.9, nesterov=True)
  # adamax = optimizers.Adamax(learning_rate=learningRate)
  model.compile(loss='mse', optimizer=adam)
  return model

def SplitDataForValidation(factor, data, labels):
  upperBoundary = int(len(data) * factor)

  trainingData = data[:upperBoundary]
  trainingLabels = labels[:upperBoundary]

  validationData = data[upperBoundary:]
  validationLabels = labels[upperBoundary:]
  return ((trainingData, trainingLabels), (validationData, validationLabels))

def Train(network, training, validation):
  trainingData, trainingLabels = training
  history = network.fit(
    trainingData
    ,trainingLabels
    ,validation_data=validation
    ,epochs=epochs
    ,batch_size=batch_size
  )

  return history

def subtractMean(data):
  mean = np.mean(data)
  data -= mean
  return mean

def rescale(data):
  max = np.amax(data)
  factor = 1 / max
  data *= factor
  return factor

def Normalize(data, labels):
  dataScaleFactor = rescale(data)
  dataMean = subtractMean(data)

  labels *= dataScaleFactor
  labelsMean = np.mean(labels)
  labels -= labelsMean

def Randomize(data, labels):
  rng_state = np.random.get_state()
  np.random.shuffle(data)
  np.random.set_state(rng_state)
  np.random.shuffle(labels)

def CreateTestData(nrOfSamples):
  data = np.zeros(shape=(nrOfSamples,2))
  labels = np.zeros(nrOfSamples)

  for i in range(nrOfSamples):
    for j in range(2):
      randomInt = np.random.randint(1, 5)
      data[i, j] = (randomInt * i) + 10
    labels[i] = data[i, 0] + math.pow(data[i, 1], 2)

  Randomize(data, labels)
  return (data, labels)
#endregion

allData, allLabels = CreateTestData(nrOfSamples)
Normalize(allData, allLabels)
training, validation = SplitDataForValidation(trainingValidationFactor, allData, allLabels)

inputDimension = np.size(allData, 1)
network = CreateNetwork(inputDimension)

history = Train(network, training, validation)

prediction = network.predict([
  [2, 2], # Should be 2 + 2 * 2 = 6
  [4, 7], # Should be 4 + 7 * 7 = 53
  [23, 56], # Should be 23 + 56 * 56 = 3159
  [128,256] # Should be 128 + 256 * 256 = 65664
])
print(str(prediction))

$f(x) = x^2 + b$ is a polynomial (more precisely, a parabola) so it is continuous, thus, a neural network (with at least one hidden layer) should be able to approximate that function (given the universal approximation theorem).

After a very quick look at your code, I noticed you aren't using an activation function for your dense layers (i.e. your activation function is None). Try to use e.g. ReLU.


What does the notation $p_t = \text{max}_{i<t} p_i$ mean in algorithm 1 of the prioritized experience replay paper? I am having a hard time converting line 6 of the prioritized experience replay algorithm from the original paper into plain English (see below):

I understand that new transitions (not visited before) are given maximal priority. On line 6 this would be done for every transition in an initial pass since the history is initialized as empty on line 2.
I’m having trouble with the notation $p_t = \text{max}_{i<t}  p_i$. Can someone please state this in plain English? If $t$ = 4 for example, then $p_t$ = 4? How is this equal to max$_{i<t}  p_i$.
It seems in my contrived example here, max$_{i<t}  p_i$ would be 3. I must be misreading this notation.

From my interpretation what it means is that $p_t$ is the priority value associated with each transition and $p_t = max_{i<t} p_i $ means that the priority of transition number $t$ will be the maximum between the values of the priorities of the previous elements.

Example: since $p_1$ is initialized to $1$, all the new experiences will be too:
\begin{equation}
p_2 = max\{p_1\} = 1,
\end{equation}

\begin{equation}
p_3 = max\{p_1,p_2\} = 1,
\end{equation}

\begin{equation}
p_4 = max\{p_1,p_2,p_3\} = 1.
\end{equation}


Are neural networks statistical models? By reading the abstract of Neural Networks and Statistical Models paper it would seem that ANNs are statistical models.

In contrast Machine Learning is not just glorified Statistics.

I am looking for a more concise/summarized answer with focus on ANNs.

According to Wikipedia:

  A statistical model is a mathematical model that embodies a set of statistical assumptions concerning the generation of sample data (and similar data from a larger population). A statistical model represents, often in considerably idealized form, the data-generating process.

Answer to your question:

To build any neural network model we assume the train, test and validation data are coming from a probability distribution. So, if you produce a neural network model based on statistical data then the network is a statistical model.

Moreover, neural networks' cost function is generally a parametric model and parametric modes are statistical models.

Please look at Goodfellow's Deep Learning book chapter Deep Feedforward Networks page 174 and 175.

From Goodfellow's book

  Fortunately, the cost functions for neural networks are more or less the same as those for other parametric models, such as linear models. In most cases, our parametric model deﬁnes a distribution $p(y \mid x; \theta)$ and we simply use the principle of maximum likelihood.

In conclusion, ANNs (e. g. MLP, CNN, etc.) are statistical models


Algorithms and strategies to help judges rule cases I'm a Rails developer with a lot of web experience, but none (still) in AI.

I'm working in a web text editor that judges use to writing their sentences.

The goal is to start to use AI to help the judge rule the case, either based on his own previous rulings, either based on his colleagues rulings.

The judge would provide the text for the plaintiffs and defendants petitions, and based on these two inputs the system would recommend previous rulings that apply for the case.

I already have a considerable dataset of judges rulings inside the database, and they can be easily attached to the plaintiffs and defendants petitions for training (so this plaintiff petition + this defendant petition = this ruling).

This is specially challenging because the complaints can contain different subjects combined into the same petition; but the fact is that many offices use the same standardized petitions, as the defendants do as well, so I think the system can have a great chance of prediction success.

What algorithms or strategies should I start studying to tackle this problem?

Any similar articles, white papers or repositories that could help in my goal?

Genuine success in this area would be beyond the state-of-the-art in research, since it likely requires analogising from relational knowledge extracted from text. In recent years, techniques for working with natural language have tended to be statistical, and are therefore somewhat deficient in this respect. You could look at 'bag of words'/latent semantic analysis approaches, but they are likely to generate many false positives unless a lot of ad hoc conditions are added manually. More recent work on 'treenets' (paper here) is more structurally informed, but is still a relatively new area.


What is the purpose of a Neural Network in Reinforcement Learning when we have a Q-learning update rule? I'm confused as to the purpose of training a neural network (NN) for reinforcement learning (RL) tasks such as Gridworld. In RL tasks, namely q-learning, we have a q-learning update rule, which is designed to take some state and action and compute the value of that state-action pair.
Performing this process several times will eventually produce a table of states and what action will likely lead to a high reward.
In RL examples, I've seen them train a neural network to output q-values and a loss function like MSE to compute the loss between the q-learning update rule q-value and the NN's q value.
So:
(a) Q-learning update rule-> outputs target Q-values
(b) NN -> outputs Q values
MSE to compute the loss between (a) and (b)
So, given we already know what the target Q-value is from a, why do we need to train a NN?

I don't think people generally do use neural nets for grid world. As long as the state and action spaces are small enough, you should be able to store Q values in a table like you suggested. Neural nets come in handy when the state space is very large (or even continuous), so you can't afford to store a table of Q values. Also, neural nets have the ability to generalize across "similar" states -- for instance, if two states are very similar the neural net would likely produce similar values for those states, whereas a tabular implementation might not have seen enough data to accurately estimate the Q values of both.


Why do all nodes in a GP tree need to be the same type? Context: I'm a complete beginner to evolutionary algorithms and genetic algorithms and programming. I'm currently taking a course about genetic algorithms and genetic programming.

One of the concepts introduced in the course is "closure," the idea that - with an expression tree representing a genetic program that we're evolving - all nodes in the tree need to be the same type. As a practical example, the lecturer mentions that implementing greater_than(a, b) for two integers a and b can't return a boolean like true or false (it can return, say, 0 and 1 instead).

What he didn't explain is why the entire tree needs to match in all operators. It seems to me that this requirement would result in the entire tree (representing your evolved program) being composed of nodes that all return the same type (say, integer).

In tree-based genetic programming (TGP), you have a tree that represents a program or a function. The nodes in this tree are functions, while the edges represent the interactions between these functions. The leaves of this tree are the inputs (or random numbers) that you pass to this function. The incoming edges into a node represent the inputs, while the outgoing edges represent the output of the associated function.
So, for example, consider the following function
\begin{align}
f(x)
&= \sin(x^2) \\
&= \sin(y)
\tag{1}\label{1},
\end{align}
where $y = g(x)$ (note that this is just a change of variable in order to illustrate the corresponding tree more clearly below!).
The function $f$ will be represented by the following tree
sin(y)
  |
 g(x)
  |
  x

So, how do we read this diagram? Essentially, we read it from the bottom-up. So, $x$ is first passed to $y = g(x) = x^2$, which then passed to $\sin(y)$. In this case, given that we are dealing with mathematical operations, you expect $x$ to be a number, because, otherwise, what would $x^2$ or $\sin(x^2)$ mean?
Let's now consider a function of 2 inputs.
\begin{align}
f(x, y)
&= x + y
\tag{2}\label{2},
\end{align}
The corresponding tree would be
  +
 / \
x   y

In this case, you naturally expect that $x$ and $y$ are also numbers. However, this may not be actually the case. Let's say that you're evolving Python functions, then x + y is well defined even if x and y are strings, i.e. that would be a concatenation operation.
So, in this sense, we naturally expected the functions (the nodes in the tree) to get parameters/arguments with the right types, but, in some cases, more types are possible for the apparently same function.
So, in general, types don't necessarily need to be enforced! It depends on the implementation of TGP.
There's a specific approach to TGP where the idea is really to ensure type-safety, i.e. strongly-typed (tree-based) genetic programming, where functions can have a different number and type for their parameters and return values, but, in that case, only functions that are consistent with the signature of the function can be "connected" with that function in the tree.
In weakly-typed GP, the types are not checked, so you could end up evolving functions of the form $f(x) = \sin(x)$, where $x$ is a string. Of course, when you execute these programs/functions, the program may crash, but this is a different story, which you need to take care of (i.e. you need to ensure that your individuals can be evaluated), if you need this flexibility. This weakly-typed approach can also be viewed as a strongly-typed approach where all functions' parameters and return values have the same type.
DEAP, a well-known Python library for GAs and GP, provides both weakly and strongly-typed GP, so, if you are familiar with Python, you may want to start from it.
To conclude and answer your question more directly, it's not true that in TGP you need all functions to have the same type for all parameters and return values.


Why is my Soft Actor-Critic's policy and value function losses not converging? I'm trying to implement a soft actor-critic algorithm for financial data (stock prices), but I have trouble with losses: no matter what combination of hyper-parameters I enter, they are not converging, and basically it caused bad reward return as well. It sounds like the agent is not learning at all.
I already tried to tune some hyperparameters (learning rate for each network + number of hidden layers), but I always get similar results.
The two plots below represent the losses of my policy and one of the value functions during the last episode of training.

My question is, would it be related to the data itself (nature of data) or is it something related to the logic of the code?

I would say it is the nature of data. Generally speaking, you are trying to predict a random sequence, especially if you use the history data as an input and try to get the future value as an output.


How to teach a model-based reflex agent for doing some task using machine learning methods? I would like to know how to teach an agent for performing prediction of the severity of disease and also for alerting patients using machine learning methods.

I found the model-based reflex agent can be used in medical diagnosis in some literature.

May I know which architecture will be good, to make such an agent?

The question is really broad---as stated by @thecomplexitytheorist---so difficult to give a meaningful answer. The following is about a clarification about the problem, and some directions.

A model-based reflex agent is a blueprint describing the key components necessary to build that agent. It is an abstract architecture to guide the creation of concrete agents. Whether software or hardware, the target agent should have a bit serving as "sensors" (a HW camera, or a software API"), "actuators", memory, etc. So if you aim at a HW agent, you need to decide what compenents, and collect them. A SW agent would be the definition of a software architecture, or the use of some modelling framework.

The Machine Learning part is about endowing the agent with some skill. It is the "smarts" programming part, what the "brains" will do in the instance of the blueprint. Please note that it does not have to be Machine Learning. It could be another "style" of programming, such as a rule-based system, or a plain hand-crafted program.

How to teach such agent is currently an open question. First steps usually start with drawing the agent as a black box, its inputs (e.g. symptom data), its desired output (diagnosis and alert). Then we detail the black box in terms of what sensors the agent needs to process the input, what actuators for the output, and what it needs to learn and memorize. Depending on the available input data and diagnosis/alert output system (email?), the next stage aims at refining step by step, until a good idea of what needs be implemented emerges.

At this point, all pieces are in place, except the internal model---the piece that really pertains to Machine Learning (if you choose ML for that). As the input data is available, and the output format decided, the "final" stage (before implementation) is to define the model. It really depends on the actual data and the goal (here we assume prediction). Labelled data (we can teach with input and output, as we already know them) usually leads to supervised learning. Unlabelled data (we can teach with the input only) leads to unsupervised learning or reinforcement learning. Once one understands the situation, we can choose some algorithm like SVM, decision trees, neural networks, etc. Note that the final decisions (before implementation) require studying the data beforehand (Is it regular? Are there missing bits? Is it in a format useful for ML? Etc.) to make appropriate choices.

Final note: The question may get closed on this site, because it is too broad. Way more useful to ask narrow questions, with a clear answer or set of answers. As you see here, this long answer is just the tip of how to teach an agent (I could not sleep anyway). And no implementation yet. In fact, all this could be summed up as "typical system development, with an ML component". Two guiding principles: (1) isolate the ML component(s), so specialists can dig and make the best of them, and (2) keep it simple.


How much can the addition of new features improve the performance? How much can the addition of new features improve the performance of the model during the optimization process?

Let's say I have a total of 10 features. Suppose I start the optimisation process using only 3 features. Can the addition of the 7 remaining ones improve the performance of the model (which, you can assume, might already be quite high).

It depends on the used network as well as the feeding mechanism but let's give an example;

When working with LSTM, giving the time data (as an integer sequence) in addition to the time-series data(coming from features) dramatically increases the performance of the network.

[$X_{0}$,$X_{1}$, ...] $\rightarrow$ [[$X_{0}$,$t_{0}$],$[X_{1}$,$t_{1}$], ...]

If you go and look for the kaggle competition winner's notebooks, they do also create additional features based on the featured data.

Let's assume that the performance is already quite high on the three features so that you can predict those three features with high reliability.
It would only make sense to increase the number of features if you would like to predict additional features!


How should we understand the evaluation metric, AUC, in link prediction problems? In link prediction problems, there are only known edges and nodes.

If there is a known edge in the node pair, the node pair is regarded as a positive sample. Except for those node pairs whose edges are known, There may exist unobserved edges in some node pairs or there really doesn't exist edges in some node pairs. Our target is to predict potential links in those candidate node pairs.

The node pair where there exist known edge is regarded as a positive sample. So the node pair whose edge are not observed can neither be regarded as a positive example, nor a negative example.

So I think link prediction problem is a semi-supervised problem. However, I find that many papers, for example, GRTR: Drug-Disease Association Prediction Based on Graph Regularized Transductive Regression on Heterogeneous Network, use AUC(Area Under the ROC Curve, a metric for supervised problems) as the metric.

How should we understand such behavior? What's the reason?

The paper you link to in the question is paywalled for me, so this answer may not be specific enough, but we'll make do.

Fundamentally, AUC is about measuring a classifier's robustness. The intuition behind the measure is that if we have classifier that outputs a score for an example, rather than a class label alone, we can choose to interpret that score in ways that trade detection rate (true positives) for false positive rate.

As a very simple example, imagine that we have a classifier that outputs a probability that an example pair is a link. By default we might chose to interpret this a probability of more than 0.5 as a link being present. Suppose this produces a certain true positive and false positive rate. Now let's change our mind. It's suddenly very important that we detect all the links, even if we get a lot of false positives. We choose to interpret any probability value > 0 as being a link. Our true positive rate will spike, but so will our false positive rate. Finally, if we care only about minimizing the prediction of fake links, we could chose to interpret only probability values that are >0.99 as being links. Our true positive rate will go way down, but our false positive rate will too.

AUC quantifies how rapidly the false positive rate increases as we increase the true positive rate by changing our interpretation of the scores the classifier outputs. Better classifiers will generally be able to increase their true positive rates quickly without introducing many false positives, while worse classifiers will not. An AUC of 1.0 indicates that we can achieve 100% true positive rate without increasing the false positive rate at all. An AUC of 0.0 indicates that increasing the true positive rate by 1% will always result in an increase of 1% in the false positive rate (i.e. the classifier is essentially just as good as a random guess without looking at the input).

In your link prediction problem, we cannot directly measure the false positive rate. However, if we treat all non-links as negative examples when we compute this measure, a classifier with a higher AUC will is one that can more rapidly separate positive examples from non-positive ones as you increase your sensitivity. Even though we aren't sure which examples are false positives and which are unseen links, a classifier with a high AUC is one where we would be more likely to trust that the non-links it labels as links are actually links we can't see.


Can I prune the tree if alpha-beta pruning finds 10 to the left of the root node? I have an AI class this semester. For our exam, we also cover alpha-beta pruning.
I found an old example, where I think we can stop already earlier.
Here is a picture of it.

I think, because $X$ wants to maximize his win, and finds 10, he knows that he cannot get better. Therefore he cuts and puts 10.
I marked my "improvement" with red.

The vanilla Alpha-Beta Pruning algorithm as it has been taught to you in class does not assume any domain knowledge / knowledge about the game / knowledge about the tree it is searching. Therefore, if it immediately finds a score of 10 directly to the left of the root node, it can not prune yet, because... maybe there's a score of 20 somewhere else in the tree?

In your description, you seem to assume prior knowledge that the maximum score that can possibly be obtained in the entire game is 10. This is not typically assumed to be available knowledge ahead of time, at least not in the original formulation of Alpha-Beta. That is why in pseudocode of the algorithm (for example on wikipedia), they initialize alpha and beta at minus and plus infinity.

You are right though. If you have better estimates of the upper and lower bounds on scores that can every be achieved throughout the entire game, you can use better initializations of alpha and beta than minus and plus infinity. alpha and beta would have to be -11 and 11 if the true bounds are -10 and 10. I suspect you'd also have to make some additional changes to the typical pseudocode to actually make use of this though.

In more complex games than Tic Tac Toe, you're often not going to have very useful knowledge of better bounds though. Typically you can't afford to search all the way to terminal game states, you'll have to stop the search earlier and use heuristic evaluation functions to estimate scores. If you do that, you'll often want to assign really high constants (like +/- 1Million) to actual terminal nodes (wins and losses), so that all your heuristic scores for non-terminal nodes can be somewhere in between. If you have such a large range of possible scores, your idea stops having much value.


Is the target assumed to be a noisy version of the output of the model in machine learning? I wonder if the following equation (you can find it in almost every ML book) refers to a general assumption that we make when using machine learning:
$$y = f(x)+\epsilon,$$
where $y$ is our output, $f$ is e.g. a neural network and $\epsilon$ is an independent noise term.
Does this mean that we assume the $y$'s contained in our training data set come from a noised version of our network output?

Not necessarily. The neural network (or whatever else you use) is a model of what you are trying to do, and usually models are not able to perfectly model reality, as it is too complex. A noise term is generally used to represent that, ie the imperfection of the model's relationship with the actual world.