Are Siri and Cortana AI programs? Siri and Cortana communicate pretty much like humans. Unlike Google Now, which mainly gives us search results when asked some questions (not setting alarms or reminders), Siri and Cortana provide us with an answer, in the same way that a person would do.
So, are they actual AI programs or not?
(By "question" I don't mean any academic-related question or asking routes/ temperature, but rather opinion based question).

Siri and Cortana are AI to some extent. The usual label is "weak AI" (also called "narrow" or "soft" AI). It turns out the Wikipedia article on Weak AI explicitly refers to Siri:

Siri is a good example of narrow intelligence. Siri operates within a limited pre-defined range, there is no genuine intelligence, no self-awareness, no life despite being a sophisticated example of weak AI. In Forbes (2011), Ted Greenwald wrote: "The iPhone/Siri marriage represents the arrival of hybrid AI, combining several narrow AI techniques plus access to massive data in the cloud." AI researcher Ben Goertzel, on his blog in 2010, stated Siri was "VERY narrow and brittle" evidenced by annoying results if you ask questions outside the limits of the application.

Important to note that "mixing" weak AIs does not make a "stronger" AI, by some arguments (see Searle's Chinese Room argument), but there is no consensus on this topic yet.


Why is Short-Time Fourier Transform used for preprocessing audio samples? I've been told this is how I should be preprocessing audio samples, but what information does this method actually give me? What are the alternatives and why shouldn't I use them.

Fourier transform is used to transform audio data to get more information (features).

For example, raw audio data usually represented by a one-dimensional array, x[n], which has a length n (number of samples). x[i] is an amplitude value of the i-th sample point.

Using the Fourier transform, your audio data will be represented as a two-dimensional array. Now, x[i] is a not a single value of amplitude, but a list of frequencies which compose original value at the i-th frame (a frame consists of a few samples).

See the image below (from wikipedia), the red graph is an original value of n samples before transformed, and the blue graph is a transformed value of one frame.


Are Q-learning and SARSA the same when action selection is greedy? I'm currently studying reinforcement learning and I'm having difficulties with question 6.12 in Sutton and Barto's book.

Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as SARSA? Will they make exactly the same action selections and weight updates?

I think it's true, because the main difference between the two is when the agent explores, and following the greedy policy it never explores, but I am not sure.

If we write the pseudo-code for the SARSA algorithm we first initialise our hyper-parameters etc. and then initialise $S_t$, which we use to choose $A_t$ from our policy $\pi(a|s)$. Then for each $t$ in the episode we do the following:

Take action $A_t$ and observe $R_{t+1}$, $S_{t+1}$
Choose $A_{t+1}$ using $S_{t+1}$ in our policy
$Q(S_t, A_t) = Q(S_t, A_t) + \alpha [R_{t+1} + \gamma Q(S_{t+1},A_{t+1}) - Q(S_t, A_t)]$

Now, in Q-learning we replace $Q(S_{t+1},A_{t+1})$ in line 3 with $\max_aQ(S_{t+1},a)$. Recall that in SARSA we chose our $A_{t+1}$ using our policy $\pi$ - if our policy is greedy with respect to the action value function then this simply means the policy is $\pi(a|s) = \text{argmax}_aQ(s,a)$ which is exactly how we choose our weight update in Q-learning.
To answer the question - no, they are not always the same algorithm.
Consider where we transition from $s$ to $s'$ where $s'=s$. I will outline the updates for SARSA and Q-learning indexing the $Q$ functions with $t$ to demonstrate the difference.
For each case, I will assume we are at the start of the episode, as this is the easiest way to illustrate the difference. Note that actions denoted by $A_i$ are for actions taken explicitly in the environment -- in the Q-Learning update the $\max$ action that is chosen for the update is not executed in the environment, the action taken in the environment is chosen by the policy after the update has happened.
SARSA

We initialise $S_0 = s$ and choose $A_0 = \text{argmax}_a Q_0(s,a)$
Take action $A_0$ and observe $R_{1}$ and $S_{1} = s' = s$.
Choose action $A_{1} = \text{argmax}_aQ_{0}(s,a)$
$Q_{1}(S_0,A_0) = Q_0(S_0,A_0) + \alpha [R_{1} + \gamma Q_0(s,A_1) - Q_0(S_0,A_0)]$

Q-Learning

Initialise $S_0 = s$
Choose action $A_0 = \text{argmax}_aQ_0(s,a)$, observe $R_{1}, S_{1} = s' = s$
$Q_{1}(S_0,A_0) = Q_0(S_0,A_0) + \alpha [R_{1} + \gamma \max_aQ_0(s,a) - Q_0(S_0,A_0)]$
Choose action $A_1 = \text{argmax}_aQ_1(s,a)$

As you can see the next action for the updates in SARSA (line 4) and Q-learning (line 3) are taken with respect to the same $Q$ function, but the key difference is that the actual next action taken in $Q$-learning is taken with respect to the updated $Q$-function.
The key for understanding this edge case is that when we transition into the same state, the Q-Learning update will update the Q-function before choosing $A_1$. I have indexed actions and Q-functions by the episode step - hopefully, it makes sense why I have done this for the Q-functions as, usually, this would not make sense, but, because we have two successive states that are the same, it is okay.


Are there ways to learn and practice Deep Learning without downloading and installing anything? As per subject title, are there ways to try Deep Learning without downloading and installing anything?

I'm just trying to have a feel of how this work, not really want to go through the download and install step if possible.

As I understand, I think you wish to directly try out some deep learning stuff and things like library downloading, tools downloading, and managing all these really stop you from even starting to try out deep learning experiments.If this is what you asked for:

Google Colab

I think this is the best place for you.
Anyone with a Google Drive account can sign up for Colab by heading to
 and following the listed instructions.

Since you mentioned that you just wanted to try out and practice stuffs, this would be
ideal for you.

All major Python libraries like TensorFlow, Scikit-learn, Matplotlib among many
others
are pre-installed and ready to be imported.
Built on top of Jupyter Notebook

Please have a quick look at:

https://medium.com/lean-in-women-in-tech-india/google-colab-the-beginners-guide-5ad3b417dfa

2.Microsoft Azure

The Azure free account is available to all new customers of Azure. If you have never
  tried or paid for Azure before, you’re eligible.
  Try out a student account:
   https://azure.microsoft.com/en-in/free/students/

Hope this helps, go ahead with practicing deep learning
if not please feel free to raise questions, always ready to help


What are the differences between an agent and a model? In the context of Artificial Intelligence, sometimes people use the word "agent" and sometimes use the word "model" to refer to the output of the whole "AI-process". For examples: "RL agents" and "deep learning models".
Are the two words interchangeable? If not, in what case should I use "agents" instead of "models" and vice versa?

Agent
The other answer defines an agent as a policy (as it's defined in reinforcement learning). However, although this definition is fine for most current purposes, given that currently agents are mainly used to solve video games, in the real world, an intelligent agent will also need to have a body, which Russell and Norvig call an architecture (section 2.4 of the 3rd edition of Artificial Intelligence: A Modern Approach, page 46), which should not be confused with an architecture of a model or neural network, but it's the computing device that contains the physical sensors and actuators for the agent to sense and act on the environment, respectively. So, to be more general, the agent is defined as follows

agent = body + policy (brain)

where the policy is what Russell and Norvig call the agent program, which is an implementation of the agent function.
Alternatively, it can be defined as follows

An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.

This is just another definition given by Russell and Norvig, which I also report in this answer, where I describe different types of agents. Note that these definitions are equivalent. However, in the first one, we just emphasize that we need some means to "think" (brain) and some means to "behave" (body).
These definitions are quite general, so I think people should use them, although, as I said above, sometimes people refer to an agent as just the policy.
Model
In this answer, I describe what a model is or what I like to think a model is, and how it is different from a function.
In AI, a model can refer to different but somehow related concepts.

For example, in reinforcement learning, a model typically refers to $p(s', r \mid s, a)$, i.e. the joint probability distribution over the next state $s'$ and reward $r$, given the current state $s$ and action $a$ taken in $s$.

In deep learning, a model typically refers to a neural network, which can be used to compute (or model) different functions. For example, a neural network can be used to compute/represent/model a policy, so, in this case, there would be no actual difference between a model and an agent (if defined as a policy, without a body). However, conceptually, at a higher-level, these would still be different (in the same way that biological neural networks are different from the brain).

More generally, in machine learning, a model typically refers to a system that can be changed to compute some function. Examples of models are decision trees, neural networks, linear regression models, etc. So, as I also state in the other answer, I like to think of a model as a set of functions, so, in this sense, a model would be a hypothesis class in computational learning theory. This definition is roughly consistent with $p(s', r \mid s, a)$, which can also be thought of as a (possibly infinite) set of functions, but note that a probability distribution is not exactly a set of functions.

In the context of knowledge bases, a model is an assignment to the variables, which represents a "possible world". See section 7.3, page 240, of the cited book.

There are possible other uses of the word model (both in the context of AI, e.g. in the context of planning, there's often the idea of a conceptual model, which is similar to an MDP in RL, and in other areas), but the definitions given above should be more or less widely applicable in their contexts.
What is the difference between an agent and a model?
Given that there are different possible definitions of a model depending on the context, it's not easy to briefly state what the difference between the two is.
So, here's the difference in the context of RL (and you can now find out the differences in other contexts by using the different definitions): an agent can have a model of the world, which allows it to predict e.g. the reward it will receive given its current state and some action that it decides to take. The model can allow the agent to plan. In this same context, a model could also refer to the specific system (e.g. a neural network) used to compute/represent the policy of the agent, but note that people usually refer to $p(s', r \mid s, a)$ when they use the word model in RL. See this post for more details.


Does the substituted variable/constant have to appear in the unified term? I'm checking out how to manually apply resolution on a first order predicate logic knowledge base and I'm confused about what is allowed or not in the algorithm.

Let's say that we have the following two clauses (where $A$ and $B$ are constants):

$\neg P(A, B) \vee H(A)$

$\neg L(x_1) \vee P(x_1, y_1)$

If I try to unify these two clauses by making the substitutions $\{x_1/A, y_1/B\}$ do I get $\neg L(A) \vee H(A)$ ? Is it allowed to substitute $y_1$ by $B$ even if $B$ doesn't appear in the unified clause?

Then we have the other way around where:

$\neg P(A, y_1) \vee H(A)$

$\neg L(x_1) \vee P(x_1, B)$

Can I do $\{x_1/A, B/y_1\}$ for $\neg L(A) \vee H(A)$ ?

What about the case where:

$\neg P(A, z_1) \vee H(A)$

$\neg L(x_1) \vee P(x_1, y_1)$

Can I substitute $\{x_1/A, y_1/z_1\}$ and get $\neg L(A) \vee H(A)$ ?

Finally there is also the cases where we have something like this:

$\neg P(x_2, y_2) \vee H(z_1)$

$\neg L(x_1) \vee P(x_1, y_1)$

Can we do $\{x_1/x_2, y_1/y_2\}$ to get $\neg L(x_3) \vee H(z_2)$ ?

I'm really confused about when unification succeeds once we have two clauses with a literal of the same kind (negation in one of them and not in the other one) that are a candidates for unification.

Hey I am currently studying First Order Logic right now and I think I can answer your question. Others please correct me if I am wrong.

For the first case, you can generally substitute variables with constants. Hence, you can make the substitution $\theta \leftarrow \{x_1 \leftarrow A, y_1 \leftarrow B \}$. This is used very commonly when you want to infer some query $\alpha$ from your knowledge base. $\alpha$ usually is in the form $P(A,B)$ as you have mentioned. When you unify you get $\neg L(A) \vee H(A)$ and ur substitution has to stay the same throughout the resolution algorithm. I.e, you cannot substitute $x_1$ for another constant / variable.

Regarding the second case, you cannot substitute a constant with a variable $B$ to another variable.

In general, you can substitute a variable with a constant, or another variable. You can also substitute a variable with a skolem function, Eg. $x_1 \leftarrow G(y)$. However, for skolem functions, you cannot substitute $x \leftarrow F(x)$, in which the variable names are the same.

Hope this helps !


How to calculate the size of a 3d object from an image? I am wondering how to calculate the size of a 3d object in an image without knowing the focal length of the camera but the distance from the camera to the object.

There are hundred of papers on this task some older than I am! Normally this is done by trying to form a box shape around the image than estimate the volume. This task is typically done with multiple images so the two can generate a more clear picture of the size of the object than one image alone. An object could be 'infinitely' large but its mass could be behind the surface you can see with the picture. With the height and length dimensions extracted from the image and with the distance from the camera calculating the size of said surface is fairly easy.

 Stack overflow about problem

Frustum PointNets for 3D Object Detection From RGB-D Data (2018) By Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, Leonidas J. Guibas

From contours to 3D object detection and pose estimation (2017) By Nadia Payet Sinisa Todorovic


How could we estimate the square footage of a room from an image? I wonder if it would be possible to know the size of a room using image, I don't see anything about this subject, do you have some idea how it could be done?

Welcome to AI.SE Hadrien!

A possible approach is:

Gather many example images of rooms for which you know the square footage. Record the square footage of each room together with each image.
Pick a machine learning model that is well suited to learning relationships between images and numerical outputs, like a Convolutional Neural Network.
Train a machine learning model using an optimisation algorithm, like gradient decent. In the case of training a CNN, this algorithm starts by setting up the network with randomly chosen connection strengths between different 'simulated' neurons. It then exposes the network to an input image, and observes the number the network outputs in response. The algorithm then makes small adjustments to the strengths of the connections in the network, so that if the network were exposed to the image a second time, it would output a number that was less wrong (i.e. closer to the correct square footage). By repeating this process many thousands of times, and with many images, the network eventually becomes quite good at guessing the square footage of new images that it hasn't seen before.

In practice, the network is quite likely to pick up on any patterns in the images that correlate with square footage, not necessarily the ones you want. For example, it might pick up on, say, the fact that humans in the picture are usually a good object to guess at the scale of the rest of the room.


How classification neural nets are different from simple dimension reduction + clustering? I know the training of neural nets involves some sort of dimension manipulation to separate classes of different features.
If there is no variation of features, no matter for neural nets or simple dimension reduction methods (e.g. PCA, LDA) + clustering, neither of them are going to distinguish different classes.
In such sense, I would like to know the true power of neural nets:
How classification neural nets are different from simple dimension reduction + clustering?
or rephrase the question:
What value do neural nets add to solving classification problems in terms of its algorithmic architecture compared with simple dimension reduction + clustering?

PCA is linear, NN are nonlinear, more generally it's a universal function approximator.
That said basic NN are not terribly usefull, the real value of NN is for structured data which is obvious for us but hard to describe analytically. Basically NN architecture has been designed to learn this structure using operations like convolution and maxpooling.
There are still problems of course, and in some ways I think clustering algorithms still have value for things like anomaly detection.


Can gradient descent training be used for nonsmooth loss functions? I have non-smooth loss function - e.g. loss(x)=min(x, 0.5). Can gradient descent be used for training neural networks with such functions. Can gradient descent be used for fairly general, mathematically not-nice functions? Pytorch or Tensorflow can calculate numerically gradients from almost any function, but it is acceptable practice to use general, not-nice loss functions?

Gradient descent and stochastic gradient descent can be applied to any differentiable loss function irrespective of whether it is convex or non-convex. The "differentiable" requirement ensures that trainable parameters receive gradients that point in a direction that decreases the loss over time.

In the absence of a differentiable loss function, the true gradient must be approximated through other methods. For example, in classification problems, the 0-1 loss function is considered the "true" loss, but it is non-convex and difficult to optimize. Instead, surrogate loss functions act as tractable proxies for true loss functions. They are not necessarily worse; negative log-likelihood loss gives a softmax distribution over $k$ classes rather than just the classification boundary.

For your problem specifically, $f(x,a)=min(x,a)$ is not a differentiable loss function. It is not differentiable at $x=0.5$, but the gradient could be estimated through the subgradient. In practice, this works because neural networks often don't achieve the local/global minima of a loss function but instead asymptotically decreasing values that achieve good generalization error. Tensorflow and PyTorch use subgradients when fed non-differentiable loss functions. You could also use a smooth approximation of the $min$ function (see this thread) to get better gradients.


Artificial life simulator I want to develop an artificial life simulator to simulate cells living in water.

I want to see how they search for food, how they life and die and how they reproduce and evolve.

My problem is that I don't know where to start, I have no idea about if there are books or tutorial about how to program this kind of simulator. And also I don't know if I can use here machine learning.

By the way, I'm a programmer and I want to do it using C++ and Unreal Engine.

Where can I find more info about how to do it?

The best approach would be starting with smaller projects involving neural networks and genetic algorithms to gain experience in order to speedup the coding of the project you have proposed; playing around with TensorFlow and Unreal Engine it is not a bad idea.

Hint: when implementing your idea of artificial life, you should consider that each cell/organism have to have some kind of sensors in order to capture informations from the environment; such informations i.e. the position and the distance of the nearest meal and/or predators, the temperature, the pressure and depth of water, should be passed through the neural network to determine the response of the cell. Also, in your environment you should promote the spreading of organisms which responses are euristically better i.e. cells that don't get caught by predators or don't die by starvation. How? Simply by evolving their brain/brains/sensors through a genetic algorithm that favors individuals/species with good parameters. I recommend you a nature-inspired AI method, it is called NEAT model. It explains how to implement a neural networks that can be evolved. The paper can be found here: Evolving Neural Networks through Augmenting Topologies.

A different approach to NEAT would be Deep Reinforcement Learning; in the link you can find a demo artifical organism that learns how to find meals.

There are a ton of parameters and implementations you can consider, the only limit is your creativity.


What is the Bellman operator in reinforcement learning? In mathematics, the word operator can refer to several distinct but related concepts. An operator can be defined as a function between two vector spaces, it can be defined as a function where the domain and the codomain are the same, or it can be defined as a function from functions (which are vectors) to other functions (for example, the differential operator), that is, a high-order function (if you are familiar with functional programming).

What is the Bellman operator in reinforcement learning (RL)? Why do we even need it? How is the Bellman operator related to the Bellman equations in RL?

The notation I'll be using is from two different lectures by David Silver and is also informed by these slides.
The expected Bellman equation is
$$v_\pi(s) = \sum_{a\in \cal{A}} \pi(a|s) \left(\cal{R}_s^a + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^a v_\pi(s')\right) \tag 1$$
If we let
$$\cal{P}_{ss'}^\pi = \sum\limits_{a \in \cal{A}} \pi(a|s)\cal{P}_{ss'}^a \tag 2$$
and
$$\cal{R}_{s}^\pi = \sum\limits_{a \in \cal{A}} \pi(a|s)\cal{R}_{s}^a \tag 3$$
then we can rewrite $(1)$ as
$$v_\pi(s) = \cal{R}_s^\pi + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^\pi v_\pi(s') \tag 4$$
This can be written in matrix form
$$\left.
\begin{bmatrix}
v_\pi(1) \\
\vdots \\
v_\pi(n)
\end{bmatrix}=
\begin{bmatrix}
\cal{R}_1^\pi \\
\vdots \\
\cal{R}_n^\pi
\end{bmatrix}
+\gamma
\begin{bmatrix}
\cal{P}_{11}^\pi & \dots & \cal{P}_{1n}^\pi\\
\vdots & \ddots & \vdots\\
\cal{P}_{n1}^\pi & \dots & \cal{P}_{nn}^\pi
\end{bmatrix}
\begin{bmatrix}
v_\pi(1) \\
\vdots \\
v_\pi(n)
\end{bmatrix}
\right. \tag 5$$
Or, more compactly,
$$v_\pi = \cal{R}^\pi + \gamma \cal{P}^\pi v_\pi \tag 6$$
Notice that both sides of $(6)$ are $n$-dimensional vectors. Here $n=|\cal{S}|$ is the size of the state space. We can then define an operator $\cal{T}^\pi:\mathbb{R}^n\to\mathbb{R}^n$ as
$$\cal{T^\pi}(v) = \cal{R}^\pi + \gamma \cal{P}^\pi v \tag 7$$
for any $v\in \mathbb{R}^n$. This is the expected Bellman operator.
Similarly, you can rewrite the Bellman optimality equation
$$v_*(s) = \max_{a\in\cal{A}} \left(\cal{R}_s^a + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^a v_*(s')\right) \tag 8$$
as the Bellman optimality operator
$$\cal{T^*}(v) = \max_{a\in\cal{A}} \left(\cal{R}^a + \gamma \cal{P}^a v\right) \tag 9$$
The Bellman operators are "operators" in that they are mappings from one point to another within the vector space of state values, $\mathbb{R}^n$.
Rewriting the Bellman equations as operators is useful for proving that certain dynamic programming algorithms (e.g. policy iteration, value iteration) converge to a unique fixed point. This usefulness comes in the form of a body of existing work in operator theory, which allows us to make use of special properties of the Bellman operators.
Specifically, the fact that the Bellman operators are contractions gives the useful results that, for any policy $\pi$ and any initial vector $v$,
$$\lim_{k\to\infty}(\cal{T}^\pi)^k v = v_\pi \tag{10}$$
$$\lim_{k\to\infty}(\cal{T}^*)^k v = v_* \tag{11}$$
where $v_\pi$ is the value of policy $\pi$ and $v_*$ is the value of an optimal policy $\pi^*$. The proof is due to the contraction mapping theorem.


Are Dreams a Form of Backpropagation? Humans often dream of random events that occurred during the day. Could the reason for this be that our brains are backpropagating errors while we sleep, and we see the result of these backpropagations as dreams?

This is a very interesting question and also an important one for AI. All the current Deep Learning successes are built on the effectiveness of backprop, so what if it doesn't play a role in the only examples of intelligence currently around?

It makes a lot of sense for the brain to employ some form of backprop, because that would allow it to create low-level features in a way most conductive for the high-level features that finally give rise to intelligent behavior. And if the brain employs backprop, dreams are a logical candidate.

Of course the problem here is that the brain doesn't calculate and propagate gradients, so the question is whether the backpropagation algorithm can somehow be implemented with the biological constraints in mind, such as the locality of neural computation and the non-symmetrical feedback connections.

It turns out this is actually possible, as explained in these talks by Hinton and Bengio. Symmetrical feedback connections (and in fact continually updated feedback connections) aren't necessary for backpropagation, however unintuitive this may seem. And backpropagation can be emulated with a form of temporal difference learning, that is plausible within biological constraints.

Of course this doesn't conclusively answer the question, but it certainly makes the possibility much more plausible than a priori assumed.


What should the value of epsilon be in the Q-learning? I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles.
What I don't understand is the impact of $\epsilon$. What value should it take? $0.5$, $0.6$, or $0.7$?
What does it mean when $\epsilon = 0$ and $\epsilon = 1$? If $\epsilon = 1$, does it mean that the agent explores randomly? If this intuition is right, then it will not learn anything - right? On the other hand, if I set $\epsilon = 0$, does this imply that the agent doesn't explore?
For a typical problem, what is the recommended value for this parameter?

What does it mean when ϵ=0 and ϵ=1? If ϵ=1, does it mean that the agent explores randomly? If this intuition is right, then it will not learn anything - right? On the other hand, if I set ϵ=0, does this imply that the agent doesn't explore?

You are correct, when  ϵ=1 the agent acts randomly. When  ϵ=0, the agent always takes the current greedy actions. Both of these scenarios are not ideal. Always acting greedily will prevent the agent from exploring possibly better parts of the  state space, and instead the agent may get stuck in a local optimum. And always exploring randomly is obviously not ideal as well. Thus, we need to balance between these two. This is often called the balance between exploration and exploitation.

For a typical problem, what is the recommended value for this parameter?

ϵ is a hyper parameter. It is impossible to know in advance what the ideal value is, and it is highly dependent on the problem at hand. There is no general answer to this question.
That being said, the most common values that I have seen typically range between 0.01 and 0.1. But I want to stress, there is no ideal value that works for all problems. A typical strategy is to try several values and see which one works best. For more information, you might want to look up hyper parameter tuning.
Another common practice is slowly decaying epsilon over time (often this is called "annealing" or "simulated annealing"). Depending on the algorithm, decaying epsilon to zero may be a requirement for convergence. In some contexts, an algorithm that decays epsilon over time is called a GLIE algorithm. For example, see this.


How to understand the concept of self-supervised learning in AI? I am new to self-supervised learning and it all seems a little magical at the moment.
The only way I can get an intuitive understanding is to assume that, for real-world problems, features are still embedded at a per-object level.
For example, to detect cats in unseen images, my self-supervised network would still have to be composed exclusively of cats.
So, if I had 100 images of cats and 100 images of dogs, then I thought self-supervised approaches would learn the features of the images. For example, if an image is rotated 90 degrees, it learns what was in the image that was rotated 90 degrees. However, if I wanted to classify just cats using this representation, then I wouldn't be able to do so without separating out what makes a cat a cat and a dog a dog.
Is my assumption correct?

I don't think your interpretation is correct. Take images as example.

Supervised Learning

e.g. classification (maybe use CNN with a L2 loss function)

Assume you have many images with different labels. You wish to find a function to approximate the function $y=f(x)$ given a lot of $(\hat x, \hat y)$ sample pairs.
Unsupervised Learning

e.g. clustering (maybe use k-means)

Assume you have many images, but we don't have the labels or we just want to see if there's a way to categorize them into different categories. So we cluster the images by some  characteristic that isn't pre-defined.
Self-Supervised Learning

e.g. super resolution (maybe use CNN with a L2 loss function)

You have many high resolution images without labels, but, your target is to train a model to up sample a low resolution image. So you can have the high resolution images as target, and down size the image to be the input, and try to train the image pairs. So the target is not some manually tagged labels, but generated directly from the data.


How does DeepMind perform reinforcement learning on a TPU? I've watched this video of the recent contest of AlphaStar Vs Pro players of StarCraft2, and during the discussion David Silver of DeepMind said that they train AlphaStar on TPUs.

My question is, how is it possible to utilise a GPU or TPU for reinforcement learning when the agent would need to interact with an environment, in this case is the StarCraft game engine?

At the moment with my training of a RL agent I need to run it on my CPU, but obviously I'd love to utilise the GPU to speed it up. Does anyone know how they did it?

Here's the part where they talk about it, if anyone is interested:

https://www.youtube.com/watch?v=cUTMhmVh1qs&t=7030s

In their blog post, they link to (among many other papers) their IMPALA paper. Now, the blog post only links to that paper with text implying that they're using the "off-policy actor-critic reinforcement learning" described in that paper, but one of the major points of the IMPALA paper is actually an efficient, large-scale, distributed RL setup.

So, until we get more details (for example in their paper that's currently under review), our best guess would be that they're also using a similar kind of distributed RL setup as described in the IMPALA paper. As depicted in Figures 1 and 2, they decouple actors (machines running code to generate experience, e.g. by playing StarCraft) and learners (machines running code to learn/train/update weights of neural network(s)).

I would assume that their TPUs are definitely being used by the Learner (or, likely, multiple Learners). StarCraft 2 itself won't benefit from running on TPUs (and probably would be impossible to even get to run on them in the first place), because the game logic likely doesn't depend on large-scale, dense matrix operations (the kinds of operations that TPUs are optimized for). So, the StarCraft 2 game itself (which only needs to run for the "actors", not for the "learners") is almost certainly running on CPUs.

The actors will still have to run forwards passes through Neural Networks in order to select actions. I would assume that their Actors are still equipped with either GPUs or TPUs to do this more quickly than a CPU would be capable of, but the more expensive backwards passes are not necessary here; only the Learners need to perform those.


How does backpropagation with unbounded activation functions such as ReLU work? I am in the process of writing my own basic machine learning library in Python as an exercise to gain a good conceptual understanding. I have successfully implemented backpropagation for activation functions such as $\tanh$ and the sigmoid function. However, these are normalised in their outputs. A function like ReLU is unbounded so its outputs can blow up really fast. In my understanding, a classification layer, usually using the SoftMax function, is added at the end to squash the outputs between 0 and 1.

How does backpropagation work with this? Do I just treat the SoftMax function as another activation function and compute its gradient? If so, what is that gradient and how would I implement it? If not, how does the training process work? If possible, a pseudocode answer is preferred.

Backprop through ReLU is easier than backprop through sigmoid activations. For positive activations, you just pass through the input gradients as they were. For negative activations you just set the gradients to 0.

Regarding softmax, the easiest approach is to consider it a part of the negative log-likelihood loss. In other words, I am suggesting to directly derive gradients of that loss with respect to the softmax input. The result is very elegant and extremely easy to implement. Try to derive that yourself!


Thoughts on Apple Mac Pro vs GCP/AWS for Deep Learning? I've been oogling the Mac Pro from Apple with loaded specs. Check it here if unfamiliar.

I'm curious to hear anyone's thoughts of the computer for deep learning/machine learning applications vs cloud based computing. Obviously, a flexible solution on GCP or AWS has the ability to scale and therefore produce results faster. However, I'm mostly interested to hear thoughts on the economics of purchasing the machine vs renting a cloud based machine.

If we assumed something like a 5 year lifetime of the Apple Mac Pro for this type of computing (before the chips/processors become immensely slower than the newest chips used in the cloud), that would be like $1000 per year amortized using the Mac Pro. Does it make more sense to rent from AWS/GCP for \$1000 per year as a renter instead? I don't have much experience in the cost of AWS/GCP so I'm looking to here an answer from anyone well versed in cloud computing.

The Mac Pro uses AMD GPUs. These don't support CUDA, but instead support the OpenCL framework.

TensorFlow, the most popular deep learning library, uses CUDA to run on GPUs, although OpenCL support is in the works. That said, one of the main approaches to OpenCL support, SYCL, isn't planning to support OSX:

  We have no plans to support OSX in near future. I know sad face.

The current advice is to avoid AMD GPUs:

  Currently, if you want to do DL and want to avoid major headaches, choose Nvidia.


How to handle the final state in experience replay? I'm using the DQN algorithm to train my agent to play a turn-based game. The memory replay buffer stores tuples of experiences $(s, a, r, s')$, where $s$ and $s'$ are consecutive states. At the last turn, the game ends, and the non-zero reward is given to the agent. There are no more observations to be made and there is no next state $s'$ to store in the experience tuple. How should the final states be handled?

You do not store a terminal state as $s$ in the replay table because by definition its value is always $0$, and there is no action, reward or next state. There is literally nothing to learn.
However, you may find it useful to store information that $s'$ is actually a terminal state, in case this is not obvious. That is typically achieved by storing an additional done boolean component. This is useful, because it allows you to branch when calculating the TD target g:
s, a, r, next_s, done = replay_memory_sample()
if done:
  g = r
else:
  g = r + gamma * max( q(s') )


Grouping products and naming groups I'm working on a home tool that will help create a shopping list from a list of recipes chosen for a coming week.

This boils down to:

Extracting ingredients and their quantities from recipes.
Grouping similar ingredients together.
Summing up quantities for similar ingredients.
Naming groups of similar products in a shopping list.

The tasks seem non-trivial for a few reasons.

Similar ingredients are described differently, depending on the recipe book/portal, e.g.:

  5 lemons
  5 lemons (to be squeezed)
  5 fresh lemons
  5 big yellow lemons

Recipes lists alternatives for ingredients (e.g., "3 lemons or 5 limes"), leaving decision up to a user.
Recipes involve some information about product-preprocessing. For instance, one has to buy lemons instead of lemon juice when the recipe says:

  100ml lemon juice
  100ml freshly squeezed lemon juice

My language has a complex inflection. For instance, there can be multiple plural forms of a noun and the form of an adjective must be agreed with a form of noun. Adapting NLP algorithms designed for English language might be not straightforward and require some lemmatizing/stemming but not for single words, but whole phrases.
Naming products group is hard. Once fresh lemons and big yellow lemons are group together and their quantities summed up, one need to decide how to name this group in a shopping list, e.g.: "10 lemons" or "10 fresh lemons".

Is there any research paper that would cover those challenges?

Especially applied in the same domain?

I worked at a start-up doing this exact task for two years. There were several developers involved in it, and it took several years; by the end (with a lot of manual input, eg categorising ingredients in an ontology, converting measurements, and mapping ingredients to shopping items from a supermarket inventory) we had something that worked reasonably well.

You rightly recognise that it is not trivial... The first step would be to identify the list of ingredients from the recipe (assuming you have scraped it from a website), and turn it into a list; beware that sometimes there might be more than one ingredient in a line. Then you need to analyse the entry: we had a grammar that distinguished between quantities, attributes (fresh), processing instructions (peeled) and a number of other things. The main item would of course be the actual ingredient. Then we would look this up in a supermarket inventory and map it to the closest item (but I don't think this is part of your remit).

We tried different ways of analysing the recipe lines, and any grammar/parser is fine, as long as it can deal with partial analyses, as you cannot always rely on lines being complete and well-formed, and there is always scope for additional comments you want to ignore ("I only use the freshest lemons directly imported from Tuscany", "get your fishmonger to take the bones out"). In the simplest case you could have a list of items (lemon) with inflections etc, that you try to match in your recipe lines. If you don't care whether it's big lemons or not, don't bother using a full-blown grammar. And even then, you could just look at a list of adjectives preceding an ingredient to identify any modifiers. We used a grammar based on recursive transition networks (RTNs, where the parser was complex but the grammars easy to handle) and one that was a simple phrase structure one (where the parser was simpler, but the grammar was much more complex). Both worked fine.

As far as I am aware there is not really much written about this kind of work. The main advice I can give you is to look at as many recipes as possible to get an idea for what you will encounter, and cut corners wherever possible -- it's a big job to do it perfectly, and you can get very far by using simple methods.

We also transferred it into different languages (Polish too, which I assume you're working with?). We didn't find that inflections were that big an issue. You just have a list of all inflected forms linking to the lemma. As you have a limited domain, that is a perfectly feasible approach. Just have a master list of ingredients, with all alternate forms (courgette and zucchini are the same thing, for example).


How could I use reinforcement learning to solve a chess-like board game? I invented a chess-like board game. I built an engine so that it can play autonomously. The engine is basically a decision tree. It's composed by:

A search function that at each node finds all possible legal moves
An evaluation function that assigns a numerical value to the board position (positive means the first player is gaining the upper hand, negative means the second player is winning instead)
An alpha-beta pruning negamax algorithm

The main problem with this engine is that the optimization of the evaluation function is really tricky. I don't know which factors to consider and which weights to put. The only way I see to improve the engine is to iterate games trying each time different combinations of factors and weights. However, it computationally seems a very tough feat (Can I backpropagate without using deep learning?).
I would like to use reinforcement learning to make the engine improve by playing against itself. I have been reading about the topic, but I am still quite confused.

What other reward is there in a game apart from the win-or-lose output (1 or 0)?

If I use other rewards, like the output from the evaluation function at each turn, how can I implement it?

How do I modify the evaluation function to give better rewards iteration after iteration?

I would like to use reinforcement learning to make the engine improve by playing against itself. I have been reading about the topic but I am still quite confused.

Be warned: Reinforcement learning is a large complex subject. Although it might take you on a detour from game-playing bots, you may want to study RL basics. A good place to start is Sutton & Barto Reinforcement Learning: An Introduction

  What other reward is there in a game a part the win-or-lose output (1 or 0)?

Depending on you game, that is usually it. Actually for a win/draw/lose game like chess then the reward from each action is 0 except for winning (+1) or losing (-1) at the end. In a zero-sum game then this aligns nicely to minimax, alphabeta pruning etc.

Reinforcement Learning is intended to address environments with delayed rewards. Adding "helper" rewards for interim non-goals is usually counter-productive.

  If I use other rewards, like the output from the evaluation function at each turn, how can I implement it?

Typically you don't. What applying self-playing RL will do is learn a return (sometimes called utility) function which predicts the expectation of your total +1/0/-1 reward by the end of the game. You would use this in place of your current heuristic for minimax search. Or, potentially you would adjust your current heuristic function to output in the same range and use RL to optimise its weights to make the best approximation to the true optimal play return function (which is probably too complex to calculate exactly).

  How do I modify the evaluation function to give better rewards iteration after iteration?

That is what the different RL approaches all attempt to do, there are a variety of different solvers. There is no short way to explain it. You could start with a simple method such as Q-Learning. Q-Learning learns estimates of Q(s,a) (called the action value) which is the expected return when in state s and taking action a, and thereafter following an optimal policy. It makes an arbitrary guess to start with and refines it closer to the true value with each step made in the learning environment. Simple tabular Q-learners do this refinement simply by storing a big table of all states and actions with the best estimate so far of the true value, and averaging in each new estimate as it is experienced.

It is also possible to combine an RL method for heuristics with look-ahead minimax search - that is what the original AlphaGo did, and what AlphaGo Zero does during training. This is a powerful approach because minimax search will work to double-check the RL-generated heuristics. Although for simple enough games, RL can learn perfect heuristics and you would only need local search (what the next move should be).

Unless your game is very simple (all possible states would fit in memory), you will need some kind of function approximator inside the RL algorithm. Neural networks are a standard choice. Having something for that part is unavoidable - although another good choice is to define a bunch of proxy features (that you might use to construct a heuristic by hand) and use a linear approximator - just a weighted sum of all the features. This can work well enough, and has been used for example in checkers (draughts) players trained using RL.

In fact, provided your own heuristic function is not too unusual, you can probably treat it just like a linear approximator and use RL to learn the best weights for it.


Converging to a wrong optimal policy if the agent is given more choices I am a bit new to Reinforcement learning. So, I am extremely sorry if I am asking something obvious. I have written a small piece of code to find the optimal policy for a 5x5 grid problem.

Scenario 1. The agent is only given two choices (Up, Right). I believe, I am getting an optimal policy.
Scenario 2. The agent is given four choices (Up, Right, Down, Left). I am getting the wrong answer.

I have represented actions with numbers:
0 - Right
1 - Up
2 - Down
3 - Left

When the action Up is chosen, with 0.9 probability it will move up or 0.1 probability move right and vice-versa. When the action Down is chosen, with 0.9 probability it will move down or 0.1 probability move left and vice-versa.
I did not use any convergence criteria. Instead let it run for sufficient iterations. I have indeed confirmed that my optimal state values and policy is converging but to a wrong number. I am attaching the code below:
def take_right(state):
    if (state/n < n-1): state = state + n
    return state

def take_up(state):
    if (state%n!=n-1): state = state + 1
    return state

def take_left(state):
    if (state/n > 0): state = state - n
    return state

def take_down(state):
    if (state%n > 0): state = state - 1
    return state

Scenario 1 result:

Scenario 2 result:

Green has a reward of 100 and Blue has a penalty of 100. Rest of the states have a penalty of 1. Discount factor is chosen as 0.5
Edit:
This was really silly question. The problem with my code was more pythonic than RL. Check the comments to get the clue.

Reinforcement Learning is really fun because the agent will find any bug in your implementation and will exploit it.
>>> take_left(0)
0
>>> take_left(1)
-4

The agent figured out your bug with negative values and exploits negative indexing to get to the target faster.


How to make a LSTM network to predict sequence only after input sequence is finished? I am learning to use a LSTM model to predict time series data. Specifically, I hope the network should output a sequence (with multiple time steps) only after the input sequence has finished feeding in, as shown in the left figure.

However, most of the LSTM sequence-to-sequence prediction tutorial I have read seems to be the right figure (i.e. each time step of the output sequence is generated after each time step of the input sequence). What's more, as far as I understand, the LSTM implementation in PyTorch (and probably Keras) can only return output sequence corresponding to each time step of the input sequence. It cannot make predictions after the input sequence is over.

I hope to know is there any way to make a sequence-to-sequence LSTM network which starts output only after the input sequence has finished feeding in? And it would be better if someone can show me an example implementation code.

You should try an architecture with an encoder and a decoder. The encoder will consume all the data you give as in put and decoder will give out the series of output.


Can the English language ever be generalized using a set of grammar rules? In programming languages, there is a set of grammar rules which govern the construction of valid statements and expressions. These rules help in parsing the programs written by the user.

Can there ever be a functionally complete set of grammar rules which can parse any statement in English (locale-specific) accurately and which can be possibly implemented for use in AI-based projects?

I know that there are a lot of NLP Toolkits available online, but they are not that effective. Most of them are trained using specific corpuses which sometimes fail to infer some complex correlations between various parts of an expression.

In other words, what I am asking is that if it is possible for a computer to parse a well-versed sentence written in English as if it were parsed by an adult English-speaking human?

EDIT:
If it cannot be represented using simple grammar rules, what kind of semantic structure can be used to generalize it?

EDIT2: This paper proves the absence of context-freeness in natural languages. I am looking for a solution, even if it is too complex.

Can there ever be a functionally complete set of grammar rules which can parse any statement in English (locale-specific) accurately and which can be possibly implemented for use in AI-based projects?

Parse it yes, accurately most likely no.

Why ?

According to my understanding on how we derive meaning from sounds, there are 2 complementary strategies:

Grammar Rules:
A rule based system for ordering words to facilitate communication, here meaning is derived from interaction of discrete sounds and their independent meaning, so you could parse a sentence based on a rule book.

E.G. "This was a triumph" : the parser would extract a pronoun (This) with corresponding meaning ( a specific person or thing ) ; a verb (was) with corresponding meaning ( occurred ); ( a) and here we start with some parsing problems , what would the parser extract, a noun or an indefinite article ? An so we consult the grammar rule book, and settle for the meaning ( indefinite article any one of ), you have to parse the next word  and refer to it though, but let's gloss over that for now, and finally (triumph) a noun ( it could also be a verb, but thanks to the grammar rule book we settled for a noun with meaning: ( victory,conquest), so in the end we have ( joining the meanings ):

A specific thing occurred of victory. Close enough and I am glossing over a few other rules, but that's not the point, the other strategy is:

A lexical dictionary (or lexicon)
Where words or sounds are associated with specific meaning. Here meaning is derived from one or more words or sounds as a unit. This introduces the problem to a parser, since well, it shouldn't parse anything.

E.G. "Non Plus Ultra" And so the AI parser would recognize that this phrase is not to be parsed and instead matched with meaning :

The highest point or culmination

Lexical units introduce another issue in that they themselves could be part of the first example, and so you end up with recursion.

  if it is possible for a computer to parse a well-versed sentence written in English as if it were parsed by an adult English-speaking human?

I believe it could be possible, most examples I've seen deal effectively with the grammar rule book or the lexicon part, but I am not aware of a combination of both, but in terms of programming, it could happen.

Unfortunately even if you solve this problem, your AI would not really understand things in the strict sense, but rather present you with very elaborate synonyms, additionally context (as mentioned in the comments) plays a role into the grammar and lexicon strategies.

  If it cannot be represented using simple grammar rules, what kind of semantic structure can be used to generalize it?

A mixed one where there are both grammar rules and a lexicon and both can change and be influenced based on the AI specific context and experience as well as a system for dealing with these objects could be one way.


How can I use the success and failure data to estimate parameters of a Dirichlet distribution? I have used Beta function to estimate the performance of the agent. I have failure and success data of the task that runs on the agent.
The parameter $\alpha$ is a number of successful tasks, while $\beta$ is the number of failures. Thus, I can estimate the performance by exploiting the expected value of Beta, as $$\mu = \frac{\alpha} {(\alpha+\beta)}$$

So, I am looking for a similar model, such that its parameter can be estimated from the success and failure data. So far I found Dirichlet distribution.

What is the expected value of Dirichlet distribution? How I can use the success and failure data to estimate parameters of this distribution?

Let's check the following example:

Suppose that we use a Dirichlet prior represented by $Dirichlet(1, 1, 1)$ and observe $13$ results with $8$ Successful, $2$ Missing, and $3$ Failures. Then we get the posterior to be $Dirichlet(1+8, 1+2, 1+3)$. Then if you define the performance value $\alpha$ to be the expectation of $P(x=Successful)$,
then $\alpha$ will be $(1+8)/[(1+8)+(1+2)+(1+3)] = 0.56$

Now
Suppose that we use a Beta prior represented by $Beta(1,1)$ and observe $13$ results with $8$ Successful, and $3$ Failures. Then we get the posterior to be $Beta(1+8, 1+3)$. Then if you define the performance value Pr to be the expectation of $P(x=Successful)$,
then $\alpha = (1+8)/[(1+8)+(1+3)] = 0.69$

Are my calculations and concept right?

Dirichlet is the Multi Variate version of the Beta distribution. In general, these distributions can be thought to model the probability of modelling a probability distribution.

The support Dirichlet distribution is defined as follows:

$$ S_K = \{ x:0 \leq x_k \leq 1, \sum_{k=1}^K x_k=1 \} $$

and the PDF is defined as:

$$Dir(x|\alpha) = \frac{1}{B(\alpha)} \prod_{k=1}^Kx_k^{\alpha_k-1}$$

where $B(\alpha)$ is the beta function of $K$ variables:

$$B(\alpha) = \frac{\prod_{k=1}^K \tau(\alpha_k)}{\tau(\sum_{k=1}^K \alpha_k)}$$

and the resultant point estimates are (Define $\sum_{k=1}^K \alpha_k = \alpha_0)$:

$$\mu(x_k) = \frac{\alpha_k}{\alpha_0}$$
$$\sigma^2(x_k) = \frac{\alpha_k(\alpha_0-\alpha_k)}{\alpha_0^2(\alpha_0+1)}$$
$$mode[x_k] = \frac{\alpha_k-1}{\alpha_0-1}$$

Beta distribution is the special case where $k=2$

Clearly, when you run an experiment a large number of times, the success of each $k$ will approach towards its expected value i.e if you define your random variables as $x_k = \frac{N_{k}}{N}$ where $N$ is the total number of trials and $N_k$ is the success of the $k$ th term, it clearly satisfies the support of the Dirichlet Distribution and hence you can use

$$\frac{\alpha_{k}}{\alpha_0} = \frac{N_{k}}{N}$$

This is assuming that the experiment follows Dirichlet Distribution.

(Taken in parts from A Probabilistic Approach to ML)


Into which subcategories can reinforcement learning be divided? In the course of a scientific work, I will discuss the different types of reinforcement learning. However, I have difficulties to find these different types.
So, into which subcategories can reinforcement learning be divided? For example, the following subdivisions seem to be useful

Model-free and Model-based
Dynamic Programming, Monte Carlo and Temporal Difference

Any others?

Your two suggestions are not mutually exclusive. If you go by this process, you'll have to do a "Cartesian product" of a bunch of different RL categorizations which would get out of hand. I recommend, if you can, to describe some sort of "RL taxonomy" instead. By this I mean describing different RL characterizations without assuming they're mutually exclusive.
To add to your list :

On-policy or off-policy
Value based or policy gradient


How do we design a neural network such that the $L_1$ norm of the outputs is less than or equal to 1? What are some ways to design a neural network with the restriction that the $L_1$ norm of the output values must be less than or equal to 1? In particular, how would I go about performing back-propagation for this net?
I was thinking there must be some "penalty" method just like how in the mathematical optimization problem, you can introduce a log barrier function as the "penalty function"

Penalty (barrier function) is perfectly valid and simplest method for simplex type constraint (L1 norm is simplex constraint on absolute values). Any type of barrier function may work, logarithmic, reciprocal or quadratic. All of them supported by any major framework(pytorch, tensorflow), just add them to loss function. You would need some hyperparameter tuning for the scale factor of penalty.

There is more efficient, though more complex way to do it. Instead of putting constraint you can automatically output value wich satisfy simplex constraint:

Assume that L1 norm constraint is $\left \|v\right \|_1 \leq 1$, $v \in \mathbb{R}^n$

put $sigmoid(v_i)$ activation on output to norm elements to [-1, 1]
add slack (fake) variable element $v_{n+1} = 1 - \sum_{1}^{n}  v_i $
project new $v{}'\in \mathbb{R}^{n+1}$, $v{}'_i = |v_i|,1\leq  i  \leq n+1$ onto unit simplex with standard algorithm (also here)

Backpropagation of last step may require differentiable sorting, which is missing in most of frameworks, you may have to look for open sourced implementation, for example extract it from here or use some automatic differentiation package. Both require some substantial code reading/debugging. However in my experience assuming constant $\Delta$ also works in many cases, in that case  differentiable sorting is not needed. Intuition behind constant $\Delta$  is that  $\Delta$  could be chosen such way that there is some interval on which it's value doesn't affect sorting order.


How does the decision tree implicitly do feature selection? I was talking with an ex-fellow worker and he told me that the decision tree implicitly applies a feature selection. He told me that the most important feature is higher in the tree because of the usage of information gain criteria.

What does he mean with this and how does this work?

Consider a dataset $S \in \mathbb{R}^{N \times (M + 1)}$ with $N$ observations (or examples), where each observation $S_i \in \mathbb{R}^{M + 1}$ is composed of $M$ elements, one value for each of the $M$ features (or independent variables), $f_1, \dots f_M$, and the corresponding target value $t_i$.

A decision tree algorithm (DTA), such as the ID3 algorithm, constructs a tree, such that each internal node of this tree corresponds to one of the $M$ features, each edge corresponds to one value (or range of values) that such a feature can take on and each leaf node corresponds to a target. There are different ways of building this tree, based on different metrics to choose the features for each internal node and based on whether the problem is classification or regression (so based on whether the targets are classes or numeric values).

For example, let's assume that the features and the target are binary, so each $f_k$ can take on only one of two possible values, $f_{k} \in \{0, 1\}, \forall k$, and $t_i \in \{0, 1\}$ (where the index $i$ correspond to the $i$th observation, while the index $k$ correspond to the $k$th column or feature of $S$). In this case, a DTA first chooses (based on some metric, for example, the information gain) one of the $M$ features, for example, $f_j$, to associate it with the root node of the tree. Let's call this root node $f_j$. Then $f_j$ will have two branches, one for each of the binary values of $f_j$. If $f_j$ were a ternary variable, then the node corresponding to $f_j$ would have three branches, and so on. The DTA recursively chooses one of the remaining features for each node of the child branches. The DTA does this until all features have already been selected. In that case, we will have reached a leaf node, which will correspond to one value of the target variable. When the DTA chooses a feature for a node, all observations of the dataset $S$ that take on the first binary value of that feature will go in the branch corresponding to that value and all other observations will go in the other branch. So, in this way, the DTA splits the dataset based on the features.

The following diagram represents a final decision tree built by a DTA.

You can see that the first feature selected (for the root node) by the DTA is "Is it male?", which is a binary variable. If yes, then, on the left branch, we have another internal node, which corresponds to another feature and, at the same time, to all observations associated with a male. However, on the right branch, we have a leaf node, which corresponds to one value of the target, which, in this case, is a probability (or, equivalently, a numerical value in the range $[0, 1]$). The shape of the tree depends on the dataset and DTA algorithm. Therefore, different datasets and algorithms might result in different decision trees.

So, yes, you can view a decision tree algorithm as a feature selection or, more precisely, feature splitting algorithm.


Shouldn't the utility function of two-player zero-sum games be in the range $[-1, 1]$? In Appendix B of MuZero, they say

  In two-player zero-sum games the value functions are assumed to be bounded within the $[0, 1]$ interval.

I'm confused about the boundary: Shouldn't the value/utility function be in the range of [-1,1] for two-player zero-sum games?

it can be either. If you consider the lack of reward as "penalty" then getting 0 reward is bad.

if you use a value estimator through a neural network, the range of rewards will dictate the squashing function you use for the output layer


What are "proxy data sets" in machine learning? The paper Assessment of Deep Generative Models for High-Resolution Synthetic Retinal Image Generation of Age-Related Macular Degeneration uses the term "proxy data sets" in this way

  To develop DL techniques for synthesizing high-resolution realistic fundus images serving as proxy data sets for use by retinal specialists and DL machines.

I googled that term, but didn't find a definition of "proxy data sets". What are "proxy data sets" in machine learning?

The paper Analysis of Manufacturing Process Sequences, Using Machine Learning on Intermediate Product States (as Process Proxy Data) mentions a similar term

  The advantage of the product state-based view is the focus on the product itself to structure the information and data involved throughout the process. Using the intermediate product states as proxy data for this purpose

Does "proxy data" mean the same thing as "proxy data sets" does?

In computer science, if you say "A is a proxy for B", then it means that "A replaces B" (temporarily or not), or that "A is used as an intermediary for B".
The term "proxy" usually refers to a server, i.e. there are the so-called proxy servers, which intuitively do the same thing (i.e. they are used as intermediaries). The following picture is worth more than 1000 words.

Let's go back to your context. In the following paragraph

To develop DL techniques for synthesizing high-resolution realistic fundus images serving as proxy data sets for use by retinal specialists and DL machines.

The deep learning techniques will be used to generate data that will be used as a replacement for real data for use by retinal specialists.
In your second paragraph

The advantage of the product state-based view is the focus on the product itself to structure the information and data involved throughout the process. Using the intermediate product states as proxy data for this purpose

The word "intermediate" is even used, so there should not be any need for further explanation, although I admit that I have no idea of what "product states" are in this context, but they will be used as "proxy data" for other data.

Does "proxy data" mean the same thing as "proxy data sets" does, if yes, I will go through that paper.

I guess so, but bear in mind that "data" is more general than "data set".
I have found other slightly different uses of the expression "proxy data", not in the context of computer science, but in the context of paleoclimatology. For example, have a look at this web article What Are Proxy Data?.
Just for completeness, here are some dictionary definitions of proxy.

authority given to a person to act for someone else, such as by voting for them in an election
a person who you choose to do something officially for you
a situation, process, or activity to which another situation, etc. is compared


Why are reinforcement learning methods sample inefficient? Reinforcement learning methods are considered to be extremely sample inefficient.
For example, in a recent DeepMind paper by Hessel et al., they showed that in order to reach human-level performance on an Atari game running at 60 frames per second they needed to run 18 million frames, which corresponds to 83 hours of play experience. For an Atari game that most humans pick up within a few minutes, this is a lot of time.
What makes DQN, DDPG, and others, so sample inefficient?

I will try to give a broad answer, if it's not helpful I'll remove it.

When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model of the environment.
In general I would say that there are two issues related to sample efficiency:
1 the size of the 'action'+'environment states' space 2 the exploration strategy used.

Regarding the first point, in reinforcement learning is really easy to encounter situations in which the number of combinations of possible actions and possible environment states explode, becoming intractable. Lets for example consider the Atari game from the Rainbow paper you linked: the environment in which the agent operate in this case is composed of rgb images of size (210, 160, 3). This means that the agent 'see' a vector of size 100800. The actions that an agent can take are simply modifications of this vector, e.g. I can move to the left a character, slightly changing the whole picture. Despite the fact that in lot of games the number of possible actions is rather small, we must keep in mind that there are also other objects in the environment which change position as well. What other object/enemies do obviously influence the choice of the best action to perform in the next time step. To a high number of possible combinations between actions and environment states is associated a high number of observations/interaction required to learn a good model of the environment. Up to now, what people usually do is to compress the information of the environment (for example by resizing and converting the pictures to grayscale), to reduce the total number of possible states to observe. DQL itself is based on the idea of using neural networks to compress the information gathered from the environment in a dense representation of fixed size.

For what concern the exploration strategy, we can again divide the issue in subcategories: 1 how do we explore the environment 2 how much information do we get from each exploration. Exploration is usually tuned  through the greedy hyper-parameter. Once in a while we let the agent perform a random action, to avoid to get stuck in suboptimal policies (like not moving at all to avoid to fall into a trap, eventually thanks to a greedy action the agent will try to jump and learn that it gives a higher reward). Exploration comes with the cost of more simulations to perform, so people quickly realise that we can't rely only on more exploration to train better policies. One way to boost performance is to leverage not only the present iteration but also past interactions as well, this approach is called experience replay. The underline idea is to update the q-values depending also on weighted past rewards, stored in a memory buffer. Other approaches point to computation efficiency rather than decreasing the amount of simulations. An old proposed technique that follow this direction is prioritised sweeping Moore et al. 1993, in which big changes in q values are prioritised, i.e. q-values that are stable over iterations are basically ignored (this is a really crude way to put it, I have to admit that I still have to grasp this concept properly). Both this techniques were actually applied in the Rainbow paper.

On a more personal level (just pure opinions of mine from here) I would say that the problem between RL agents and humans is the fact that we (humans) have lot of common sense knowledge we can leverage, and somehow we are able, through cognitive heuristics and shortcuts, to pay attention to what is relevant without even being aware of it. RL agents learn to interact with an environment without any prior knowledge, they just learn some probability distribution through trial and errors, and if something completely new happens they have no ability at all to pick up an action based on external knowledge.
One interesting future direction in my opinion is reward modelling, described in this video: https://youtu.be/PYylPRX6z4Q

I particularly like the emphasis on the fact that the only true thing that human are good at is judging. We don't know how to design proper reward functions, because again, most of the actions we perform in real life are driven by reward of which we are not aware, but we are able in a glimpse to say if an agent is performing a task in a proper way or not. Combining this 'judging power' into RL exploration seems to be a really powerful way to increase sample efficiency in RL.


Transformers: how does the decoder final layer output the desired token? In the paper Attention Is All You Need, this section confuses me:

In our model, we share the same weight matrix between the two embedding layers [in the encoding section] and the pre-softmax linear transformation [output of the decoding section]

Shouldn't the weights be different, and not the same? Here is my understanding:
For simplicity, let us use the English-to-French translation task where we have $n^e$ number of English words in our dictionary and $n^f$ number of French words.

In the encoding layer, the input tokens are $1$ x $n^e$ one-hot vectors, and are embedded with a $n^e$ x $d^{model}$ learned embedding matrix.

In the output of the decoding layer, the final step is a linear transformation with weight matrix $d^{model}$ x $n^f$, and then applying softmax to get the probability of each french word, and choosing the french word with the highest probability.

How is it that the $n^e$ x $n^{model}$ input embedding matrix share the same weights as the $d^{model}$ x $n^f$ decoding output linear matrix? To me, it seems more natural for both these matrices to be learned independently from each other via the training data, right? Or am I misinterpreting the paper?

I found the answer by reading the paper referenced by that section, Using the output embedding to improve language models

Based on this observation, we propose threeway weight tying (TWWT), where the input embedding of the decoder, the output embedding of
the decoder and the input embedding of the encoder are all tied. The single source/target vocabulary of this model is the union of both the source
and target vocabularies. In this model, both in the
encoder and decoder, all subwords are embedded
in the same duo-lingual space.

It seems like they learned a single embedding matrix ($n^e + n^f$) x $d^{model}$ in dimension.


How does an AI learn? I can't seem to understand how an AI learns. Without having a programmer tell it what to do, how would a program create or generate some solution to a problem and then use information gained in future problems? I understand how chess AI works. But what's really confusing is how an AI would improve or learn.

There is no strong evidence that a single mechanism of learning exists in the neural networks of the human brain.  The scientifically collected evidence does not support that simplistic conception.

Learning is an umbrella term under which many things reside.

Accumulation of rules triggered by corresponding sets of conditions
Pattern recognition, such as the recognition of the probable source of a knock or a saxophone note
Connection between intention, motor output, and sensory feedback, backed by pattern recognition, as with hand-eye coordination
Construction and refinement of internal models of subjective reality that can be called upon to make choices or determine quantities
Construction and refinement of external models of objective reality, such as Newton's recognition that Kepler's description of elliptical orbits, Aristotle's attraction of objects to the ground, and the constant rate of velocity increase in falling objects were manifestations of the same phenomenon, thus the simultaneous invention of the foundations of both calculus and physics
Dreams during which insights can be gained, often of a seemingly illogical nature, yet reportedly useful in life goal attainment
Sleep in between dreams where it is possibly that the day's queued observations are integrated with existing learned material or otherwise assimilated
Largely creative or intuitive, where models and rules are combined to create hypotheses to be tested through experimentation or through long periods of in situ observation (observation in natural environment)
Subconscious and related to archetypes

There are several approach to simulating these partially understood learning mechanisms in computers. It is also important to note that not all machine learning is a simulation of a mechanism of the human nervous system.  What arrangements of architectures, algorithms, parallelism, and interconnections optimally learn depend on many things.

The nature of the things to be learned (tennis, calculus, how to recognize a zip code in a mail sorter, someone's voice, someone's retina or fingerprint, how to identify social or story structure in a drama, self-improving real time trajectory prediction and antiaircraft aiming)
The mechanism of input and output (tabular, natural language, sound, visual, tactile, stepper motor signals)
The balance of the dimensions of the quality of intelligence to which the term optimal can be applied (accuracy, reliability, range of conditions for which learning is functional at all, relationship between complexity and speed of response capacity for complexity, depth of abstraction, wit in presentation of conclusions, listing of reference, auditability of thinking)
Hardware running the algorithms, their processes, and interconnecting them

How Learning Presumably Occurs

The current scientific theory (which is subject to change as more discovery occurs and paradigms shift) is that the human brain is programmed by the portions of DNA that define neural structures.  It is theorized that structural changes inside of and between neurons persist memories and establish pathways that lead to adaptations beyond instinctive neural activity and organism behavior, which is part of the DNA programming.

Organic or Digital Circuit Based Learning

For learning to occur in either a meat or silicon based system, there are a few requirements, the listing of which will give a general view into the answer to the question.

Way for the system to understand what is expected of it (real time acceptability/unacceptability/quality feedback, built in goal achievement detection, crisis detection, or multiple cases of conditions and expected results)
Way for the system to persist information needed to hold a representation of what had thus far been learned
Way for the system to acquire input or sense conditions
Way for the system to output results, execute decisions made, or impact conditions
Way to load algorithms into processes, interconnect them, and manage the overall system throughput
Hardware to support the above
Optionally, typical enterprise type features such as crash/disaster recovery, the ability to independently handle concurrent learning processes on a single system, or the ability to scale the system to handle larger problems or more concurrent learning processes

All of the above is dismissing entirely the possibility of concepts such as synchronicity, the metaphysical, or deity, none of which have been scientifically disproven.

Code

There are a few code bases that may help begin an investigation into learning using some of the most common and publicly available approaches.  Some of these have evolved to learn for specific problem sets.

https://github.com/mnielsen/neural-networks-and-deep-learning
https://github.com/kootenpv/neural_complete
https://github.com/rougier/neural-networks
https://github.com/jcjohnson/neural-style
https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers
https://github.com/uncomplicate/bayadera
https://github.com/timnugent/naive-bayes
https://sourceforge.net/projects/cpp-fuzzy-logic/
https://www.codeproject.com/Articles/316668/Cplusplus-Fuzzy-Logic-API-plus-Simple-DSL
https://github.com/abudnik/tcalc
https://github.com/metagol/metagol
https://sourceforge.net/projects/clipsrules/
http://openrules.com/
https://www.drools.org/

Truly Learning About Learning

Rather than running example code, it may be best to commit to actual learning about learning.  Downloading, compiling, configuring and running some existing POC is not your proof of concept.  It may be your starting point, but until you change those POCs to perform some other task you've devised and for which you've located sufficiently realistic data, you haven't learned anything about learning yet.

This is an example of a self-constructed lab project that wades right into the center of learning about learning software.

Develop an object oriented model of a tic-tac-toe game
Develop classes to represent the possible moves available for two opponents
Add an object oriented model of HOW the next move could be decided based on some arbitrary logic that can be mutated or permuted
Create some mechanism to try various HOWs in actual play
Create an algorithm that processes as input win-loose statistics and the
corresponding HOWs that converges on the optimal HOW for winning

Representing the tic-tac-toe three by three grid and the possible moves that players can make may only require a few dozen lines of code, but the simulation of human learning of the game (or some other form of learning the game that humans might not actually do) is not simple.

Even for the limited problem set of tic-tac-toe games, to write a program that actually learns the always-win-if-winning-is-possible algorithm by actually learning tic-tac-toe by playing it is complex and raises questions about random versus educated guessing worth investigation and about which many articles and books may be found and studied.

Developing such a program that learn successful tic-tac-toe play without knowing how to win in advance may take some brilliant thinking and days of programming, testing, experimenting, and re-thinking.  However, that will reveal more than reading articles, posts, books, or library documentation.

Writing a program that would deduce how to play tic-tac-toe from the rules of the game represented as predicate logic is a longer term project but may be rewarding and quite educational.


What is Bayes' theorem? What is Bayes' theorem? How does it relate to conditional probabilities?

Bayes' theorem relates conditional probabilities:

$$P(A \mid B) = \frac{P(B \mid A) P(A)}{P(B)}$$


How to create vector representation of roadmap like scans What would be the best way to create a vector representation of roadmap like scans? The goal I am trying to achieve is illustrated below. The left side represents the source image, the right side the output in the form of three vectors. The fussiness on the left is a simulation, not the actual source image:

The actual source image would look more like:

Currently I am looking at a combination of skeletonization and Hough transform. The result is rather messy though, and seems to warrant quite some extra engineering. Any other suggestions?

It turned out that my intuition was not far off. The skeletonization is a good step. The Hough transform though is not a good way to create a graph of the roadmap. It seems that the Ramer–Douglas–Peucker algorithm can help out here. This algorithm first takes all the skeleton pixels as input, and sees this as a starting graph. The algorithm then proceeds to remove intermediary pixels that do not add information to the shape of the graph.


Compute state space from variables in Q-learning (RL) I'm trying to use Q-learning, but I'm stuck because I don't know how to compute the state.
Let's say, in my problem, there are the following variables, which I'm using to compute state:
x in range 0-3
y in range 0-3
d in range 0-3
g in range 0-1
a in range 0-1
s in range 0-4
br in range 0-4
bu in range 0-4
gl in range 0-1
So, the state space is equal to $64000$ ($4 * 4 * 4 * 2 * 2 * 5 * 5 * 5 * 2$). I'd like to create a number, from the above variables, which is contained in the range $[0, 63999]$.
My previous idea was to create a binary number from the binary representation of state variables (just write them next to each other and convert into an int). It seems to fail if a variable is not a power of two (bonus question: why doesn't it work?).

It seems to fail if a variable is not a power of two (bonus question: why doesn't it work?).

This does not work because you are wasting some space, some values are not used in your representation.
For example with your 0-4 variables, you need 3 bits, but you only use 000, 001, 010, 011 and 100 values. The values 101, 110 and 111 are still part of the representation you are using. It doesn't matter that you don't need to use them, you have created a representation where they exist. Every time you encode 5 values as 3 bits, you are being 62.5% efficient, and the efficiencies multiply to get the overall efficiency of your representation (proportion of states you need to represent compared to the size of the representation).
Sometimes this is acceptable. Your example state representation using bitwise coding per variable would still fit easily into a 32 bit integer for storage, and you also have the convenience of easily extracting the individual values quickly using bit masks. It's a bit less convenient if you were hoping to build a Q table as an array with the state as a simple offset.
In your example case it is not super-wasteful, your array will be around 4 times larger than optimal due to wastage in 3 separate places. For convenience of coding simplicity you might accept this 8MB space per action over 2MB space per action (assuming 32 bit floating point values for action values). 6MB of dead space is not much to worry about - the chances are that the programming language you have loaded to run it wastes far more space on features that you are not using.
If you have a few more variables and a few more wasted bit patterns, then the waste could be more noticeable and important. That might also be true if you are using a space-efficient compiled language on a system where memory resources are low.

Let's say in my problem there are following variables which i'm using to compute state . . . So the state space is equal to 64000 (4 * 4 * 4 * 2 * 2 * 5 * 5 * 5 * 2)

You are very close to a working answer for the most efficient representation here. You can use products of each variable's size to separate terms, to create multiplication factors for each variable. Imagine building up a cuboid out of the first three terms, and needing to address each cubic building block sequentially with an index position $i$. You would do this:
$$i = x + 4y + (4 \times 4)d = x + 4y + 16d$$
If you had only thie first 5 variables you would do this:
$$i = x + 4y + (4 \times 4)d + (4 \times 4 \times 4)g + (4 \times 4 \times 4 \times 2)a = x + 4y + 16d + 64g + 128a$$
To cover your whole set of variables, keep extending the same pattern. Each variable added to the end of the list is multiplied by the product of the space required by all previous variables. The last variable in your example, gl, would be multiplied by $32000$.
Reversing this encoding is more involved, especially if you only want to access a single variable. You need to start with the value of $i$ then repeatedly use an integer divmod operation to find the next unknown variable at the end of the list - using the same multiplier for that variable as when constructing the code - and a remainder for calculating the next variable along.
Luckily in reinforcement learning you probably don't need to do any reversing of state id to individual variables - you can maintain a structure of all current variables for the state when running the environment, and only need this compressed version to create an offset into the Q table for lookups for the policy or updates to stored values.


How do I generate a feature representation of a saliency map (or mask)? Generally, CNNs are used to extract feature representations of an image. I'm right now dealing with the class of CNN that produces saliency maps, which are generally in the format of a mask. I'm trying to generate a feature representation of that specific Mask. What could be the best way to approach this problem?

I found a method to do it in the paper Cross-Modality Personalization for Retrieval (2020, accessed: 20-Feb-2020).

  Representation. For images, we extract Inception-v4 CNN features [36]. We then mask the image convolution feature with the BubbleView saliency map, by resizing the saliency map to the convolution feature size and multiplying them together. Finally, average pooling is performed to obtain a 1536-dimensional feature vector. We represent textual descriptions as their average 200-dimensional Glove embedding [31]. For personality, we use a 10-dimensional feature vector containing the scores for the personality questions in [32]. Below, we describe how we learn projections of these representations that place them in the same feature space.


How are training hyperparameters determined for large models? When training a relatively small DL model, which takes several hours to train, I typically start with some starting points from literature and then use a trial-and-error or grid-search approach to fine-tune the values of the hyper-parameters, in order to prevent overfitting and achieve sufficient performance.
However, it is not uncommon for large models to have training time measured in days or weeks [1], [2], [3].
How are hyperparameters determined in such cases?

In general, it is definitely very computationally expensive, so an exhaustive search is not performed in practice. However, there are some recent approaches for determining whether the architecture is "fine" without training the neural network first - by looking at the covariance matrix after forwarding the data, for example, in a recent paper Neural Architecture Search without Training. However, such an approach is very limited.


Why don't we decorrelate transitions for policy-based data? I'm implementing PPO myself strictly follow the steps:

sample transitions
randomly shuffle the sampled transitions
compute gradients and update networks using the sampled transitions
drop transitions and repeat the above steps

I observe a strange phenomenon that randomly shuffling transitions makes the algorithm perform significantly worse than keeping it as it is. This is very strange to me. To my best understanding, neural networks perform badly when the input data are correlated. To decorrelate transitions, algorithms like DQN introduce replay buffer and randomly sample from it. But this seems not the same story to policy-based methods. I'm wondering why policy-based methods do not require to decorrelate the input data?

We do decorrelate training experience, even for policy gradient methods. This is because decorrelation helps training data be more like IID data, which helps with the convergence of SGD-like optimizers.

The shuffling is done on line 151 of OpenAI's "baselines" implementation of PPO.

I'm going to guess that there's a bug somewhere in your implementation. If you're using the same truncated advantage estimation as in the PPO paper,

$$\begin{align}
&\hat{A}_t = \delta_t + (\gamma\lambda)\delta_{t+1}+\dots+(\gamma\lambda)^{T-t+1}\delta_{T-1}\\
&\text{where}\quad\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)
\end{align}$$

make sure you're not shuffling experience before computing your advantage estimates $\hat{A}_1,\dots,\hat{A}_T$. Each of these estimates is a function of several unbroken steps of experience. After you compute your advantage estimates, then create the tuples $(s_t, a_t, \hat{A}_t)$ and shuffle and sample from those. I think that's all the information you need  per time step for constructing their objective function.


Problem with fitness calculation in NEAT I am working on a project in which a drone is up to learn to fly. I am using NEAT.

For the first experiment I want it to learn how to hover inside a 3x3x3 meters box. My input is 6 sensors for each direction.  Output is same as in a drone so thrust (normalized to 0-1), aileron, rudder and elevator.

Initially just used a time as fitness, and after many generations it hovers inside the box.  It only really learns to use the thrust in function of up and down sensors, but it never learns to react to input from other sensors because they are not directly connected to fitness.

I would like to get some ideas about a good test for my problem. Should the drone be put to fly a track with obstacles? Should I have some more input data ? Should I define fitness to better reflect good reactions for input sensor data?

Thanks in advance

NOTE: the drone uses relatively accurate physics, I am able to do tasks using a controller.

Seem like time is a good fitness, though you need it to engage into learning side sensor inputs and side movement.

I would consider adding a bit of randomness to the environment. How about adding some random mild forces that might sway it left, right, front and rear a bit so that that bots are forced to use other sensors and inputs to stay in the center.

In cases when drone is not simulated, this task is a little harder but adding randomness to the environment can still be done. For example, tilt your drone a tiny amount in a random direction at random intervals. This will force your drone to learn to correct for being jostled by wind, without you having to actually produce wind.


Is it okay to think of any dataset in artificial intelligence as a mathematical set? A dataset is a collection of data points. It is known that the data points in the dataset can repeat. And the repetition does matter for building AI models.
So, why does the word dataset contain the word set? Does it have any relation with the mathematical set, where order and repetition do not matter?

It's true that your original dataset can contain duplicates, so it should not be called a set, in order to be consistent with the mathematical definition of a set. There are mathematical objects known as multi-sets that can contain duplicates, but the order of the elements is still not relevant. There are also tuples and sequences, where the order of the elements matters.
If you want to get rid of the duplicate elements in your dataset, you could perform a pre-processing step where you remove them. Even if you do that, it is often the case that, if you are learning with mini-batches (i.e. using mini-batch stochastic gradient descent), these mini-batches could contain the same elements, because you may sample the same element in different batches or even in the same batch (this is known as sampling with replacement). Of course, this depends on how you sample your training dataset to build the batches (or mini-batches). So, if you do not want duplicates even in the mini-batches, you need to perform sampling without replacement.
Moreover, there are datasets that contain elements whose order in the dataset can be relevant for the predictions, such as datasets of time-series data, while, in mathematical sets and multi-sets, the order of the elements does not matter.
So, yes, it is often called a dataset (or data set), but it is not necessarily a set in a mathematical sense. In general, it should just be interpreted as a collection of data. In scenarios where the order of the elements or the existence of duplicates in the dataset (or any other information or property of your collection of data) is relevant, you should probably emphasize/note it.


What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers? Consider a feedforward neural network. Suppose you have a layer of inputs, which is feedforward to a hidden layer, and feedforward both the input and hidden layers to an output layer.  Is there a name for this architecture? A layer feeds forward around the layer after it?

This could be called a residual neural network (ResNet), which is a neural network with skip connections, that is, connections that skip layers.

Here's a screenshot of a figure from the paper Deep Residual Learning for Image Recognition (2015), an important paper that shows the usefulness of these architectures.


Can a NN be configured to indicate which points of the input influenced its prediction and how? Suppose I want to classify a dataset like the MNIST handwritten dataset, but it has added distractions. For example, here we have a 6 but with extra strokes around it that don't add value.

I suppose a good model would predict a 6, but maybe with less than 100% certainty (or maybe with 100% certainty - I don't know that it matters for the purpose of this question).

Is there any way to get information about which pixels most strongly influenced the decision of the CNN, and which pixels were not so important? So to represent that visually, green means that those pixels were important:

Or conversely, is it possible to highlight pixels which did not contribute to the outcome (or which cast doubt on the outcome thereby reducing the certainty from 100%)

Yes there definitely is, and research into this has actually resulted in some really cool behaviour.

One of the simplest ways is to simply back propagate the gradient all the way back to the input. Areas of the input that affected the final decision will receive larger gradients. Interestingly, this also sort of works as a rudimentary form of semantic segmentation.

Other ways are to change segments of the input and to see how that affects the output (like add 0.1 to a pixel).

You can also determine what each filter in a convolutional layer is looking at using similar techniques.

It's a super interesting field of machine learning, and I highly recommend taking a look at this lecture that is completely free and one of the most interesting ones I've personally seen. it will explain all this much better than I have done.