Classifying generated samples with Wasserstein-GAN as real or fake I'm quite new to GANs and I am trying to use a Wasserstein GAN as an augmentation technique. I found this article
https://www.sciencedirect.com/science/article/pii/S2095809918301127,
and would like to replicate their method of evaluating the GAN. The method is shown in the figure.
In the article they write that they extract the generated samples that fooled the discriminator and use these to train a classifier. They also say that they use a Wasserstein GAN. Does anyone know how it is possible to extract samples that fooled the discriminator, since for a Wasserstein GAN the critic (discriminator) only puts a rating and not a label on the generated data?

My suggestion is to compare the critics rating for the real data and the generated data. Compute the mean of the ratings on real data to get some kind of threshold. If your critic is designed so that a high rating indicates a real sample, then generated data with ratings greater than the mean of the ratings of the real data could be viewed as good enough to fool the critic at that point in training.
This might be a too harsh criteria, but you will at least not accept poorly generated data.


Handle non-existing states in q-learning I am using Q-learning to solve an engineering problem. The objective is to generate a Q-table associating state to Q-values.

I created a State vector DS = [s1, s2, ..., sN] containing all "desired" states. So the Q-table has the form of Q-table =[DS, Q-values].

On the other hand, my agent follows a trajectory. Playing action a at state s (which is a point of the trajectory) leads the agent to another state s' (another point of the trajectory). However, I don't have the s' state in the initially desired states vector DS.

One solution is to add new states to the DS vector while the Q-learning algorithm is running, but I do not want to add new states.

Any other ideas on how to handle this problem?

You have two options, either interpolate or restrict the actions only to values that produce states which are in your state vector.

The simplest interpolation scheme is a linear interpolation, which works as follows (assuming DS contains a set of grid points in increasing order). For a state $s'$ you can locate its closest neighbours from the array DS and the value in state $s'$ will, then, be a weighted average of the values in those neighbouring states. Formally, $$Q(s') = \frac{s_{i+1} - s'}{s_{i+1}-s_i}Q(s_i) + \frac{s'-s_i}{s_{i+1}-s_i}Q(s_{i+1}),$$ for $s_i < s'< s_{i+1}$, where $s_i$ and $s_{i+1}$ are the neighbours, $s_i$ is the $i$-th element of DS(i.e. si = DS[i] = Q-table[i,1]), and $Q(s_i)$ is related to the Q-table as Qsi = Q-table[i,2] (asuming array indexing starts from 1).

Restricting the actions would work as follows. For simplicity, assume that the agent chooses the next state directly, i.e. $s'=a$. Then, if you have an array of actions A = [a1, a2, ... , aM], $M \le N$ then each $a_i$ needs to be present in the state array DS (i.e. A is a subset of DS, formally $A \subseteq S$). This may not be desirable but it is an option.


How is the DQN loss derived from (or theoretically motivated by) the Bellman equation, and how is it related to the Q-learning update? I'm doing a project on Reinforcement Learning. I programmed an agent that uses DDQN. There are a lot of tutorials on that, so the code implementation was not that hard.
However, I have problems understanding how one should come up with this kind of algorithms by starting from the Bellman equation, and I don't find a good understandable explanation addressing this derivation/path of reasoning.
So, my questions are:

How is the loss to train the DQN derived from (or theoretically motivated by) the Bellman equation?
How is it related to the usual Q-learning update?

According to my current notes, the Bellman equation looks like this
$$Q_{\pi} (s,a) = \sum_{s'} P_{ss'}^a (r_{s,a} + \gamma \sum_{a'} \pi(a'|s') Q_{\pi} (s',a')) \label{1}\tag{1} $$
which, to my understanding, is a recursive expression that says:
The state-action pair gives a reward that is equal to the sum over all possible states $s'$ with the probability of getting to this state after taking action $a$ (denoted as $P_{ss'}^a$, which means the environment acts on the agent) times the reward the agent got from taking action $a$ in state $s$ + discounted sum of the probability of the different possible actions $a'$ times the reward of the state, action pair $s',a'$.
The Q-Learning iteration (intermediate step) is often denoted as:
$$Q^{new}(s,a) \leftarrow  Q(s,a) + \alpha (r + \gamma \max_a Q(s',a') - Q(s,a)) \label{2}\tag{2}$$
which means that the new state, action reward is the old Q value + learning rate, $\alpha$, times the temporal difference, $(r + \gamma \max_a Q(s',a') - Q(s,a))$, which consists of the actual reward the agent received + a discount factor times the Q function of this new state-action pair minus the old Q function.
The Bellman equation can be converted into an update rule because an algorithm that uses that update rule converges, as this answer states.
In the case of (D)DQN, $Q(s,a)$ is estimated by our NN that leads to an action $a$ and we receive $r$ and $s'$.
Then we feed in $s$ as well as $s'$ into our NN (with Double DQN we feed them into different NNs). The $\max_a Q(s',a')$ is performed on the output of our target network. This q-value is then multiplied with $\gamma$ and $r$ is added to the product. Then this sum replaces the q-value from the other NN. Since this basic NN outputted $Q(s,a)$ but should have outputted $r + \gamma \max_a Q(s',a')$ we train the basic NN to change the weights, so that it would output closer to this temporal target difference.

The Bellman equation in RL is usually defined $$v_\pi(s) = \sum_a \pi(a|s) \sum_{s', r} p(s', r|s, a)\left[r + v_\pi(s')\right] = \mathbb{E}_{s' \sim p, a \sim \pi}\left[r(s, a) + v_\pi(s')\right] \; .$$
The way you have written it is correct, but I just thought I would point this out. Regardless, your intuition is correct in that it expresses a recursive relationship such that the value of your current state $s$ is equal to the sum of the expected reward from this state plus the expected value of the state you transition into.
You do, in fact, implement the Q-learning update in Deep Q-Learning. The loss function that you minimise in DQN is
$$ L(\theta) = \mathbb{E}_{(s,a,r,s')\sim U(D)}\left[\left( r + \gamma \max_{a'}Q(s', a'; \theta^-) - Q(s, a; \theta)\right)^2 \right]\;$$
where $U(D)$ denotes uniformly at random from replay buffer $D$ and $\theta$ are your network parameters (the network parameterises the Q-function), and $\theta^-$ are a previous iteration of the parameters that are updated every $c$ episodes to help with convergence of the network.
As you can see, the loss function is minimising the 'Bellman error' error from your equation 2. Lets think about why this is.
The TD update you provide is gradually shifting the Q value for $(s, a)$ towards $r + \max_a Q(s', a)$ - this is what we want after all since it eventually converges to the optimal Q-function.
Now lets think about the Deep Q-learning case. We want our network to approximate $Q(s, a)$ and so if we train the network, using the MSE loss, with $r + \max_a Q(s', a)$ as our target then our network will gradually be shifted towards predicting $r + \max_aQ(s', a)$ (which again would give us optimal Q-values for state-action pairs), just like with the TD update.
This is assuming that you know how training of neural networks works so if you don't then I would recommend asking/searching for a relevant question that explains this.


How can I use feature extraction in CNN with image segmentation? I'm just started to learn about meta learning and CNN and in most paper that I've read they mention to have one CNN to feature extraction. These features will help the another network.

I don't know what is feature extraction (I don't know what are those features) but I'm wondering if I can use it on image segmentation.

The idea is to use the first network to feature extraction without doing image classification, and pass those features to the other network.

My question is:
How can I use feature extraction in CNN on image segmentation?

Feature extraction is a way that people use pretrained model to extract information from input data. For example, image segmentation task may use the VGG network or other image classifying network for feature extraction. The output of the last convolution layer is taken. Then, the features are feed into the untrained network to get outputs. The bottom network for image segmentation usually consists of upsampling and convolutional layers. Then output of size of original image is resulted in teh main network. Hope I can help you


What does "reimplementations of deep learning algorithms which replicate performance from the papers" mean? In the OpenAI's Machine Learning Fellow position, it is written

We look for candidates with one or more of the following credentials:

...
Open-source reimplementations of deep learning algorithms which replicate performance from the papers

What exactly do they mean by this? Do they want us to implement the algorithms exactly as described in the papers (i.e. with the same hyper-parameters, weights, etc.)?

I believe that this:

"reimplementations of deep learning algorithms"
He is asking that they are looking for people who have made and completed an AI that perform similarly or exactly as those given in papers (IDK which papers)
"Open-source" The Fellow wants to be able to see the source code of the project.


How to tell a neural network that: "your i-th input is special" Assume that I have a fully connected network that takes in a vector containing 1025 elements. First 1024 elements are related to the input image of size 32 x 32 x 1, and the last element in the vector (1025-th element) is a control bit that I call it special input.
When this bit is zero, the network should predict if there is a cat in the image or not, and when this bit is one, it should predict if there is a dog in the image or not.
So how can I tell the network that your 1025-th element should be special to you and you should pay more attention to it?
Note that it's just an example and the real problem is more complex than this. So please don't bypass the goal of this question by using tricks special to this example. Any idea is appreciated.

Assume the image can contain objects of class $C_1 \dots C_c$. Assume a set of additional inputs that has a meaning of questions as "contains the image a C_i or C_j or ... ?".
The main problem for the system is classify the image in classes $C_i$. Second problem is answer the implicit question proposed by the remainder inputs.
Thus, better combine two NNs:

first one an object recognizer, with input the image data.
second one a NN that will answer the question implicit in remainder bits, with inputs the output of previous NN and the "question" bits.

In concrete for the example that the question describe:

a NN to classify dog/cat from 32x32 image. Two outputs for probabilities of dog an cat. Or simply one binary output 0:"is a dog", 1:"is a cat".
a second NN with the "1025" binary input ( 0: "look for dogs", 1: "look for cats") and output from previous one. In this case, if everything is ok, it will infer the logic "a==b": "is a dog" and "look for dogs" or "is a cat" and "look for cats".

Note that if you try to solve the problem directly with a full-connected NN with all inputs (1025 in the example), you loss the possibility of use CNN and max-pooling layers, etc. Moreover, split decreases training times, join subjects increases them "exponentially". Not a promising way.


What is the intuition behind the entropy formula used in the ID3 algorithm? What is the intuition behind the following entropy formula used in the ID3 algorithm?

$$
\text{info}(D) = -\sum_{i=1}^m p_i \log_2(p_i)
$$

Please, take a look at Understanding Shannon's Entropy metric for Information.

The answer for the minus sign is in section 6. The probability logs are less than or equal to $0$, so the minus sign guarantees that information (entropy) is always greater than or equal to $0$.


How do AIs like Siri and Alexa respond to their names being called? AIs like Siri and Alexa respond to their names being called. How does the system recognize the name by ignoring all the other words that have been said before their name? For example, "Hey Siri" would trigger Siri to start listening for commands, but if a user said "hey how are you hey Siri" the system will ignore "hey how are you" but trigger the system to "hey Siri". Is it because their listening function reloads in milliseconds or even nanoseconds, or is there a different way it works?

Is it because their listening function reloads in milliseconds or even nanoseconds

Yes, it expects the keyword to start every moment of time and it ignores the rest.
Overall, the algorithm is described here, you can read for details:
https://machinelearning.apple.com/research/hey-siri


RMSprop equation - dividing by a matrix? I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me.
$dW$ and $db$ are matrices (that's what I understand from the element-wise comment), so that must mean that $S_{dW}$ is a matrix (or tensor) as well.
So, in the update rule, they divide a matrix by another matrix? From what I saw in google, no such action exists.

Yes, you are correct, $S_{dW}$, in this case, is the matrix/tensor and it will have the same shape as gradient matrix/tensor. The update rule divides each element in gradient to each gradient in $S_{dW}$ or you can say "element-wise division".
To avoid the zero variable in $S_{dW}$, the equation usually like this:
$$
W=W-\alpha \frac{dW}{\sqrt{S_{dW}+\epsilon}} \text{  where } \epsilon=10^{-8}
$$
If you want to implement it manually with Python, you can take a look at this link


When is a knowledge base consistent? I am studying a knowledge base (KB) from the book "Artificial Intelligence: A Modern Approach" (by Stuart Russell and Peter Norvig) and from this series of slides.
A formula is satisfiable if there is some assignment to the variables that makes the formula evaluate to true. For example, if we have the boolean formula $A \land B$, then the assignments $A=\text{true}$ and $B=\text{true}$ make it satisfiable. Right?
But what does it mean for a KB to be consistent? The definition (given at slide 14 of this series of slides) is:

a KB is consistent with formula $f$ if $M(KB \cup \{ f \})$ is non-empty (there is a world in which KB is true and $f$ is also true).

Can anyone explain this part to me with an example?

I will first recapitulate the key concepts which you need to know in order to understand the answer to your question (which will be very simple, because I will just try to clarify what is given as a "definition").
In logic, a formula is e.g. $f$, $\lnot f$, $f \land g$, where $f$ can be e.g. the proposition (or variable) "today it will rain".  So, in a (propositional) formula, you have propositions, i.e. sentences like "today it will rain", and logical connectives, i.e. symbols like $\land$ (i.e. logical AND), which logically connect these sentences. The propositions like "today it will rain" can often be denoted by a single (capital) letter like $P$. $f \land g$ is the combination of two formulae (where formulae is the plural of formula). So, for example, suppose that $f$ is composed of the propositions "today it will rain" (denoted by $P$) or "my friend will visit me" (denoted by $Q$) and $f$ is defined as "I will play with my friend" (denoted by $S$). Then the formula $f \land g = (P \lor Q) \land S$. In general, you can combine formulae in any logically appropriate way.
In this context, a model is an assignment to each variable in a formula. For example, suppose $f = P \lor Q$, then $w = \{ P=0, Q = 1\}$ is a model for $f$, that is, each variable (e.g. $P$) is assigned either "true" ($1$) or "false" ($0$) but not both. (Note that the word model may be used to refer to different concepts depending on the context; again, in this context, you can simply think of a model as an assignment of values to the variables in a formula.)
Suppose now we define $I(f, w)$ to be a function that receives the formula $f$ and the model $w$ as input, and $I$ returns either "true" ($1$) or "false" ($0$). In other words, $I$ is a function that automatically tells us if $f$ is evaluated to true or false given the assignment $w$.
You can now define $M(f)$ to be a set of assignments (or models) to the formula $f$ such that $f$ is true. So, $M$ is a set and not just an assignment (or model). This set can be empty, it can contain one assignment or it can contain any number of assignments: it depends on the formula $f$: in some cases, $M$ is empty and, in other cases, it may contain say $n$ valid assignments to $f$, where by "valid" I mean that these assignments make $f$ evaluate to "true". For example, suppose we have formula $f = A \land \lnot A$. Then you can try to assign any value to $A$, but $f$ will never evaluate to true. In that case, $M(f)$ is an empty set, because there is no assignment to the variables (or propositions) of $f$ which make $f$ evaluate to true.
A knowledge base is a set of formulae $\text{KB} = \{ f_1, f_2, \dots, f_n \}$. So, for example, $f_2 = $ "today it will rain" and $f_3 = $ "I will go to school AND I will have lunch".
We can now define $M(\text{KB})$ to be the set of assignments to the formulae in the knowledge base $\text{KB}$ such that all formulae are true. If you think of the formulae in $KB$ as "facts", $M(\text{KB})$ is an assignment to these formulae in $KB$ such that these facts hold or are true.
In this context, we then say that a particular knowledge base (i.e., a set of formulae as defined above), denoted by $\text{KB}$, is consistent with formula $f$ if $M(\text{KB} \cup \{ f \})$ is a non-empty set, where $\cup$ means the union operation between sets: note that (as we defined it above) $\text{KB}$ is a set, and $\{ f \}$ means that we are making a set out of the formula $f$, so we are indeed performing an union operation on sets.
So, what does it mean for a knowledge base to be consistent? First of all, the consistency of a knowledge base $\text{KB}$ is defined with respect to another formula $f$. Recall that a knowledge base is a set of formulae, so we are defining the consistency of a set of formulae with respect to another formula.
When is then a knowledge base $\text{KB}$ consistent with a formula $f$? When $M(\text{KB} \cup \{ f \})$ is a non-empty set. Recall that $M$ is an assignment to the variables in its input such that its inputs evaluate to true. So, $\text{KB}$ is consistent with $f$ when there is a set of assignments of values to the formulae in $\text{KB}$ and an assignment of values to the variables in $f$ such that both $\text{KB}$ and $f$ are true. In other words, $\text{KB}$ is consistent with $f$ when both all formulae in $\text{KB}$ and $f$ can be true at the same time.


Could clustering be used to parse pdf documents to get headings and titles? I'm a bit new to AI and I'd like to use some kind of clustering algorithm to solve a problem:

I'm trying to parse pdf documents to get headings and titles. I can parse pdf to html and I'm then able to get some information on the lines of the document. I've identified some properties that can be useful for identifying the headings.

font-size (int): of course it's quite usual that heading's font-size is bigger than normal text
font-family (string): it's possible for headings to be  bold so font-family may differ
left property (int): it's also possible that headings are aligned a bit to the right, there's an indentation that's not always there on normal paragraphs
bonus boolean: I have identified some properties that I can combine to get a boolean value. When the boolean is set to true it can increase the chances of the paragraph being a heading.

Of course, these are not rules that apply to all headings. Some headings may follow some of these but not all of them. It could also be possible that some 'normal' paragraphs follow all these points, but what I've seen is that, in general, those rules where what made headings different from paragraphs.

With this information, is there a way of doing what I'm looking for? As I said, I'm new to AI even though I have a background in CS and mathematics. I thought clustering could be interesting since I'm trying to create 2 clusters: headings and normal paragraphs.

What algorithm do you think might work for this use case. Should I look outside clustering?

Yes, you could use clustering: Encode your features as a feature vector and feed it into a clustering algorithm (see Finding Groups in Data for a comprehensive description of these). You could use agglomerative clustering, which would give you groups of similar items; perhaps different level headings will be clustered together.

Alternatively you could try a decision tree, something like ID3, which would also be suitable; for this you'd need some annotated training data, though. But with a small amount of data you might solve it, if your items are clearly separated.


Which algorithm should I use to map an input sentence to an output sentence? I am new to NLP realm. If you have an input text "The price of orange has increased" and output text "Increase the production of orange". Can we make our RNN model to predict the output text? Or what algorithm should I use?

In your case you can use either RNN (especially BiLSTM with ELMo and attention mechanism for better accuracy) or Transformer based architectures (the best of them today is BERT). But for both cases you need data to train the model (i.e sequences of input/output like in your question).

I believe the best choice in your case is BERT as it's achieving state of the art performance in most NLP tasks and is already pretrained so you don't need a massive data to retrain the model. Also, BERT is pretrained on "Next Sentence Prediction" and allows "2 separate sentences" as input which helps a lot in your case. The only drawback, compared to other methods, is that you need fine-tuning the model so it's not a completly "ready to use" model.

For more information:

Here is the github repository of BERT: BERT-repo. For quick explanation check: The Illustrated BERT. For detailed explanation see BERT paper: BERT-paper.


Heavy loss and inaccurate answer in pytorch As my first AI model I have decided to make an AI model to predict multiplication of two numbers EX - [2,4] = [8]. I wrote the following code, but the loss is very high, around thousands, and it's very inaccurate. How do I make it more accurate?

import torch
import torch.nn as nn
import torch.nn.functional as F

data = torch.tensor([[2,4],[3,6],[3,3],[4,4],[100,5]],dtype=torch.float)
values = torch.tensor([[8],[18],[9],[16],[500]],dtype=torch.float)
lossfun = torch.nn.MSELoss()
model=Net()
optim = torch.optim.Adam(model.parameters(),lr=0.5)

class Net(nn.Module):
    def __init__(self):

        super(Net,self).__init__();

        self.fc1 = nn.Linear(in_features=2,out_features=3)
        self.fc2 = nn.Linear(in_features=3,out_features=6)
        self.out = nn.Linear(in_features=6,out_features=1)

    def forward(self,x):
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.out(x)
        return x
for epoch in range(1000):

    y_pred=model.forward(data)

    loss = lossfun(y_pred,values)

    print(loss.item())

    loss.backward()

    optim.step()

Note: I am a newbie in AI and ML.

There are a few things you could do to improve this NN, but are probably worth covering in different questions.

Your main problem though is that you forgot to reset the gradient after each training batch. You need to call optim.zero_grad() in order to do this, at the start of each training loop. Otherwise, using PyTorch, the gradient values keep accumulating inside the model's training data (sometimes you want this effect if you are adding gradients from multiple sources, that's why PyTorch is not clearing them automatically for you).

In addition, a learning rate of 0.5 is very high for the Adam optimiser - it is very common to leave it at the default value because Adam is an adaptive optimiser that will adjust step sizes depending on gradients seen so far.

Here is a working version of your code:

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):

        super(Net,self).__init__();

        self.fc1 = nn.Linear(in_features=2,out_features=3)
        self.fc2 = nn.Linear(in_features=3,out_features=6)
        self.out = nn.Linear(in_features=6,out_features=1)

    def forward(self,x):
        x = self.fc1(x)
        x = torch.relu(x)
        x = self.fc2(x)
        x = torch.relu(x)
        x = self.out(x)
        return x

data = torch.tensor([[2,4],[3,6],[3,3],[4,4],[100,5]],dtype=torch.float)
values = torch.tensor([[8],[18],[9],[16],[500]],dtype=torch.float)

lossfun = torch.nn.MSELoss()
model=Net()
optim = torch.optim.Adam(model.parameters(),lr=0.001)

for epoch in range(20000):
    optim.zero_grad()
    y_pred=model.forward(data)
    loss = lossfun(y_pred,values)
    if (epoch % 1000 == 0):
        print(loss.item())
    loss.backward()
    optim.step()

This version can be tweaked to quite easily reach 0 loss for your data set.

This has not in really learned how to multiply two values. The approximation to multiplying will be very weak as there is very little data. However, playing with some very basic data and a simple NN is a first step towards understanding details like this . . .


How to detect when human voice / speech appears in an microphone stream? I want to build a personal assistant that listens to me continuously..

The flow looks like this:

continuously record voice
stream it to google speech api.
get back the text in real time -> parse for intent etc..

Problem is, google speech api gets expensive if you record for hours. A better way to do it is to only submit the parts where I'm actually speaking to it.. Then the cost of running this full time (17 hours a day, every day) becomes very accessible.
Now my question is:

How can i detect that a voice is present in the microphone stream?

I have a lot of noise in the background, dumb increase in volume detection is not a very good solution. I need something more intelligent. It doesn't need to be very accurate - just good enough to not break my cloud computing budget. I'm thinking that human voice sounds distinct enough that is not such a big problem to detect when is there.

What do you recommend me to do, given this is a real time stream - not an audio file.

The audio will be generated in chromium browser (electron) - with getUserMedia API, and using node.js i plan to handle the streaming logic.

Note: there is a built in speechRecognition api in electron - but from my experience currently it doesn't work (not even after i give it my API key), and even if would have worked, i think it has the same cost problem. So this is why i'm trying to provide my own implementation.

I don't know what i'm doing, any insight is welcomed:) Thank you.

Your problem is an old one. There are many methods, referred to as Voice Activity Detection (VAD) methods, which detect speech from an audio signal.

The typical design of a VAD algorithm follows one of more of these 3 approaches:

A noise reduction stage, e.g. via spectral subtraction.
Some features or quantities are calculated from a section of the input signal.
A classification rule is applied to classify the section as speech or non-speech – often this classification rule finds when a value exceeds a threshold.

There may be some feedback in this sequence, where parameters (like the noise threshold or the classification threshold) are tuned to improve the estimate or to adaptively vary the threshold(s). These feedback operations improve the VAD performance in non-stationary noise (i.e. when the noise varies a lot).

Check out this wiki page on Voice Activity Detection for more information.


Convolution layer neurons when extracting multiple feature maps I've recently been reading up on CNNs and this part of the architecture is really confusing me.
Assume, I have an input of size [32*32*3] and pass it to a convolution layer. Now, if my kernel size were to be [5*5*3] and the depth of my convolution layer were to be 1, only one feature map would be produced for the image. Here, each neuron would have a 75 weights (+1 bias). If I wanted to calculate multiple feature maps in this layer, say 3, is each local section (in this example [5*5*3]) of the image looked on by three different neurons and each of their weights trained individually? And what would be the output volume of this layer?

Each feature map (or kernel) is independent of each other. If you had $3$ of these filters, your output shape would be $(28, 28, 3)$ (given the appropriate amount of padding and stride) with a total of $75*3=225$ trainable weights.


Why is the expectation calculated over finite number of points drawn from a probability distribution? This is from the book Pattern Recognition by Bishop. Why is expectation here a simple average? Why is $f(x)$ not being multiplied by $p(x)$?

When we say that we have $N$ points that were "drawn from the probability distribution or probability density", this means that every point $x_n$ had the correct probability $p(x_n)$ of being sampled from the distribution when we were sampling our $n^{th}$ point.

For example, suppose that we wish to compute/estimate the expected value of a distribution given by flipping a coin, which is weighted such that it lands on heads two-thirds of the time (value of heads $= 1$), and tails one-thirds of the time (value of tails $= 0$). This means that the ground-truth probabilities are:

$p(1) = \frac{2}{3}$
$p(0) = \frac{1}{3}$

In this case, we actually wouldn't have to estimate anything by sampling; the exact probabilities are known, so we could just compute the expected value as $p(1) \times 1 + p(0) \times 0 = \frac{2}{3}$.

But now suppose that we do not know exactly how the coin is weighted, i.e. we do not know the exact values of $p(1)$ and $p(0)$. If we simply flip our weighted coin $N = 100$ times, we will in expectation find about $67$ samples of $x_n = 1$, and about $33$ samples of $x_n = 0$ (rounded to integers because we can't obtain a non-integer number of observations). So we cannot explicitly use the probabilities (because we do not know them), but they will implicitly be present in the number of repeated observations we have for every possible datapoint.


Which problems in AI are not machine learning? Which problems in AI are not machine learning? Which problems involve both AI and machine learning?

This is a common question and it has already been answered in many different places:

What is the difference between artificial intelligence and machine learning?

Artificial Intelligence vs. Machine Learning vs. Deep Learning

  Artificial intelligence is a broader concept than machine learning, which addresses the use of computers to mimic the cognitive functions of humans. When machines carry out tasks based on algorithms in an "intelligent" manner, that is AI. Machine learning is a subset of AI and focuses on the ability of machines to receive a set of data and learn for themselves, changing algorithms as they learn more about the information they are processing.

More specific to your question:

AI without machine learning

  If you insert a small amount of knowledge into a machine, you can call it an engineering product. But if you instill a sufficiently large amount of knowledge such that the machine makes better decisions than a human, that can be could AI. For example, if you take hundreds of medical doctors and each spends hundreds of hours detailing correlations between symptoms and disease, if you then pack that knowledge into an easy-to-use machine, that's AI without machine-learning


Simple text recognition with neural network In my attempt at trying to learn neural network and machine learning I'm am trying to create a simple neural network which can be trained to recognise one word from a given string (which contains only one word). So in effect if one where to feed it a string containing the trained word but spelled wrong the network would be able to still recognise the word. Can anybody help me with some pseudo code or a start of a code. Or a general explanation of how to to this because I have read like 6 articles and 8 example projects and still have no clue how to do this

If I'm reading it correctly, this question has nothing to do with optical character recognition. You want to create a system that takes a digital string of characters as input, then finds the best match from a predetermined list of words. That sounds like a task for if-then-else logic and dictionary lookup. It might be possible to use a neural net, but not easy.

A neural net takes a fixed number of inputs, each of which are a value between zero and one. A major hurdle is that you probably want variable-sized inputs. Another hurdle is that you'll need to code the inputs some way onto numbers.

These hurdles can be overcome but they are tipoffs that neural networks aren't well-suited for the task.


Why do we sample vectors from a standard normal distribution for the generator? I am new to GANs. I noticed that everybody generates a random vector (usually 100 dimensional) from a standard normal distribution $N(0, 1)$. My question is: why? Why don't they sample these vectors from a uniform distribution $U(0, 1)$? Does the standard normal distribution has some properties that other probability distributions don't have?

It has become our human bias that data will arrive from a normal distribution. It is also the most prevalent distribution in nature occurring in many places. Hence, we sample from a normal distribution. Also, central limit theorem works around means lying around normal distribution.
It is not taboo to use others if they are helpful to your network. But, a data from one distribution can transformed into other and if that is something that is required, the network will to do it since it can approximate anything (although a weak assumption, but, hey, its working right?).


What happens when an opponent a neural network is playing with does not obey the rules of the game (i.e. cheats)? For example, if AlphaZero plays with an opponent who has a right to move chess figures any way she wants, or make more than 1 move in a turn? Will a neural network adapt to that, as it adapted to an absurd move made by Lee Sedol in 2015?

The behaviour when playing against "cheats" depends on how the agent has been trained, and how different the game becomes from the training scenarios. It will also depend on how much of the agent's behaviour is driven by training, and how much by just-in-time planning.
In general, unless game playing bots are written specifically to detect or cope with opponents that are given unfair advantages, they will continue to play in the same style as if the cheating had not occurred, and assuming that the rules are still being followed strictly. If the cheating player only makes one or two rules-breaking moves, and the resulting game state is still something feasible within the game, then the agent should continue to play well. If the agent significantly outclasses the human opponent, it may still win.
A completed, trained agent will not adapt its style to "now my opponent can cheat". An agent still being trained could do so in theory, but it would take many games with cheating allowed for it to learn tactics that cope with an opponent that had an unfair advantage.
Agents that plan by looking ahead during play can cope with more unusual/unseen game state - things that may not have been seen in training. However, they still look ahead on the assumption that game play is as desiged/trained for, they cannot adapt to new rules unless those rules are added to the planning by the bot designers. For instance if the allowed cheating was a limited number of extra moves, but only for the human player, the effects of that could be coded into the planning engine, and the bot would "adapt" with help from its designers.

[AlphaGo] adapted to an absurd move made by Lee Sedol in 2015?

Assuming you are referring to game 4, then as far as I know, AlphaGo did not "adapt" to this play, after Lee Sedol managed to put it in a losing position then it started playing badly as it could not find a winning strategy from the board positions it was in, and could not recover. I don't think any effort was put into refining AlphaGo during this game or afterwards to patch it for game 5.


Concrete examples of OpenCog's functionality Does anyone know what specific tasks the OpenCog environment is capable of performing? I have glanced though their wiki and a few of the pages on Goertzel's site and the AI.SE. So far I could only find some technical documentation regarding theory and engineering, but nothing on concrete results.

From the technical description of AtomSpaces it seems that OpenCog is capable of some "representational inference", but I haven't come across any sources that concretely describes what it is capable of doing.

Apparently there is some collaboration between Sophia the Robot and OpenCog, but to what extent I am unclear. I am aware however that the dialogue functions is powered by ChatScript (though I also suspect that the high profile interviews Sophia gives are completely scripted...)

Can anyone provide concrete examples or evidence of OpenCogs' functional behavior. Like transcripts of chat, examples of reasoning, video or demonstrations of its emotion-emulating; and not just claims of functions.

It depends what you mean by "what OpenCog can do?".

OpenCog, at a high-level, is a loosely coupled collection of various theoretical and variants of conventional methods aimed at constructing the beginnings of an AGI.

With that said, it's purely applied uses are fairly limited. It can do some typical NLP and ML tasks if used correctly, albeit, it will almost always be ineffective when compared to a problem specific solution. However, this is understandable, as OpenCog is not geared at solving narrow problems.

OpenCog, as well as Goertzel's other projects(SingularityNet, AGI society), are broad, top-down attempts at formulating an AGI system.
Outside of AGI circles they are also not oft referenced or researched(although, I am utilizing some aspects of the system in work I am currently crafting).


Why did "Artificial Intelligence" stay intact as a coherent, unified field of study? Artificial Intelligence is a rather pernicious label to attach to a very mixed bunch of activities, and one could argue that the sooner we forget it the better. It would be disastrous to conclude that AI was a Bad Thing and should not be supported, and it would be disastrous to conclude that it was a Good Thing and should have privileged access to the money tap. The former would tend to penalise well-based efforts to make computers do complicated things which had not been programmed before, and the latter would be a great waste of resources. AI does not refer to anything definite enough to have a coherent policy about in this way.---Dr. R. M. Needham, in a commentary on the Lighthill Report and the Sutherland Reply, 1973

43 years later...

  There is already strong demand for engineers and scientists working on artificial intelligence in many of the fields you mention, and many more. But expertise in making real-time systems for controlling trains doesn't make you know anything about robotics. Analyzing human behavior to detect crime has virtually nothing in common with self-driving cars (beyond CS/pattern recognition building blocks). There is never going to be demand for someone with a broad sense of all these areas without any deep expertise, and there is never going to be someone with 300 PhDs who can work in all of them. TL;DR -- AI is not a branch, it's a tree. --Matthew Read, in a comment on Area 51 Stackexchange, 2016

AI is a label that is applied to a "very mixed bunch of activities". The only unifying feature between all those activities is the fact that they deal with machines in some fashion, but since there are so many ways to use a machine, the field's output may seem rather incoherent and incongruent. It does seem to make more sense for the AI field to collapse entirely, and instead be replaced by a multitude of specialized fields that don't really interact with one another. Sir James Lighthill appeared to have supported this sort of approach within his 1973 report on the state of artificial intelligence research.

Yet, today, this Artificial Intelligence SE exist, and we still talk of AI as a unified, coherent field of study. Why did this happen? Why did AI survive, despite its "big tent" nature?

AI is a rather unusual research field in that the label persists more because it represents a highly desired goal, rather than (as with most other fields) the means, substrate or methodology by which that goal is achieved.

  we still talk of AI as a unified, coherent field of study

Despite recent efforts in AGI, I don't think that AI is actually a very unified or coherent field. This is not necessarily a bad thing - when attempting to mimic the most complex phenomenon known to us (i.e. human intelligence) then multiple, sometimes seemingly conflicting perspectives may be our best way of making progress.


When do SARSA and Q-Learning converge to optimal Q values? Here's another interesting multiple-choice question that puzzles me a bit.

In tabular MDPs, if using a decision policy that visits all states an infinite number of times, and in each state, randomly selects an action, then:

Q-learning will converge to the optimal Q-values
SARSA will converge to the optimal Q-values
Q-learning is learning off-policy
SARSA is learning off-policy

My thoughts, and question: Since the actions are being sampled randomly from the action space, learning definitely seems to be off-policy (correct me if I'm wrong, please!). So that rules 3. and 4. as incorrect. Coming to the first two options, I'm not quite sure whether Q-learning and/or SARSA would converge in this case. All that I'm able to understand from the question is that the agent explores more than it exploits, since it visits all states (an infinite number of times) and also takes random actions (and not the best action!). How can this piece of information help me deduce if either process converges to the optimal Q-values or not?
Thanks a lot!

Source: Slide 2/55

The true answers are 1 and 3. 1 because the required conditions for tabular Q-learning to converge is that each state action pair will be visited infinitely often, and Q-learning learns directly about the greedy policy, $\pi(a|s) := \arg \max_a Q_\pi(s,a)$,  and because Q-learning converges to the optimal Q-value function we know that the policy will be optimal (because the optimal policy is the greedy policy wrt the optimal Q-function).
3 is true because Q-learning is by definition an off-policy algorithm, because we learn about the greedy policy whilst following some arbitrary policy.
2 is false because SARSA is on-policy, so it will be learning the Q-function under the random policy, and 3 is false because SARSA is strictly on-policy, for reasons analogous to why Q-learning is off-policy.


Why are GRU and LSTM better than standard RNNs? It seems that older RNNs have a limitation for their use cases and have been outperformed by other recurrent architectures, such as the LSTM and GRU.

These newer RNNs (LSTMs and GRUs) have greater memory control, allowing previous values to persist or to be reset as necessary for many sequences of steps, avoiding "gradient decay" or eventual degradation of the values passed from step to step. LSTM and GRU networks make this memory control possible with memory blocks and structures called "gates" that pass or reset values as appropriate.


Confusing on GAN loss function I was trying to understand the loss function of GANs, while I found a little mis-match between different papers.

This is the screen-shot from the original paper of Goodfellow at https://arxiv.org/pdf/1406.2661.pdf: ,

And equation (1) in this version of pix2pix paper at https://arxiv.org/pdf/1611.07004.pdf

Putting aside the fact that pix2pix is using conditional GAN, which introduces a second term $y$, the 2 formulas are quite resemble, except that in the pix2pix paper, they try to get minimax of ${\cal{L}}_{cGAN}(G, D)$, which is defined to be $E_{x,y}[...] + E_{x,z}[...]$, whereas in the original paper, they define $\min\max V(G, D) = E[...] + E[...]$.

I am not coming from a good math background, so I am quite confused. I'm not sure where the mistake is, but assuming that $E$ is expectation (correct me if I'm wrong), the version in pix2pix makes more sense to me, although I think it's quite less likely that Goodfellow could make this mistake in his amazing paper. Maybe there's no mistake at all and it's me who do not understand them correctly.

What is meant by both papers is that we have two agents (generator and discriminator) playing a game with the value function V defined as a sum of the expectations (i.e. an expectation of the outcome value defined as a sum of two terms, or actually a logarithm of a product...). The generator uses a strategy G encoded in the parameters of its neural network (θg), the discriminator uses a strategy D encoded in the parameters of its neural network (θd). Our goal is to (hopefully) find such a pair of strategies (a pair of parameter sets θgmin and θdmax) that produce the minimax value.

While trying to find the (θgmin, θdmax) pair using gradient descent, we actually have two loss functions: one is the loss function for G, parameterized by θg, another is the loss function for D, parameterized by θd, and we train them alternatively on minibatches together.

If you look at the Algorithm 1 in the original paper, the loss function for the discriminator is -log(D(x; θd)) - log(1 - D(G(z); θd), and the loss function for the generator is log(1 - D(G(z; θg)) (in both cases, in the original paper, x is sampled from the reference data distribution and z is sampled from noise):

The ideal value for the loss function of the discriminator is 0, otherwise it's greater than 0. The "loss" function of the generator is actually negative, but, for better gradient descent behavior, can be replaced with -log(D(G(z; θg)), which also has the ideal value for the generator at 0. It is impossible to reach zero loss for both generator and discriminator in the same GAN at the same time. However, the idea of the GAN is not to reach zero loss for any of the game agents (this is actually counterproductive), but to use that "double gradient descent" to "converge" the distribution of G(z) to the distribution of x.


If the Turing test is passed, does this imply that computers exhibit intelligence? Turing test was created to test machines exhibiting behavior equivalent or indistinguishable from that of a human. Is that the sufficient condition of intelligence?

We don't know.

However, an important line will have been crossed - it will be impossible to tell the difference between an intelligent agent and the machine by use of a text interface. Which is the main point of the test - "if it quacks like a duck".

It is also an important philosophical point. Whether intelligence is defined purely by behaviour in an environment, or by the mechanisms that arrive at that behaviour. A suitably large database of conversational openers and "correct" responses can in theory mimic a lot of real world conversations. Some chatbots take advantage of this and use modern computer capacity to store a lot of responses, and that approach has gained competitive scores in the Loebner prize competition (although not to the stage of actually passing the test). This leads us to the Chinese Room issue, and wondering which part of the system is actually intelligent, or even how much of human conversation is actually intelligent or meaningful (and it what ways).


Will quantum computing have any kind of effect on the development of AI? Recently, according to some reports Google achieved something called 'Quantum Supremacy'. Whether its true or not remains to be seen.

But my question is does Quantum Computers or the principle they work on has any effect on AI. We know that according to Shor's Algorithm Quantum Computing can reduce the complexity of breaking an encryption from $O(2^n)$ to $O(2^{\frac{n}{2}})$ (not sure).

So similar to this, will Quantum computing have any kind of effect in accelerating AI training or something similar?

Preface

  "Before answering this question, let me preface by stating that the
  following is simply MY answer as a Machine Learning Researcher and
  "Hobbyist" Theoretical Physicist, although I have strong feelings that
  my answer will most certainly be proven as true, I am more than sure
  that others will have differing opinions as with everything else in
  this world!" ~ Me

Answer

That being said, I would say that the answer to your question, is unequivocally YES! I could write a short book on the subject (that might be a good idea actually) but I will keep this response brief, although I am happy to answer any further questions your may have to which I possess the answers in the comments!

The primary reasons that I believe the newly blossoming field of Quantum Computation will have a massive impact on the field of Machine Learning in general, are as follows:

The most simple reason for my belief is that the primary goal of Machine Learning, is to create an entity which is capable of coherent, self-aware thought much like we exhibit as human beings. We know that the brain is what allows us to be capable of such feats, and thus I view the field something like brain counterfeiting. Without going into esoteric detail, there are many subtleties of the brain's workings which are thought to be quantum mechanical in operation, and thus would suggest that the path of least resistance to replicating the system would require a quantum mechanical computational medium.
The second primary rationale which solidifies my position, is the point which you briefly touched upon by stating that,

  "We know that according to Shor's Algorithm Quantum Computing can reduce the complexity > of breaking an encryption from O(2n) to O(2n2) (not sure)."

This is primarily due to the quantum phenomena referred to as Super Position, which allows for a multiple qubit gate to not only work with the options of 00, 01, 10 and 11 (assuming a two qubit gate) but to also work with any combination in-between, during the computation. Although when the result is obtained (this is what is referred to as collapsing the wave function) you will only still have a resulting state space  with 2^n possibilities, where n is the number of qubits. This being said, there are very clever ways, by which one can design their algorithms to make full use of this technically infinite computational space before observing the final results.

Conclusion

I hope that my answer is helpful to you in some way, although I am aware that it is not a very in-depth answer, I feel that it hits the primary reasons why my personal belief is that there will be a wall which is hit, in the pursuit of a general AI, while we are limited to classical computation faculties; and thus will require quantum based computation before we are able to truly mimic the brain's most well kept secrets! The next couple decades should be VERY interesting in the fields, keep a close eye on the latest happenings!


What is the difference between the heuristic function and the evaluation function in A*? I am reading college notes on state search space. The notes (which are not publicly available) say:

To do state-search space, the strategy involves two parts: defining a heuristic function, and identifying an evaluation function.

The heuristic is a smart search of the available space. The evaluation function may be well-defined (e.g. the solution solves a problem and receives a score) or may itself be the heuristic (e.g. if chess says pick A or B as the next move and picks A, the evaluation function is the heuristic).

Understand the difference between the heuristic search algorithm and the heuristic evaluation function.

I'm trying step 3 (to understand). Can I check, using the A* search as an example, that the:
Heuristic function: estimated cost from the current node to the goal, i.e. it's a heuristic that's calculating the simplest way to get to the goal (in A*; $h(n)$), so the heuristic function is calculating $h(n)$ for a series of options and picking the best one.
Evaluation function: $f(n) = g(n) + h(n)$.

What is the difference between the heuristic function and the evaluation function in A*?

The evaluation function, often denoted as $f$, is the function that you use to choose which node to expand during one iteration of A* (i.e. decide which node to take from the frontier, determine the next possible actions and which next nodes those actions lead to, and add those nodes to the frontier). Typically, you expand the node $n$ such that $f(n)$ is the smallest, i.e. $n^* = \operatorname{argmin}f(n)$.
In the case of informed search algorithms (such as A*), the heuristic function is a component of $f$, which can be written as $f(n) = g(n) + h(n)$, where $h(n)$ is the heuristic function. The heuristic function estimates the cost of the cheapest path from $n$ to the goal. Just for completeness, $g(n)$ is the actual cost from the start node to $n$ (which can be computed exactly during the search). In the case of uninformed search algorithms, you can actually view the evaluation function as just $f(n) = g(n)$, i.e. the heuristic function is always zero.
So, you're right.
For more details, you can take a look at section 3.5 (p. 92) of the book Artificial Intelligence: A Modern Approach (3rd edition), by Norvig and Russell (you can find the pdf online).


Can OpenAI simulations be used in real world applications? I know that classical control systems have been used to solve the problem of the inverted pendulum - inverted pendulum.

But I've seen that people have also used machine learning techniques to solve this nowadays - machine learning of inverted pendulum.

I came across a video on how to apply a machine learning technique called reinforcement learning on openAI gym - OpenAI gym reinforcement learning.

My question is, can I use this simulation and use it to train a controller for a real-world application of inverted pendulum?

In general, you can use a simulation to prepare and train a controller for a real world application. A good example of this being done for robotics is in the paper Autonomous helicopter flight via reinforcement learning where a Reinforcement Learning agent was trained on a model of helicopter dynamics before being used in reality. Often, as in this case, such work is done to avoid expensive failures due to the trial and error nature of RL - if an error is expensive, such as crashing a helicopter, then ideally the agent performs the checks to avoid it in simulation, by planning or some other virtual environment as opposed to in the real world.

The main hurdle to completing training in simulation then transfering to the real world, is the fidelity of the simulation. The simulation's physics, including measurements of physical quantities, the size of time steps, amount of randomness/noise, should match between the simulation and the target real-world environment. If they do not match, then a learning agent could generate a policy that works in simulation, but that fails in reality.

For the autonomous helicopter, the researchers used data from a human operator controlling the real helicopter, to help generate a predictive model that was used in the simulation.

Can you do the same with Open AI Gym environments? Probably not, unfortunately. The main issue is that the units used are fixed in most environments, and are unlikely to closely relate to any specific real world implementation of the same kind of system. In addition, the physics is often simplified - probably a minor issue for CartPole, but a more major one for environments like LunarLander which ignores weight of fuel used and is a 2D simulation of a 3D environment.

So, for instance, in CartPole environments, the following values are fixed:

Size of time step
Mass of cart
Mass and length of pole to be balanced
Force that cart motor pushes with

There are a couple of approaches you could use to work around this:

Make a new version of the environment and adjust it so that values match to a real environment you want to train for. Note this may still be limited, as the physics model is still quite simple, and may not allow for the real operating characteristics of the cart motor.
Use the CartPole environment as-is, not to train a controller directly, but to select hyper parameters, such as neural network size, learning rate etc. That will result in a learning agent that you are reasonably confident can learn policies with the state representation and general behaviour of your target system. You then train "for real" again in the physical system.

You can combine these ideas, creating a best-guess controller from simulation, then refining it in a real environment by continuing the training on a real system.


Is artificial intelligence really just human intelligence? Essentially, AI is created by human minds, so is the intelligence & creativity of algorithms properly an extension of human intelligence & creativity, rather than something independent?

I assume that intelligence does not necessarily require creativity, however, creativity can result from machine learning.  (A simple example is AlphaGo discovering novel strategies.)

This is an old question, going back at least to 1950. It is one of the original objections to AI that Turing considers and attempts to refute in his seminal 1950 paper Computing Machinery and Intelligence.

Turing actually attributes this objection to Lady Lovelace, apparently quoted by another author. In Turing's paper, this is objection #6: Lady Lovelace's Objection, in section 6 of the paper. The objection is concisely stated as

  The Analytical Engine has no pretensions to originate anything. It can
  do whatever we know how to order it to perform.

where "The Analytical Engine" was an early design for an all-mechanical general purpose computer.

Turing offers two replies to this objection. First, he reminds us that computer programs have bugs. That is, they often do things their creators did not intend. This is unsatisfying to many readers, but it does address the objection: programs may act in ways that are unrelated to our intelligence, and in doing so, might display unexpected intelligent behaviors. In this sense, their intelligence would not be an intentional product of human intelligence.

Turing's stronger objection comes from an anticipation that learning would eventually move to the center of AI research (keep in mind again, this is written in 1950, well before any reasonable learning algorithms had been proposed!). Turing uses the example of a robotic child in Section 7 of the paper (Learning Machines) to elaborate on his point. A child is created by its parents, but, endowed with the ability to learn, quickly begins to display behaviors its parents do not anticipate or intend. No one would suggest that a person's intelligence is "really just" the intelligence of their parents, even though their parents created them, and are partially responsible for that intelligence.

Likewise, Turing's proposed robotic child is created by a parent, but, endowed with learning, quickly begins to engage in behaviors the parent does not anticipate or intend. Therefore, machine intelligence need not be reduced to just human intelligence.

I think that if Turing were alive today, he would agree that we are now beginning to move into the era of learning machines he anticipated. Some of our programs now engage in intelligent behaviors that we do not anticipate or understand. For example, self-driving cars now kill or maim people, because they have learned behaviors their creators did not intend or anticipate, perhaps not unlike a reckless teenage driver.


Why does estimation error increase with $|H|$ and decrease with $m$ in PAC learning? Why does estimation error increase with $|H|$ and decrease with $m$ in PAC learning?

I came across this statement in the section 5.2 of the book "understanding machine learning: from theory to algorithms". You just search "increases (logarithmically)" in your browser and then you can find the sentence.

I just can't understand the statement. And there is no proof in the book either. What I would like to do is prove that estimation error $\epsilon_{est}$ increase (logarithmically) with || and decrease with . Hope you can help me out. A rigorous proof can't be better!

Definitely, you can find the proof in different resources (for example, in these notes or in the paper that originally proposed PAC learnability, A Theory of the Learnable). However, the intuition behind your question is when the size of the hypothesis increases, if you do not change anything, you can't see more part of the space. Hence, the estimation error will increase. Moreover, when you increase the number of samples, you have more chance to see more part of the hypothesis space, hence, the estimation error decrease.
Also, you can see some lemma about the relation of the PAC learnability and other similar concepts in the Wikipedia article Probably approximately correct learning:

Under some regularity conditions these three conditions are equivalent:

The concept class $C$ is PAC learnable.
The VC dimension of $C$ is finite.
$C$ is a uniform Glivenko-Cantelli class.


What are recent AI software systems and research papers close to J. Pitrat's ideas? J. Pitrat (born in 1934) was a French leading artificial intelligence scientist (the first to get a Ph.D. in France mentioning "artificial intelligence"). His blog is still online and of course refer to most of his papers (e.g. A Step toward an Artificial Artificial Intelligence Scientist, etc.) and books, notably Artificial Beings: the conscience of a conscious machine (his last book). He passed away in October 2019. I attended (and presented a talk) at a seminar in his memory.
What are recent AI systems or research papers related to the idea of symbolic AI, introspection, declarative metaknowledge, meta-learning, meta-rules, etc.?
Most of those I know are more than 20 years old (e.g. Lenat Eurisko; I am aware of OpenCyC). I am interested in papers or systems published after 2010 (perhaps AGI papers with actual complex open source software prototypes).
-see also the RefPerSys system-

Today one of the challenges is learning representations/concepts that are causally invariant. Once we have good representations then we can work on the reasoning aspect. There are 2 camps of people today. One believes that symbolic manipulation cannot be achieved properly by deep networks. Hence, they separate the task of extracting a lower-dimensional representation of objects from the visual scenes from the task of reasoning with knowledge-graphs. The other camp feels that we can do end-to-end training of a neural network and it can learn how to jointly learn a good lower-dimensional representation for each symbol along with learning how to reason with them. I am no expert at this but here are a few papers that I find are worthy for you to read -

Neuro Symbolic Concent Learner Paper, Code
Learning Reasoning Strategies in End-to-End Differentiable Proving Paper, Code
Neuro-Symbolic Visual Reasoning: Disentangling “Visual” from “Reasoning” Paper
Knowledge Infused Learning (K-IL):
Towards Deep Incorporation of Knowledge in Deep Learning Paper
Visual Concept-Metaconcept Learning Paper Project Page
CVPR 2020 workshop on Neuro-Symbolic Visual Reasoning and Program Synthesis youtube videos

If you are looking for community maintained lists, here is one  list of papers


Why does keras model get bigger after training? I notice that I create a model using tensorflow.keras.Sequential(), save it and the file size is around 5 MiB, but after I call model.fit(..), the file size is increased to 17 MiB. I copied the model to reduce the filesize and see that the accuracy is the same.

My question is, what exactly is the content of extra 12 MiB that fit() produces?
How can I access such content? If I remove those extra 12 MiB, could it affect prediction accuracy or any weird side-effect?

See my experiment code here: https://nbviewer.jupyter.org/github/off99555/TensorFlowExperiments/blob/master/test-save-keras-model.ipynb

The answer is that it's the size of the Adam optimizer state. When I change the optimizer to SGD (the vanilla optimizer), the size is not big anymore.
As far as I know, the Adam optimizer maintains gradients information of previous training iterations. And the gradient size can be as big as the model size. That's why it causes the file size to be so big.

With this in mind, when you save your model please make sure to set include_optimizer=False if you seem to use an optimizer that maintains big state similar to Adam.

Beware though, it means that you cannot load the model and continue training it again, it should only be used for inference.


In which cases is the categorical cross-entropy better than the mean squared error? In my code, I usually use the mean squared error (MSE), but the TensorFlow tutorials always use the categorical cross-entropy (CCE). Is the CCE loss function better than MSE? Or is it better only in certain cases?

As a rule of thumb, mean squared error (MSE) is more appropriate for regression problems, that is, problems where the output is a numerical value (i.e. a floating-point number or, in general, a real number).  However, in principle, you can use the MSE for classification problems too (even though that may not be a good idea). MSE can be preceded by the sigmoid function, which outputs a number $p \in [0, 1]$, which can be interpreted as the probability of the input belonging to one of the classes, so the probability of the input belonging to the other class is $1 - p$.

Similarly, cross-entropy (CE) is mainly used for classification problems, that is, problems where the output can belong to one of a discrete set of classes. The CE loss function is usually separately implemented for binary and multi-class classification problems. In the first case, it is called the binary cross-entropy (BCE), and, in the second case, it is called categorical cross-entropy (CCE). The CE requires its inputs to be distributions, so the CCE is usually preceded by a softmax function (so that the resulting vector represents a probability distribution), while the BCE is usually preceded by a sigmoid.

See also Why is mean squared error the cross-entropy between the empirical distribution and a Gaussian model? for more details about the relationship between the MSE and the cross-entropy. In case you use TensorFlow (TF) or Keras, see also How to choose cross-entropy loss in TensorFlow?, which gives you some guidelines for how to choose the appropriate TF implementation of the cross-entropy function for your (classification) problem. See also Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions? and Does the cross-entropy cost make sense in the context of regression?.


What is the exact output of the Inception ResNet V2's feature extraction layer? I am working with the Inception ResNet V2 model, pre-trained with ImageNet, for face recognition.
However, I'm so confused about what the exact output of the feature extraction layer (i.e. the layer just before the fully connected layer) of Inception ResNet V2 is. Can someone clarify exactly this?
(By the way, if you know some resource that explains Inception ResNet V2 clearly, let me know).

Due to this article: https://arxiv.org/pdf/1512.00567v3.pdf?source=post_page--------------------------- ,

I try to flatten the 3-d tensor in to 1d vector: 8*8*2048, because in the article, the pool layer of inception resnet v2 at page 6 is Pool: 8 * 8 * 2048.

But at the end, my code showed the error:
 ValueError: cannot reshape array of size 33423360 into shape (340,131072)

This is all my code:

from keras.applications.inception_resnet_v2 import InceptionResNetV2
from keras.applications.inception_resnet_v2 import preprocess_input
from keras.models import Model
from keras.preprocessing.image import load_img
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report
from imutils import paths
from keras.applications import imagenet_utils
from keras.preprocessing.image import img_to_array
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from keras.preprocessing import image
import random
import os
import numpy as np
import cv2

# Path to image
image_path = list(paths.list_images('/content/drive/My Drive/casia-299-small'))
# Random image path
random.shuffle(image_path)
# Get image name
labels = [p.split(os.path.sep)[-2] for p in image_path]

# Encode face name in to number
le = LabelEncoder()
labels = le.fit_transform(labels)

# Load model inception v2, include_top = Fale to ignore Fully Connected layer
model = InceptionResNetV2(include_top = False, weights = 'imagenet')

# Load images and resize into required input size of Inception Resnet v2 299x299
list_image = []
for (j, imagePath) in enumerate(image_path):
    image = load_img(imagePath, target_size = (299, 299, 3))
    image = img_to_array(image)

    image = np.expand_dims(image, 0)
    image = imagenet_utils.preprocess_input(image)

    list_image.append(image)

# Use pre-trained model to extract feature
list_image = np.vstack(list_image)
print("LIst image: ", list_image)
features = model.predict(list_image)
print("feature: ", features)
print("feature shape[0]: ", features.shape[0])
print("feature shape: ", features.shape)
features = features.reshape((features.shape[0], 8*8*2048))

# Split training set and test set n ratio of 80-20
x_train, x_test, y_train, y_test = train_test_split(features, labels, test_size = 0.2, random_state =42)

params = {'C': [0.1, 1.0, 10.0, 100.0]}
model = GridSearchCV(LogisticRegression(), params)
model.fit(x_train,y_train)
model.save('/content/drive/My Drive/casia-299-small/myweight1.h5')
print('Best parameter for the model {}'.format(model.best_params_))

preds = model.predict(x_test)
print(classification_report(y_test, preds))
```


Precise localization and characterization of rudimentary shapes with neural networks I understand that there are flavors of (convolutional) neural networks that are useful for object localization and detection tasks of reasonable difficulty. In all of the examples I have seen so far, localization is formulated as finding the corners of a bounding box. Often, the fit is not expected to be very precise:

Conversely, I am interested in a task I want to achieve very precise localization and characterization of some simple shapes or objects. As an example of one of the simplest cases I can think of, my inputs will be images like the following:

Given this 60x60 image, I want my neural net(s), via regression, to tell me that the circle's diameter is 18px and its centre is located at (28, 21) from top left. (I will train it using similar 60x60 images with white circles of various sizes on black backgrounds.)

Later I am interested in dealing with similar tasks in the real world, e.g. spheres/cubes/cylinders with different viewing angles, light conditions, occlusions, etc. However, I am interested in solving this simplest case first. (One reason is that I can generate this data very easily.)

I have the following specific questions:

Has anyone used neural nets for this sort of tasks before? (e.g. precisely determining sizes and centroids of objects)
My understanding is that these things are at least theoretically possible using convolutional nets, or even sufficiently complicated vanilla fully connected nets. Is this correct?
What architecture(s) would be appropriate for these tasks?

Note: I am aware that fitting a bounding box to the circles and calculating its centre and size will solve this particular case, but it will not generalize to handle occlusions, changing lights, etc. I would like to move towards a method which can, for example, calculate the centroids and diameters of spheres in real-world B&W photos.

I think you're describing "object localization and detection" which combines object identification with discovery of its spatial placement in the field of view.  There's been a lot of work on this in the past 5 years using CNNs and variants.  It's very much in demand for pedestrian/obstruction detection, identification, and avoidance in autonomous car systems.

Here's a place to start:  https://github.com/kjw0612/awesome-deep-vision

Especially, take a look at "Faster R-CNN" by Ren et al, 2016.


Many of the best probabilistic models represent probability distributions only implicitly I am currently studying Deep Learning by Goodfellow, Bengio, and Courville. In chapter 5.1.2 The Performance Measure, P, the authors say the following:

The choice of performance measure may seem straightforward and objective, but it is often difficult to choose a performance measure that corresponds well to the desired behavior of the system.
In some cases, this is because it is difficult to decide what should be measured. For example, when performing a transcription task, should we measure the accuracy  of the system at transcribing entire sequences, or should we use a more fine-grained performance measure that gives partial credit for getting some elements of the sequence correct? When performing a regression task, should we penalize the  system more if it frequently makes medium-sized mistakes or if it rarely makes very large mistakes? These kinds of design choices depend on the application.
In other cases, we know what quantity we would ideally like to measure, but measuring it is impractical. For example, this arises frequently in the context of density estimation. Many of the best probabilistic models represent probability distributions only implicitly. Computing the actual probability value assigned to a  specific point in space in many such models is intractable. In these cases, one must design an alternative criterion that still corresponds to the design objectives, or design a good approximation to the desired criterion.

It is this part that interests me:

Many of the best probabilistic models represent probability distributions only implicitly.

I don't have the experience to understand what this means (what does it mean to represent distributions "implicitly"?). I would greatly appreciate it if people would please take the time to elaborate upon this.

The probabilistic models that represent distributions implicitly are, for example, the GANs. (Goodfellow is one of the authors of the original GAN model).
In the paper Variational Inference using Implicit Distributions (2017), the authors write

Implicit distributions are probability models whose probability density function may be intractable, but there is a way to

sample from them exactly and/or calculate and approximate expectations under them, and

calculate or estimate gradients of such expectations with respect to model parameters.

Implicit models have been successfully applied to generative modelling in generative adversarial networks (GANs)
A popular example of implicit models are stochastic generative networks: samples from a simple distribution - such as uniform or Gaussian - are transformed nonlinearly and non-invertably by a deep neural network

They also provide a table (table 1) that shows some probabilistic models that use an implicit distribution. Here $I$ denotes inference only. VI stands for variational inference.

See this blog post Variational Inference using Implicit Models, Part I: Bayesian Logistic Regression. See also the paper Learning in Implicit Generative Models (2017).


Linear regression: why is distance *squared* used as an error metric? Usually when performing linear regression predictions and gradient descent, the measure of the level of error for a particular line will be measured by the sum of the squared-distance values.

Why distance squared?

In most of the explanations I heard, they claim that:

the function itself does not matter
the result should be positive so positive and negative deviations are still counted

However, an abs() approach would still work. And isn't it inconvenient that distance squared minimizes the distance result for distances lower than 1?

I'm pretty sure someone must have considered this already -- so why is distance squared the most used approach to linear regression?

The squared form is sometimes called the Euclidean norm or L2 norm. One of its very helpful properties is that it has an easily defined derivative, which can be used in mathematical analysis and translated fairly easily into code.

Intuitively it is thought that it is advantageous to exaggerate the differences according to the value of the error, which squaring does. You might also use the powers 3 or 4, but the derivative is more complex.

A number of different norms may be used, according to the particular circumstances of the problem at hand.


What are the differences between an agent that thinks rationally and an agent that acts rationally? Stuart Russell and Peter Norvig pointed out 4 four possible goals to pursue in artificial intelligence: systems that think/act humanly/rationally.

What are the differences between an agent that thinks rationally and an agent that acts rationally?

Maybe a good example to think about would be something like the Sphyx story. The wasp in the story appears to behave like a rational being: it seems to have a plan of action, it seems to be able to do advanced operations like counting, and it seems to execute the plan well. However, if you disrupt the wasp's plan, it becomes apparent that it is not thinking rationally. Instead, it has evolved a very complex behaviour that appears to include rational components, but is still just an instinct.
In the context of AI, consider a GOFAI system for planning like GraphPlan as opposed to a machine learning system for generating plans. While the former is a general-purpose algorithm for reasoning about planning problems, the latter is an input/output mapping that may mimic reasoning, or may be more "instinctual". Some AI'ers would say that the latter system is not really engaged in rational thought, while the former is. Both systems exhibit rational action, however.
Instinctive systems often work very well, and I'd say that the increasing effectiveness of machine learning approaches since that edition of R&N was published (almost 10 years ago now?) makes this a fuzzier distinction in practice than the book might suggest.


Are Genetic Algorithms suitable for problems like the Knuth problem? We all know that Genetic Algorithms can give an optimal or near-optimal solution. So, in some problems like NP-hard ones, with a trade-off between time and optimal solution the near-optimal solution is good enough.
Since there is no guarantee to find the optimal solution, is GA considered to be a good choice for solving the Knuth problem?
According to Artificial intelligence: A modern approach (third edition), section 3.2 (p. 73):

Knuth conjectured that, starting with the number 4, a sequence of
factorial, square root, and floor operations will reach any desired
positive integer.

For example, 5 can be reached from 4:

floor(sqrt(sqrt(sqrt(sqrt(sqrt((4!)!))))))

So, if we have a number (5) and we want to know the sequence of the operations of the 3 mentioned ones to reach the given number, each gene of the chromosome will be a number that represents a certain operation with an additional number for (no operation) and the fitness function will be the absolute difference between the given number and the number we get from applying the operations in a certain order for each the chromosome (to min). Let's consider that the number of the iterations (generations) is done with no optimal solution and the nearest number we have is 4 ( with fitness 1), the problem is that we can get 4 from applying no operation on 4 while for 5 we need many operations, so the near-optimal solution is not even near to the solution.
So, is GA is not suitable for this kind of problems? Or the suggested chromosome representation and fitness function are not good enough?

Before trying to answer your question more directly, let me clarify something.
People often use the term genetic algorithms (GAs), but, in many cases, what they really mean is evolutionary algorithms (EAs), which is a collection of population-based (i.e. multiple solutions are maintained at the same time) optimization algorithms and approaches that are inspired by Darwinism and survival of the fittest. GAs is one of these approaches, where the chromosomes are binary and you have both the mutation and cross-over operation. There are other approaches, such as evolution strategies or genetic programming.
As you also noticed, EAs are meta-heuristics, and, although there is some research on their convergence properties [1], in practice, they may not converge. However, when any other potential approach has failed, EAs can be definitely useful.
In your case, the problem is really to find a closed-form (or analytical) expression of a function, which is composed of other smaller functions. This really is what genetic programming (in particular, tree-based GP) was created for. In fact, the Knuth problem is a particular instance of the symbolic regression problem, which is a typical problem that GP is applied to. So, GP is probably the first approach you should try.
Meanwhile, I have implemented a simple program in DEAP that tries to solve the Knuth problem. Check it here. The fitness of the best solution that it has found so far (with some seed) is 4 and the solution is floor(sqrt(float(sqrt(4)))) (here float just converts the input to a floating-point number, to ensure type safety).  I used the difference as the fitness function and ran the GP algorithm for 100 generations with 100 individuals for each generation (which is not a lot!). I didn't tweak much the hyper-parameters, so, maybe, with the right seed and hyper-parameters, you can find the right solution.
To address your concerns, in principle, you could use that encoding, but, as you note, the GA could indeed return $4$ as the best solution (which isn't actually that far away from $5$), which you could avoid my killing, at every generation, any individuals that have just that value.
I didn't spend too much time on my implementation and thinking about this problem, but, as I said above, even with genetic programming and using only Knuth's operations, it could get stuck in local optima. You could try to augment my (or your) implementation with other operations, such as the multiplication and addition, and see if something improves.


How to train a neural network with a data set that in which the target is a mix of 0-1 label and numeric real value label? I am running into an issue in which the the target (label collums) of my dataset contain a mixture of binary label (yes/no) and some numeric value label.

The value of these numeric value (resource 1 and resource 2 collumns) experience a large variation margin. Sometime these numeric value can be like 0.389 but sometimes they can be 0.389 x 10^-4 or something.

My goal is to predict the binary decision and the amount of resource allocated to a new user who have input feature 1 (numeric) and input feature 2 (numeric).

My initial though would be that the output neuron corresponding to the 0-1 decision would use logistic regression activation function. But for the neuron that corresponding to the resource I am not quite sure.

What would be the appropriate way to tackle such situation in term of network structure or data pre-processing  strategy ?

Thank you for your enthusiasm !

Your question are missing some details and i will assume some scenarios.

If you have a classification problem: you can try group the values in intervals that make sense (you should analyze and decide for this setup), if its possible. For example: 0.000-0.250 (0), 0.251-0.500 (1), 0.501-0.750 (2) and so on. Note that neural networks are sensible for distance between values (1 is closer to 0 than 2, so 1 is more similar to 0 than 2 and so on). If that is not your case, you should binarize the values in One Hot Encode manner.
If you have a regression problem, you should be ok without anything else. You can try normalize your outputs and observe the results, but generally it's not necessary for regression problems.
Be sure if your dataset are free of outliers and noisy data as much as possible.
It's important choose activations functions that are adequate for the range of values in your attributes and output. This can depend on how do you treat and setup your dataset, the range of values, normalization etc.

Update after more details in question

Your neural network should have 3 neurons in the output layer, with linear activation. As said before, normalization usually is not necessary in regression problems, but if your values are too diferent (like the range in resource 1 and resource 2) maybe some kind of adjustment (normalization, standardization etc) can be helpful. But you need try and see the results.


Understanding the configuration of replay memory and epsilon in deep reinforcement learning I am tentatively reusing a codebase of pacman to train my own deep reinforcement learning model. While most of the components seems reasonable and understandable to me, there are two things that seem obscure to me:

How to decide the size of the replay memory? Currently, since I set the total step of learning as 4000 (note that in the referred codebase this value is set as 4000000), so I just proportionally decrease the replay_memory_size as 400. Would that make sense?
What is the return value epsilon when calling function PiecewiseSchedule? I also proportionally decrease its parameters as follows:

        epsilon = PiecewiseSchedule([(0, 1.0),
                                     (40, 1.0), # since we start training at 10000 steps
                                     (80, 0.4),
                                     (200, 0.2),
                                     (400, 0.1),
                                     (2000, 0.05)], outside_value=0.01)
        replay_memory = PrioritizedReplayBuffer(replay_memory_size, replay_alpha)

where the original function call is like this:

        epsilon = PiecewiseSchedule([(0, 1.0),
                                     (10000, 1.0), # since we start training at 10000 steps
                                     (20000, 0.4),
                                     (50000, 0.2),
                                     (100000, 0.1),
                                     (500000, 0.05)], outside_value=0.01)
        replay_memory = PrioritizedReplayBuffer(replay_memory_size, replay_alpha)

And in general, what is the principle (guideline) behind setting a good size of "replay memory" and calling function PiecewiseSchedule? Thank you!

Replay memory is a recording of the games that the agents has played. It is this data that is used to train the neural network (or whatever machine learning method you are using).
The training is performed by looking at the games played and the resulting reward signal (in Pacman, this might be your score), and then learning which game strategies worked well and which did not.
It is difficult to say exactly how big the replay memory should be, this is something that you will need to experiment with. The larger it is, then the more examples will be available to be learned from during a particular training period, but of course it takes longer to acquire them.

Epsilon is a parameter that governs the trade-off between exploration and exploitation. By this, I mean that for each action that your agent takes, it needs to decide whether to do the best thing that it has already discovered to do (exploitation), or try something new (exploration). Usually, a value of epsilon near to 1 means that the agent will choose to explore more, and a value closer to 0 will mean that the agent will exploit more.

You can imagine therefore that at the beginning of an agent's training you might want epsilon ~1, because it has lots of new strategies to try to discover, but as it gets better you would reduce it to ~0 so that it can explore strategies that are only reachable after a long sequence of game steps.
Naturally, when the agent is fully trained and ready to be deployed then epsilon would = 0.
In the example that you posted, you can see that Pacman starts off doing nothing but exploring, but after 500000 games it exploits 95% of the time.


Can an AI model be based on the principles of rationalism? Are current AI models entirely empiricist?
Can they be rationalist?If so How?

It depends on how you define rationality.  In the field of AI, the game theoretic conception is utilized, in sense of a rational agent.

It's important to note that a decision making agent can be perfectly rational and still be "wrong" (i.e. make sub-optimal decisions.)

This is primarily because rationality is bounded for all non-trivial problems.  The quality of decisions depends on quality of information, time and space available to make the decision in relation to the problem size, and the quality of the analytic processes.

Logic itself has pitfalls, in that any formal system can either be complete or consistent, but never both.

I would say that the current, strong AIs are empiricists, in that the methods are statistical, and based on testing and evaluation of results.  For instance, a Monte Carlo Tree Search (MCTS) game algorithm will play billions of games against itself and analyze the results to make the strongest choice for any given state.

But, if one considers any decision-making agent, no matter how primitive or weak, to be a form of AI, then there can be non-empirical AI in the sense that the processes heuristic as opposed to statistical.


Updating action-value functions in Semi-Markov Decision Process and Reinforcement Learning Suppose that the transition time between two states is a random variable (for example, unknown exponential distribution); and between two arrivals, there is no reward. If $\tau$ (real number not an integer number) shows the time between two arrivals, should I update Q-functions as follows:
$Q(s,a) = Q(s,a)+\alpha.(R+\gamma^{\tau} \max_{b \in A}Q(s^{\prime},b)-Q(s,a))$
And, to compare different algorithms, total rewards ($TR=R_{1}+ R_2+R_{3}+...+R_{T}$) is used.
What measure should be used in the SMDP setting? I would be thankful if someone can explain the Q-Learning algorithm for the SMDP problem with this setting.
Moreover, I am wondering when Q-functions are updated. For example, if a customer enters our website and purchases a product, we want to update the Q-functions. Suppose that the planning horizon (state $S_{0}$) starts at 10:00 am, and the first customer enters at 10:02 am, and we sell a product and gain $R_1$ and the state will be $S_1$. The next customer enters at 10:04 am, and buy a product, and gain reward $R_2$ (state $S_{2}$). In this situation, should we wait until 10:02 to update the Q-function for state $S_0$?
Is the following formula correct?
$$V(S_0)= R_1 \gamma^2+  \gamma^2V(S_1)$$
In this case, if I discretize the time horizon to 1-minute intervals, the problem will be a regular MDP problem. Should I update Q-functions when no customer enters in a time interval (reward =0)?

Personally, I find the best way to think of SMDPs intuitively by just imagining that you just discretise the time into such small steps (infinitesimally small steps if necessary) that you can treat it as a normal MDP again, but with some extra domain knowledge that you can exploit primarily for computational efficiency:

Only at time steps that really correspond to "events" in your SMDP can you observe non-zero rewards; at all other time steps you just get rewards equal to $0$.
Only at time steps that really correspond to "events" in your SMDP do you have an action space greater than $1$; in all the "fake" time steps, you have no agency, you just have a single action available (say, a "dummy" or "null" action). So all of these "fake" time steps do not contribute in any way to the "credit assignment" problem in RL, and you can kind of ignore them in your learning steps; only the time spent in them can still be important for discount factors $\gamma < 1$.

If $\tau$ (real number not an integer number) shows the time between two arrivals, should I update Q-functions as follows:

Yes, an update rule like that looks correct to me. Let's take an example situation, where $\tau = 2.0$, and instead of using the update rule you suggest, we take the "proper" approach of discretising into smaller time steps, and treating it as a regular MDP. In this simple example case, it is sufficient to discretise by taking time steps that correspond to durations of $1.0$.
In the SMDP, we'll have only a single transition $s_0 \rightarrow s_2$ (it will become clear why I use slightly strange time-indexing here soon), after which we observe a reward, and this transition takes time $\tau = 2.0$. In the corresponding MDP, we'll have two state transitions; $s_0 \rightarrow s_1$, and $s_1 \rightarrow s_2$, with two reward observations $R_1$ and $R_2$, where we know for sure that:

$R_2 = 0$ (because it does not actually correspond to any event in the SMDP)
We have a meaningful choice between multiple actions at $s_0$, each of which can have different transition probabilities for taking us into different "dummy" states $s_1$, and yield possibly-non-zero rewards $R_1$. In the dummy state $s_1$, we'll always only have the choice for a single dummy/null action (because this state does not correspond to any event in the SMDP), which always yields $R_2 = 0$ as mentioned above.

So, the correct update rule for $s_1$ where we picked a forced dummy action $\varnothing$ and are doomed to receive a reward $R_2 = 0$, would be:
$$Q(s_1, \varnothing) \gets Q(s_1, \varnothing) + \alpha \left( 0 + \gamma \max_{a'} Q(s_2, a') - Q(s_1, \varnothing) \right)$$
and the correct update rule for $s_0$, where we picked a meaningful action $a_0$ and may get a non-zero reward $R_1$, would be:
$$Q(s_0, a_0) \gets Q(s_0, a_0) + \alpha \left( R_1 + \gamma \max_{a'} Q(s_1, a') - Q(s_0, a_0) \right)$$
In this last update rule, we know that $s_1$ is a dummy state in which the dummy action $\varnothing$ is the only legal action. So, we can get rid of the $\max$ operator there and simplify it to:
$$Q(s_0, a_0) \gets Q(s_0, a_0) + \alpha \left( R_1 + \gamma Q(s_1, \varnothing) - Q(s_0, a_0) \right)$$
Since we know that $s_1$ is a dummy state in which we are never able to make meaningful choices anyway, it seems a bit wasteful to actually keep track of $Q(s_1, \varnothing)$ values for it. Luckily, we can easily express $Q(s_1, \varnothing)$ directly in terms of $Q(s2, \cdot)$ -- which is exactly the next set of $Q$-values that we would be interested in keeping track of again:
$$Q(s_1, \varnothing) = \mathbb{E} \left[ 0 + \gamma \max_{a'} Q(s_2, a') \right]$$
So if we want to skip learning $Q$-values for $s_1$ (because it's kind of a waste of effort), we can just use this definition and plug it straight into the update rule for $Q(s_0, a_0)$. $Q$-learning is inherently an algorithm that just concrete samples of experience to estimate expectations (and this is a major reason why it typically uses learning rates $\alpha < 1.0$, so we can simply get rid of the expectation operator when doing this:
$$Q(s_0, a_0) \gets Q(s_0, a_0) + \alpha \left( R_1 + \gamma \left[ \gamma \max_{a'} Q(s_2, a') \right] - Q(s_0, a_0) \right)$$
and this is basically the update rule that you suggested. Note; here I assumed that you receive your rewards directly when you do take actions in your SMDP, which is why I had $R_1$ as a possibly-non-zero reward, and always $R_2 = 0$. I suppose you could also in some cases envision SMDPs where the reward only arrives on the next SMDP-time-step, and that the amount of time that ends up being elpased in between the two events is important to take into account via the discount factor $\gamma$. So you could also choose to model a problem where $R_1 = 0$ and $R_2$ may be non-zero, and this would yield a different update rule (I think one where the reward gets multiplied by $\gamma^{\tau - 1}$? not sure, would have to go through the steps again).

What measure should be used in the SMDP setting? I would be thankful if someone can explain the Q-Learning algorithm for the SMDP problem with this setting.

I think it would be important to involve the amount of time that you take somehow in your evaluation criterion. You could run episodes for a fixed amount of time, and then just evaluate agents based on the sum of rewards. If you don't run for a fixed amount of time (but instead for a fixed number of steps, each of which may take variable amounts of time, for example), you would probably instead want to evaluate agents based on the average rewards per unit of time. You could also include discount factors in your evaluation if you want, but probably don't have to.

Moreover, I am wondering when Q-functions are updated. For example, if a customer enters our website and purchases a product, we want to update the Q-functions. Suppose that the planning horizon (state $S_0$) starts at 10:00 am, and the first customer enters at 10:02 am, and we sell a product and gain $R_1$ and the state will be $S_1$. The next customer enters at 10:04 am, and buy a product, and gain reward $R_2$ (state $S_2$). In this situation, should we wait until 10:02 to update the Q-function for state $S_0$?

This depends on you state representation, and how you model a "state", and to what extent previous actions have influence the state you end up in. Keep in mind that the update rule for $Q(S_0)$ also requires for $S_1$ (or even $S_2$ if $S_1$ is a "dummy state" that you skip) to have been observed. So, if your state representation includes some features describing the "current customer" for which you want to pick an action (do you offer them a discount or not, for example?), then you can only update the $Q$-value for the previous customer when the next customer has arrived. This model does assume that your previous actions have some level of influence over the future states that you may end up in though. For example, you might assume that if your actions make the first customer very happy, you get a better reputation and are therefore more likely to end up in future states where other customers visit more frequently.