How UCT in MCTS selection phase avoids starvation? The first step of MCTS is to keep choosing nodes based on Upper Confidence Bound applied to trees (UCT) until it reaches a leaf node where UCT is defined as
$$\frac{w_i}{n_i}+c\sqrt{\frac{ln(t)}{n_i}},$$
where

$w_i$= number of wins after i-th move
$n_i$ = number of simulations after the i-th move
$c$ = exploration parameter (theoretically equal to $\sqrt{2}$)
$t$ = total number of simulations for the parent node

I don't really understand how this equation avoids sibling nodes being starved, aka not explored. Because, let's say you have 3 nodes, and 1 we'll call it node A is chosen randomly to be explored, and just so happens to simulate a win. So, node A's UCT$=1+\sqrt(2)\sqrt{\frac{ln(1)}{1}}$, while the other 2 nodes UCT = 0, because they are unexplored and the game just started, so by UCT the other 2 nodes will never be explored no? Because after this it'll go into the expansion phase and expansion only happens it reaches a leaf node in the graph. So because node A is the only one with a UCT $> 0$ it'll choose a child of node A and it will keep going down that node cause all the siblings of node A have a UCT of 0 so they never get explored.

First explore the nodes A,B,C once.
For reference see this paper by David Silver and Sylvain Gelly, Combining Online and Offline Knowledge in UCT

If any action from the current state $s$ is not represented in the tree, $\exists a \in \mathcal{A}(s),(s, a) \notin \mathcal{T},$ then the uniform random policy $\pi_{\text {random }}$ is used to select an action from all unrepresented actions, $\tilde{\mathcal{A}}(s)=\{a \mid(s, a) \notin \mathcal{T}\}$.


How to implement an "unknown" class in multi-class classification with neural networks? For example, I need to detect classes for MNIST data. But I want to have not 10 classes for digits, but also I want to have 11th class "not a digit", so that any letter, any other type of image, or random noise would be classified as "not a digit". Similarly, with CIFAR-10, I want to have the 11th "unknown" class to classify any image that contains something out of the available 10 classes of CIFAR-10.
So, how to implement such a feature? Maybe there are some examples somewhere, preferable with Keras.

The usual way to implement this would be to add the new class with data examples.

Some things you need to address:

Sourcing new data for your "other" class.
Ensuring the amount and variation of data in "other" class examples matches how the predictor will be used.

Code examples for this are not necessary, as you would just use the same network design as you already have and just add another output. This is a data and model definition problem.

Logically you have another option: As well as outputting the predicted class, you predict separately whether there is any detectable object at all as a true/false value. This still requires the additional data, but is for example how the YOLO algorithm works for object detection. Object detection has a specific meaning - it involves finding the co-ordinates and class of possibly multiple objects in an image. This goes beyond the wording of your question, but is a typical end goal if you are asking this kind of question.

YOLO predicts the presence of an object separately from the class of object. The additional data for YOLO training comes from segmenting the source images, so many parts of the target image are background with no objects. In that case the additional data you require is due to more detailed labelling within each image example.

YOLO is quite complicated architecture, so you might want to look at this example using Keras on a Github project for more details, if object detection is your goal.


Why should each filter have different weights for each input channel? From the answers to this question In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?, I got the fact that each filter has different weights for each input channel. But why should that be the case? What if we apply the same weights to each input channel? Does it work or not?

For simplicitly, let's consider only the first convolutional layer, that is, the one applied to the image. If you consider an RGB image, then there are $3$ channels: the red channel, the green channel and the blue channel. Thus, a kernel that is applied to this image will also have $3$ channels: the red channel, the green channel and the blue channel. In general, the distributions of the intensity of the red, green and blue colors in the image are different, so, in general, the red, green and blue channels of the kernel will also be different because they need to keep track of different information.


How is the number of parameters reduced in the group convolution? I think I don't understand group convolutions well.
Say you have 2 groups. This means that the number of parameters would be reduced in half. So, assuming you have an image and 100 channels, with a filter size of $3 \times 3$, you would have 900 parameters (ignore the bias for this example). If you separate this into 2 groups, if I understand it well, you would have 2 groups of 50 channels.
This can be made faster, by running the 2 groups in parallel, but how does the number of parameters get halved? Isn't each group having $50*9=450$ parameters, so, in total, you still have 900 parameters? Do they mean that the number of parameters that the backpropagation goes over (in each branch) gets halved?
Because overall, I don't see how it can get reduced. Also, is there a downside in using more groups (even going to 100 groups of 1 channel each)?

The number of parameters is filters*input_channels*output_channels

Groups are formed among input and output channels.

So instead of input_channels*output_channels with two groups you get (input_channels/2)*(output_channels/2) + (input_channels/2)*(output_channels/2)


If AI can perceive, can it be sentient? I think that AI perceive the world. Can an AI be sentient?

I'd call any system that receives input to be "perceiving", and that can constitute a basic definition of "awareness" or even "consciousness", distinct from "self-awareness" or consciousness in the human sense, which we tend to think of in terms of sentience.

What the Chinese Room seems to argue is that sentience is based on qualia, which may be difficult or impossible to validate in another entity.  (With humans, we can assume another individual is sentient because we perceive ourselves to be sentient, and share the same medium of human brains and bodies.)

It seems increasingly likely that "algorithms sufficiently advanced" will one day be able to model human cognition, but the real question will be is it actual cognition in the sense of understanding, or mere imitation? ("Semantics" vs. "Syntactics")

Cognition is derived from the Latin cognosco, itself related to the Greek words such as γνωστός (a root of "gnostic")

The question and definition of sentience is less clear cut, but the argument re: algorithms relates to cognition thus:

"The lights may be on, but is anybody home?"


How much of Deep Mind's work is actually reproducible? Deep Mind has published a lot of works on deep learning in the last years, most of them state-of-the-art on their respective tasks. But how much of this work has actually been reproduced by the AI community? For instance, the Neural Turing Machine paper seems to be very hard to reproduce, according to other researchers.

On the suggestion of the O.P. rcpinto I converted a comment about seeing "around a half-dozen papers that follow up on Graves et al.'s work which have produced results of the  caliber" and will provide a few links. Keep in mind that this only answers the part of the question pertaining to NTMs, not Google DeepMind itself, plus I'm still learning the ropes in machine learning, so some of the material in these papers is over my head; I did manage to grasp much of the material in Graves et al.'s original paper{1] though and am close to having homegrown NTM code to test. I also at least skimmed the following papers over the last few months; they do not replicate the NTM study in a strict scientific manner, but many of their experimental results do tend to support the original at least tangentially:

• In this paper on a variant version of NTM addressing, Gulcehere, et al. do not try to precisely replicate Graves et al.'s tests, but like the DeepMind team, it does demonstrate markedly better results for the original NTM and several variants over an ordinary recurrent LSTM. They use 10,000 training samples of a Facebook Q&A dataset, rather than the N-grams Graves et al. operated on in their paper, so it's not replication in the strictest sense. They did however manage to get a version of the original NTM and several variants up and running, plus recorded the same magnitude of performance improvement.2

• Unlike the original NTM, this study tested a version of reinforcement learning which was not differentiable; that may be why they were unable to solve several of the programming-like tasts, like Repeat-Copy, unless the controller wasn't confined to moving forwards. Their results were nevertheless good enough to lend support to the idea of NTMs. A more recent revision of their paper is apparently available, which I have yet to read, so perhaps some of their variant's problems have been solved.3

• Instead of testing the original flavor of NTM against ordinary neural nets like LSTMs, this paper pitted it against several more advanced NTM memory structures. They got good results on the same type of programming-like tasks that Graves et al. tested, but I don't think they were using the same dataset (it's hard to tell from the way their study is written just what datasets they were operating on).4

• On p. 8 of this study, an NTM clearly outperforms several LSTM, feed-forward and nearest-neighbor based schemes on an Omniglot character recognition dataset. An alternative approach to external memory cooked up by the authors clearly beats it, but it still obviously performs well. The authors seem to belong to a rival team at Google, so that might be an issue when assessing replicability.5

• On p. 2 these authors reported getting better generalization on "very large sequences" in a test of copy tasks, using a much smaller NTM network they evolved with the genetic NEAT algorithm, which dynamically grows topologies.6

NTMs are fairly new so there hasn't been much time to stringently replicate the original research yet, I suppose. The handful of papers I skimmed over the summer, however, seem to lend support to their experimental results; I have yet to see any that report anything but excellent performance. Of course I have an availability bias, since I only read the pdfs I could easily find in a careless Internet search. From that small sample it seems that most of the follow-up research has been focused on extending the concept, not replication, which would explain the lack of replicability data. I hope that helps.

1 Graves, Alex; Wayne, Greg and Danihelka, Ivo, 2014, "Neural Turing Machines," published Dec. 10, 2014.

2 Gulcehre, Caglar; Chandar, Sarath; Choy, Kyunghyun and Bengio, Yoshua, 2016, "Dynamic Neural Turing machine with Soft and Hard Addressing Schemes," published June 30, 2016.

3 Zaremba, Wojciech and Sutskever, Ilya, 2015, "Reinforcement Learning Neural Turing Machines," published May 4, 2015.

4 Zhang; Wei; Yu, Yang and Zhou, Bowen, 2015, "Structured Memory for Neural Turing Machines," published Oct. 25, 2015.

5 Santoro, Adam; Bartunov, Sergey; Botvinick, Matthew; Wierstra, Daan and Lillicrap, Timothy, 2016, "One-Shot Learning with Memory-Augmented Neural Networks," published May 19, 2016.

6 Boll Greve, Rasmus; Jacobsen, Emil Juul and Sebastian Risi, date unknown, "Evolving Neural Turing Machines." No publisher listed

All except (perhaps) Boll Greve et al. were published at the Cornell Univeristy Library arXiv.org Repository: Ithaca, New York.


Why does PyTorch use a different formula for the cross-entropy? In my understanding, the formula to calculate the cross-entropy is

$$
H(p,q) = - \sum p_i \log(q_i)
$$

But in PyTorch nn.CrossEntropyLoss is calculated using this formula:

$$
loss = -\log\left( \frac{\exp(x[class])}{\sum_j \exp(x_j)} \right)
$$

that I think it only addresses the $\log(q_i)$ part in the first formula.

Why does PyTorch use a different formula for the cross-entropy?

When you one-hot-encode your labels with $p_i \in \{0,1\}$ you get $p_i = 0$ iff $i$ is not correct and, equivalently, $p_i =1$ iff $i$ is correct.

Hence, $p_i \log(q_i) = 0 \log(q_i) = 0 $ for all classes except the "truth" and $p_i \log(q_i) = 1 \log(q_i) = \log(q_i) $ for the correct prediction.

Therefore, your loss reduces to:
$$
H(p,q) = - \sum p_i \log(q_i) = - \log(q_{truth})
$$


What's the optimal policy in the rock-paper-scissors game? A deterministic policy in the rock-paper-scissors game can be easily exploited by the opponent - by doing just the right sequence of moves to defeat the agent. More often than not, I've heard that a random policy is the optimal policy in this case - but the argument seems a little informal.
Could someone please expound on this, possibly adding more mathematical details and intuition? I guess the case I'm referring to is that of a game between two RL agents, but I'd be happy to learn about other cases too. Thanks!
EDIT: When would a random policy be optimal in this case?

For this, we will need game theory.
In game theory, an optimal strategy is one that cannot be exploited by the opponent even if they know your strategy.
Let's say you want a strategy where your move selection is not based on what happened before (so you are not trying to model your opponent, or trick them into believing you will always play scissors and then throw them off, anything like that).
A strategy will look like $(P, S, R)$, where $P, S, R \in [0, 1], P+S+R = 1$. You select paper with probability $P$, scissors with probability $S$, rock with probability $R$.
Now, if your probabilities are a bit uneven (for example $(0.5, 0.2, 0.3)$) an opponent can abuse that strategy. If your opponent plays with probabilities $(p, s, r)$, their expected reward (counting +1 for win, -1 for loss, 0 for draw) would be $0.5(s - r) + 0.2(r - p) + 0.3(p - s) = 0.1p + 0.2s - 0.3r$. If they wish to maximize their wins, they would play scissors all the time against you, and expect to have a distinct advantage over you.
In general, for a strategy $(P, S, R)$ for you and $(p, s, r)$ for your opponent, your opponent's winnings would be $P(s - r) + S(r - p) + R(p - s) =  p(R-S) + s(P-R) + r(S - P)$. If all the partial derivatives of this, with respect to $p$, $s$ and $r$ are 0, the opponent has no way to maximize his winnings; they would have no incentive to play a particular move over any other move.
This occurs when $P = S = R = \frac13$.
That's basically how to approach game theory: find a strategy so your opponent has no incentive to choose one action over another. The approach seems a bit counter-intuitive at first (you're trying to find the optimal strategy for your opponent instead of for yourself) but it works for many similar problems.


Optimizing Max Pooling Algorithm The below code is a max pooling algorithm being used in a CNN. The issue I've been facing is that it is offaly slow given a high number of feature maps. The reason for its slowness is quite obvious-- the computer must perform tens of thousands of iterations on each feature map. So, how do we decrease the computational complexity of the algorithm?

('inputs' is a numpy array which holds all the feature maps and 'pool_size' is a tuple with the dimensions of the pool.)

def max_pooling(inputs, pool_size):

    feature_maps = []
    for feature_map in range (len(inputs)):
        feature_maps.append([])
        for i in range (0, len(inputs[feature_map]) - pool_size[0], pool_size[0]):
            for j in range (0, len(inputs[feature_map]) - pool_size[0], pool_size[0]):
                feature_maps[-1].append(np.array(max((inputs[feature_map][j:j+pool_size[0], i:i+pool_size[0]]).flatten())))

    return feature_maps

The reason for its slowness is quite obvious-- the computer must perform tens of thousands of iterations on each feature map. So, how do we decrease the computational complexity of the algorithm?

In terms of computational complexity / algorithm, there is not a lot to gain; max pooling simply has to go through all the feature maps to find the maximum numbers in each of the sections to be "merged/pooled" by taking the max.

There likely is a lot to gain in terms of implementation though. The current implementation is entirely in pure Python, and pure Python is notoriously slow. Those kinds of loops can be run significantly faster using numpy operations rather than manual Python loops. Such operations tend to be much faster due to:

running optimized C code rather than Python code, and
in some cases, using vectorized operations to perform multiple similar computations at different indices simultaneously, rather than doing them one-by-one

I did not yet try to "translate" your pure python code into python code using numpy. However, some examples of numpy-based implementations can be found in various answers to this question on StackOverflow.

I assume that your choice to manually implement things like max pooling is because you want to learn about implementing it / understand it better. If, instead, your goal is simply to get something running as quickly as possible, it may be a good idea to look into using a framework such as Tensorflow or PyTorch. These come with efficient implementations of many things you'll want for Neural Networks, including Max Pooling.


Fundamentally, what is a perfect language model? Suppose that we want to generate a sentence made of words according to language $L$:
$$
W_1 W_2 \ldots W_n
$$
Question: What is the perfect language model?
I ask about perfect because I want to know the concept fundamentally at its fullest extent. I am not interested in knowing heuristics or shortcuts that reduce the complexity of its implementation.

1. My thoughts so far
1.1. Sequential
One possible way to think about it is moving from left to right.  So, 1st, we try to find out value of $W_1$. To do so, we choose the specific word $w$ from the space of words $\mathcal{W}$ that's used by the language $L$.  Basically:
$$
w_1 = \underset{w \in \mathcal{W}}{\text{arg max }} \Pr(W_1 = w)
$$
Then, we move forward to find the value of the next word $W_2$ as follows
$$
w_2 = \underset{w \in \mathcal{W}}{\text{arg max }} \Pr(W_2 = w | W_1 = w_1)
$$
Likewise for $W_3, \ldots, W_n$:
$$
w_3 = \underset{w \in \mathcal{W}}{\text{arg max }} \Pr(W_3 = w | W_1 = w_1, W_2=w_2)
$$
$$
\vdots
$$
$$
w_n = \underset{w \in \mathcal{W}}{\text{arg max }} \Pr(W_n = w | W_1 = w_1, W_2=w_2, \ldots W_{n-1}=w_{n-1})
$$
But is this really perfect?  I personally doubt.  I think while language is read and written usually from a given direction (e.g. left to right), it is not always done so, and in many cases language is read/written possibly in a funny order as we always do.  E.g. even when I wrote this question, I jumped back and forth, then went to edit it (as I'm doing now).  So I clearly didn't write it from left to right!  Similarly, you, the reader; you won't really read it in a single pass from left to right, will you?  You will probably read it in some funny order and go back and forth for awhile until you conclude an understanding.  So I personally really doubt that the sequential formalism is perfect.
1.2. Joint
Here we find all the $n$ words jointly. Of course ridiculously expensive computationally (if implemented), but our goal here is to only know what is the problem at its fullest.
Basically, we get the $n$ words as follows:
$$
(w_1, w_2, \ldots, w_n) = \underset{(w_1,w_2,\ldots,w_n) \in \mathcal{W}^n}{\text{arg max }} \Pr(W_1 = w_1, W_2=w_2, \ldots W_n=w_n)
$$
This is a perfect representation of language model in my opinion, because its answer is gauranteed to be correct.  But there is this annoying aspect which is that its words candidates space is needlessly large!
E.g. this formalism is basically saying that the following is a candidate words sequence: $(., Hello, world, !)$ even though we know that in (say) English a sentence cannot start by a dot ".".
1.3. Joint but slightly smarter
This is very similar to 1.2 Joint, except that it deletes the single bag of all words $\mathcal{W}$, and instead introduces several bags $\mathcal{W}_1, \mathcal{W}_2, \ldots, \mathcal{W}_n$, which work as follows:

$\mathcal{W}_1$ is a bag that contains words that can only appear as 1st words.
$\mathcal{W}_2$ is a bag that contains words that can only appear as 2nd words.
$\vdots$
$\mathcal{W}_n$ is a bag that contains words that can only appear as $n$th words.

This way, we will avoid the stupid candidates that 1.2. Joint evaluated by following this:
$$
(w_1, w_2, \ldots, w_n) = \underset{w_1 \in \mathcal{W}_1,w_2 \in \mathcal{W}_2,\ldots,w_n \in \mathcal{W}_n) \in \mathcal{W}^n}{\text{arg max }} \Pr(W_1 = w_1, W_2=w_2, \ldots W_n=w_n)
$$
This will also guarantee being a perfect representation of a language model, yet it its candidates space is smaller than one in 1.2. Joint.
1.4. Joint but fully smart
Here is where I'm stuck!
Question rephrase (in case it helps): Is there any formalism that gives the perfect correctness of 1.2. and 1.3., except for also being fully smart in that its candidates space is smallest?

One of your hypothesis is very close to the truth, it's 1.2
So, a language model measures the probability of a given sentence in a language $L$. The sentences can have any length and the sum of probabilities of all the sentences in the language $L$ is 1. It's very difficult to compute, thus people use some simplifications, like say if the words are located far enough from each other, then the occurrence of a current word doesn't depend on a word which was occurred far away in the past.
Each sentence is a sequence $w_1, \dots, w_n$ and a language model computes the probability of the sequence $p([w_1, \dots w_n])$ (it's not joint distrribution yet). It can be decomposed into a joint distribution with some special tokens added $p(BOS, w_1, \dots w_n, EOS])$. BOS is begin of the sentence and EOS is end of sentence. Then this joint distribution can be decomposed using the chain rule $p(BOS, w_1, \dots w_n, EOS]) = p(BOS) p(w_1 | BOS) \Big[ \prod\limits_{i=1}^n p(w_i | BOS, w_1, \dots, w_{i-1}) \Big] p(EOS | BOS, w_1, \dots, w_n)$. There are 2 types of probabilities that are usually modelled differently: a prior probability $p(BOS)$ which is always equal to 1, because you always have BOS as the first token in the augmented sequence. Then conditional probabilities can be computed as follows $p(w_i | BOS, w_1, \dots, w_{i-1}) = \frac{c(BOS, w_1, \dots, w_{i-1}, w_i)}{\sum_{w_i \in W} c(BOS, w_1, \dots, w_{i-1}, w_i)}$. Where $c$ is a counter function that measures how many times a given sequence occured in the dataset you specified to train your model. You can notice it's a maximum likelihood estimate of the unknown conditional probabilities. Obviously if you're using a certain dataset you compute a model of that dataset, not of a language, but that's the to approximate true probabilities of sentences in a language.
The EOS token is needed to make difference between a probability of a non-finished yet sequence and that which has finished, because if you take those counters from above and forget about adding the EOS into your dataset in the end of all sentences, you'll get probabilities that don't sum into 1 (which is bad).


Which model can I use for this problem with multiple inputs and outputs? Which model is the most appropriate for this problem with multiple inputs and outputs?

The data set is

A1, A2, A3, A4, A5, A6, B1, B2, B3, B4

where A1, A2, A3, A4, A5, A6 are the inputs and B1, B2, B3, B4 the outputs (this is what I want the model to predict).

What an LSTM be appropriate for this task? Any advice or hint would be much appreciated. Also if anyone can share already done examples, it would really help me a lot.

This depend on type of data you use.

Time sequence data

If the data advanced in time, a LSTM or similar RNN should be used. RNN calculate output through time. It works very good on time series data as it have a real sense of time. While CNN and MLP could work for time series data, it often don't work that well as different timestep of data is not defined.

Non- Time sequence data

According to you previous comment, the data seems to be of this kind. In the case of your data, a normal Multi-layer perceptron works well for this. The data is a direct mapping between the input and the output. If the input data is an image, use a CNN.

For example code in keras, see here:
You need the pandas module for this to work. Run pip install pandas to install pandas.
    from keras.models import Sequential
    from keras.utils import np_utils
    from keras.layers.core import Dense, Activation, Dropout

import pandas as pd
import numpy as np

# Read data
train = pd.read_csv('../input/train.csv')
labels = train.ix[:,0].values.astype('int32')
X_train = (train.ix[:,1:].values).astype('float32')
X_test = (pd.read_csv('../input/test.csv').values).astype('float32')

# convert list of labels to binary class matrix
y_train = np_utils.to_categorical(labels)

# pre-processing: divide by max and substract mean
scale = np.max(X_train)
X_train /= scale
X_test /= scale

mean = np.std(X_train)
X_train -= mean
X_test -= mean

input_dim = X_train.shape[1]
nb_classes = y_train.shape[1]

# Here's a Deep Dumb MLP (DDMLP)
model = Sequential()
model.add(Dense(128, input_dim=input_dim))
model.add(Activation('relu'))
model.add(Dropout(0.15))
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.15))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))

# we'll use categorical xent for the loss, and RMSprop as the optimizer
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

print("Training...")
model.fit(X_train, y_train, nb_epoch=10, batch_size=16, validation_split=0.1, show_accuracy=True, verbose=2)

print("Generating test predictions...")
preds = model.predict_classes(X_test, verbose=0)

def write_preds(preds, fname):
    pd.DataFrame({"ImageId": list(range(1,len(preds)+1)), "Label": preds}).to_csv(fname, index=False, header=True)

write_preds(preds, "keras-mlp.csv")

Code source: https://www.kaggle.com/fchollet/simple-deep-mlp-with-keras

In conclusion, in the case of our data, use a multi layer perceptron  should work. Hope kit can help you and have a nice day!


Is automated feature engineering a path to general AI? I recently came across the featuretools package, which facilitates automated feature engineering.  Here's an explanation of the package:
https://towardsdatascience.com/automated-feature-engineering-in-python-99baf11cc219

Automated feature engineering aims to help the data scientist by
automatically creating many candidate features out of a dataset from
which the best can be selected and used for training.

I only have limited experience with ML/AI techniques, but general AI is something that I'd been thinking about for a while before exploring existing ML techniques.  One idea that kept popping up was the idea of analyzing not just raw data for patterns but derivatives of data, not unlike what featuretools can do.  Here's an example:

It's not especially difficult to see the figure above as two squares, one that is entirely green and one with a blue/green horizontal gradient.  This is true despite the fact that the gradient square is not any one color and its edge is the same color as the green square (i.e., there is no hard boundary).
However, let's say that we calculate the difference between each pixel and the pixel to its immediate left.  Ignoring for a moment that RGB is 3 separate values, let's call the difference between each pixel column in the gradient square X.  The original figure is then transformed into this, essentially two homogenous blocks of values.  We could take it one step further to identify a hard boundary (applying a similar left-to-right transformation again) between the two squares.

Once a transformation is performed, there should be some way to assess the significance of the transformation output.  This is a simple and clean example where there are two blocks of homogenous values (i.e., the output is clearly not random).  If it's true that our minds use any kind of similar transformation process, the number of transformations that we perform would likely be practically countless, even in brief instances of perception.
Ultimately, this transformation process could facilitate finding the existence of order in data.  Within this framework, perhaps "intelligence" could be defined simply as the ability to detect order, which could require applying many transformations in a row, a wide variety of types of transformations, an ability to apply transformations with a high probability of finding something significant, an ability to assess significance, etc.
Just curious if anyone has thoughts on this, if there are similar ideas out there beyond simple automated feature engineering, etc.

Automated feature engineering, if it is part of any aproach towards general intelligence, cannot be the whole solution. The search for features that are meaningful, as opposed to those that simply exist with no utility, needs some guidance.
In machine learning, feature engineering is typically a search for features that improve performance at a specific task, such as classification or regression. The "intelligence" is partially in the manner of searching, and in setting the goals in the first place. Automated feature engineering typically uses fairly crude search algorithms to look for good features, such as random combination, and follows up with raw processing power in order to cover large numbers of options. Automated feature engineering also does not set the goals for any task, or feed back active behaviour from the output of ML to direct the search. An intelligent agent might actively search for data in its environment in order to test an idea (e.g. repeat an action in order to discover whether the same thing happens, or move in order to observe an interesting event better). Feature engineering is a separate issue to this.
There are some theories of general intelligence that are focused on smart pattern matching as a key component. For instance, Jürgen Schmidhuber has long been a proponent of the reward system for intelligence being compression of observations and predictions. In such a system, better pattern matching discovered by an agent is an intrinsic reward signal as it allows for better compression of world models used by the agent.
Marcus Hutter is another well-known AI researcher, who has proposed a framework called AIXI which incorporates similar ideas. An agent operating using AIXI will benefit from discovering features in its observations that improve its own predictions of what will happen next. Some form of automated feature engineering could well be core to such an agent.


Picking a random move in exploitation in Q-Learning I've been unsure about a principle of Q-Learning, I was hoping someone could clear it up.

When a new state is encountered, and thus there are no existing Q values, and that the algorithm decides to exploit, and not explore, how is the move chosen, since all the values are 0?

Is it chosen randomly? This intuitively would make sense, since after this, the state-move pair would have a value  and thus the matrix would get filled up throughout the iterations. But I just want to make sure I understand this correctly...

Thanks

It depends on the implementation of the software package that you are using. If you call a function that returns the maximum value and all values are the same then it might return the value at first index or some other one. The point is it doesn't matter which action is chosen since all of them are the best at the same time. So it's basically random but you should treat it as if you are trying to pick the best action.


Detect patterns in sequences of actions I have to analyse sequences of actions that look more or less like this JSON blob. The question I'm trying to answer is whether there are recurring (sub)patterns that different users adopt when asked to perform a certain specific task -- in this case, the task is to build a mathematical formula using this editor. In particular I'd like to know if there are multiple significantly different ways in which people build the same expression.

I thought of creating a Markov model, but that would only give me the most likely sequence of actions of length N. An obvious alternative would be to build trees and count how many times a certain path occurs in the dataset. However, the nature of the expression-building process means that the sequences can be polluted by many confounding, non-significant actions (such as streaks of UNDO-REDO, deleting symbols, and the likes).

I might go the "longest common subsequence" route, but I'm not sure that would tell me if there are "significantly different" ways of building the same expression (in quotes because, for now, I don't have a rigorous definition of "significantly different", but, for example, one way would be to drag and drop-in-place all the symbols in the correct order, and another way would be to drag all the symbols onto the canvas, and then place them in the correct spots).

I thought this might be a nice challenge for some AI algorithm, but I'm quite a noob at that, so I'm open to suggestions.

This task falls within the overlapping fields of information extraction and pattern mining. Information extraction involves automatically extracting instances of specified relations from data. While pattern mining involves using data mining algorithms to discover interesting, unexpected and useful patterns between data in databases (Philippe F).

On your question you have stated that you have experimented with markov models with poor results. A better approach if you prefer working with markov models would be to use hierarchical markov models. Hierarchical markov models have multiple 'levels' of states which can describe input sequences at different levels of granularity. Hierarchical markov models are good at categorizing human behavior at various levels of abstraction i.e. a persons location in a room can be further interpreted to determine more complex information such as what activity the person is performing.

However my recommendation is that you implement random forest classifiers for this problem. Random forests provide excellent classification accuracy with a relatively simple implementation. Additionally random forests provide the ability to inspect trees for tweaking parameters to improve accuracy. You can also use cross validation to evaluate your model and calculate its accuracy. Consider using Python's Scikit-learn library's implementation of random forest classifier for this analysis.

In your json code action package you have declared metrics such as DRAG_START, TRASH_SYMBOL, OPEN and CLOSE. My suggestion is that for your model to be accurate you also need to declare lower level actions such as: time between clicks, change in the direction of mouse motion, screen region hover count, task completion time and time between a click and a succeeding mouse movement.

For further reference I recommend that you look at the papers below which I found useful and relevant to your question.

Hierarchical Hidden Markov Models for Information Extraction https://www.biostat.wisc.edu/~craven/papers/ijcai03.pdf

Detecting Abnormal User Behavior Through Pattern-mining Input Device Analytics https://www.ignacioxd.com/files/bib/Dominguez2015-Concentration.pdf

Hierarchical Hidden Markov Model in Detecting Activities of Daily Living in Wearable Videos for Studies of Dementia   https://arxiv.org/ftp/arxiv/papers/1111/1111.1817.pdf


Does epsilon-greedy approach always choose the "best action" (100% of the time) when it does not take the random path? I'm now reading the following blog post but on the epsilon-greedy approach, the author implied that the epsilon-greedy approach takes the action randomly with the probability epsilon, and take the best action 100% of the time with probability 1 - epsilon.

So for example, suppose that the epsilon = 0.6 with 4 actions. In this case, the author seemed to say that each action is taken with the following probability (suppose that the first action has the best value):

action 1: 55% (.40 + .60 / 4)
action 2: 15%
action 3: 15%
action 4: 15%

However, I feel like I learned that the epsilon-greedy only takes the action randomly with the probability of epsilon, and otherwise it is up to the policy function that decides to take the action. And the policy function returns the probability distribution of actions, not the identifier of the action with the best value. So for example, suppose that the epsilon = 0.6 and each action has 50%, 10%, 25%, and 15%. In this case, the probability of taking each action should be the following:

action 1: 35% (.40 * .50 + .60 / 4)
action 2: 19% (.40 * .10 + .60 / 4)
action 3: 25% (.40 * .25 + .60 / 4)
action 4: 21% (.40 * .15 + .60 / 4)

Is my understanding not correct here? Does the non-random part of the epsilon (1 - epsilon) always takes the best action, or does it select the action according to the probability distribution?

Epsilon-greedy is most commonly used to ensure that you have some element of exploration in algorithms that otherwise output deterministic policies.

For example, value-based algorithms (Q-Learning, SARSA, etc.) do not directly have a policy as output; they have values for states or state-action pairs as outputs. The standard policy we "extract" from that is a deterministic policy that simply tries to maximize the predicted value (or, technically, a "slightly" nondeterministic policy in that, in proper implementations, it should break ties (where there are multiple equal values at the top) randomly). For such algorithms, there is not sufficient inherent exploration, so we typically use something like epsilon-greedy to introduce an element of exploration. In these cases, both of the possible explanations in your question are identical.

In cases where your algorithm already produces complete probability distributions as outputs that do not so much focus all of the probability mass on a single or a couple of points, like the probability distribution you gave as an example in your question, it's generally not really necessary to use epsilon-greedy on top of it; you already get exploration inherently due to all actions having a decent probability assigned to them.

Now, I've actually personally mostly worked with value-based methods so far and not so much with e.g. policy gradient methods yet, so I'm not sure whether there tends to be a risk that they also "converge" to situations where they place too much probability mass on some actions and too little on others too quickly. If that's the case, I would expect an additional layer of epsilon-greedy exploration might be useful. And, in that case, I would indeed find your explanation the most natural. If I look through, for example, the PPO paper, I didn't find anything about them using epsilon-greedy in a quick glance. So, I suppose the combination of epsilon-greedy with "nondeterministic" policies (ignoring the case of tie-breaking in value-based methods here) simply isn't really a common combination.


What can be done to correct for sampling bias introduced from (noisy) training data while training a DNN? The obvious solution is to ensure that the training data is balanced - but in my particular case that is impossible. What corrections can one perform in such a scenario?

I know that my training data is heavily biased towards a particular class, say, and I cannot change that. Moreover, the labels are very noisy. Conditioned on this piece of information, is there anything I can do by tweaking the training process itself/ something else, to correct for the bias in the training data?

The data comes from an experiment (from an electron microscope), and I cannot collect more data. It's always going to be biased in this way, so alternatively-biased is also not an option. I'm sorry that I'm unable to provide any more details due to confidentiality.

I feel like from the information your giving (some sort of biased data) you can't get an answer as robust as you'd like (what algorithmic changes can be made).
In general, the reason these methods like DNN's work is that they learn from the data. What you train it to do is what it is capable of, and there's little one can do to 'balance' it to classes of data it just never sees. It's like training someone to do algebra then giving them a trigonometry test. It's all math, sure, but you just never can expect much without the proper learning.
That being said, you should perhaps look at other methods to work with this data, or to approach the problem. Given that you cannot collect unbiased data and that you can't explain more due to confidentiality, I really doubt anyone here can help you that much.
I can at the most point you to this article : "Classification on Data with Biased Class Distribution".
And suggest that perhaps your current approach may not be the most appropriate given the unfortunate circumstances.


Given statements $A$ and $B$, are the formulae $(\lnot A) \land (\lnot B)$ and $(\lnot A) \lor (\lnot B)$ equivalent? If I have 2 statements, say $A$ and $B$, from which I formed 2 formulae:

$f_1: (\lnot A) \land (\lnot B)$

$f_2: (\lnot A) \lor (\lnot B)$

Are $f_1$ and $f_2$ equivalent?

One way of verifying whether two boolean expressions are equivalent is to assign all possibilities to all variables, and comparing all results.

A
B
f1
f2

T
T
F
F

T
F
F
T

F
T
F
T

F
F
T
T

We can see (F, F, F, T) does not equal (F, T, T, T), for example for the assignment (A, B) = (T, F) we get result (f1, f2) = (F, T) , meaning f1 != f2.


Is GPT-3 an early example of Strong AI in a Narrow setting? In GPT-2, the large achievement was being able to generate coherent text over a long-form while maintaining context. This was very impressive but for GPT-2 to do new language tasks, it had to be explicitly fine-tuned for the new task.
In GPT-3 (From my understanding), this is no longer the case. It can perform a larger array of language tasks from translation, open domain conversation, summarization etc. with only a few examples. No explicit fine-tuning is needed.
The actual theory behind GPT-3 is fairly simple, which would not suggest any level of ability other than what would be found in common Narrow Intelligence systems. However, Looking past the media hype and the news coverage, GPT-3 not explicitly programmed to "know" how to do these wider arrays of tasks. In fact, with limited examples, it can perform many language tasks quite well and "learn on the fly" so to speak. To me, this does seem to align fairly well with what most people would consider strong AI but in a narrow context, which is language tasks.
Thoughts? Is GPT-3 an early example of strong AI but in a narrower context?

GPT-3 is based on in-context learning. It’s common wisdom one can hope that bigger models will yield better in-context capabilities. And indeed, this holds true, in the case of GPT-3 175B or "GPT-3".
Neverthless GPT-3 is more powerful than it's predecessors. In some of the tasks, GPT-3 failed miserably. This might be due to the choice to use an autoregressive LM, instead of incorporating bidirectional information (similarly to Bert).
While in-context learning is more straightforward with autoregressive LMs, bidirectional models are known to be better at downstream tasks after fine-tuning.
In the end, training a bidirectional model at the scale of GPT-3 or trying to make bidirectional models work with few-shot learning is a promising direction for future research.
Check out this, this and the paper on Scaling Laws for Neural Language Models.


Are mult-adds and FLOPs equivalent? I am comparing different CNN architectures for edge implementation. Some papers describing architectures refer to mult-adds, like the MobileNet V1 paper, where it is claimed that this net has 569M mult-adds, and others refer to floating-point operations (FLOPs), like the CondenseNet paper claims 274M FLOPs.
Are these comparable? Is 1 multiply-add equivalent to 2 floating-point operations? Any direction will be greatly appreciated.

According to this source, one FLOP is roughly equal to two MAC (multiply accumulate). My guess/understanding would be that the distinction is made because neural nets spend compute overwhelmingly on multiply-accumulate operations, and thus optimizations and statistics over MAC operations would be more significant than FLOPs.


What are the conditions for the convergence of SARSA to the optimal value function? Is it correct that for SARSA to converge to the optimal value function (and policy)

The learning rate parameter  $\alpha$ must satisfy the conditions:
$$\sum \alpha_{n^k(s,a)} =\infty \quad \text{and}\quad \sum \alpha_{n^k(s,a)}^{2} <\infty \quad \forall s \in \mathcal{S}$$
where $n_k(s,a)$ denotes the $k^\text{th}$ time $(s,a)$ is visited
$\epsilon$ (of the $\epsilon$-greedy policy) must be decayed so that the policy converges to a greedy policy.
Every state-action pair is visited infinitely many times.

Are any of these conditions redundant?

I have the conditions for convergence in these notes SARSA convergence by Nahum Shimkin.

The Robbins-Monro conditions above hold for $α_t$.
Every state-action pair is visited infinitely often
The policy is greedy with respect to the policy derived from $Q$ in the limit
The controlled Markov chain is communicating: every state can be reached from any other with positive probability (under some policy).
$\operatorname{Var}{R(s, a)} < \infty$, where $R$ is the reward function


What is the calcium equivalent role in neural networks I understand that neural networks model biological neurons.  Each node in the network represents a neuron cell and the connections between nodes represent the connections between cells.  As in nature, a neuron fires an electrical signal to connected neurons based on some kind of threshold or function that mimics such.

Recent discoveries on how the brain works reveal the importance of calcium within the cells.  See http://link.springer.com/article/10.1007/BF01794675 for more information.  To summarize, calcium affects the regulation, stimulation and transmission of electrical activity as well as the destruction of neurones.

From my study of neural networks, there does not seem to be a calcium equivalent.  Having one would imply that the functions, connections and weights in an artificial network are configured during the training and execution process and can change over time.   I understand that back-propagation is used to train the weights, but have not seen anything that trains the function nor the connections (although a zero weight could imply no connection).

Does anyone know of such a network (or training algorithm)?  If so, do these networks perform better than a network that is pre-configured?

Neural networks don't model biological neurons.

They are at best inspired by biological neurons, in that they get excited by certain inputs and fire once the excitation crosses a threshold. And this second point even holds only approximately because the backpropagation algorithm needs smoothed out steps to learn by gradient descent. And backpropagation is not even inspired by biology, that would rather be hebbian learning.

Generally in machine learning people do what works. The connection to biology is tenuous at best. You will usually not find one to one correspondence of low level details between machine learning setups and biological neurons. For that you'll have to turn to brain simulations.


What techniques are used to make MDP discrete state space manageable? Generating a discretized state space for an MDP (Markov Decision Process) model seems to suffer from the curse of dimensionality.

Supposed my state has a few simple features:

Feeling: Happy/Neutral/Sad
Feeling: Hungry/Neither/Full
Food left: Lots/Some/Low/Empty
Time of day: Morning/Afternoon/Evening/Night
Located: Home/Work/Library/Shops
Money in wallet: \$0, \$0-\$50, \$51-\$200, \$200-\$1000, \$1000+

This is a relatively compact set of information however since the number of states multiplies out, the corresponding MDP state space is 3 x 3 x 4 x 4 x 4 x 5 = 2880.

For a non-trivial problem with a greater number of choices per factor, the state space quickly becomes unmanageably large.

It seems to me that the usefulness of MDPs with large complex problems would be very limited.

Is that the case, or are there ways of keeping the state space manageable for more complex problems?

In general, what is a manageable number of states for an MDP to have, considering the need to generate the transition matrix and reward matrix?

tl:dr
Read chapter 9 of an Introduction of Reinforcement Learning

There is definitely a problem (a curse if you will) when the dimensionality of a task (MDP) grows. For fun, lets extend your problem to a much harder case, continuous variables, and see how we deal with it.

Mood: range [-1, 1]   // 1 is Happy, 0 is Neutral, -1 is Sad

Hunger: range [0, 1]  // 0 is Very Hungry, 1 is Full

Food: range [0, 100] // amount of food, capping at 100 for numbers sake

Time of day: [0, 23] // Hours, 9.5 would be 9:30 am

Position x: range [-10, 10] // assuming the area the agent stays in is 10x10 km

Position y: range [-10, 10]

Money: range [0, 2000]

Now its impossible to even count the number of states the agent could be in. An example of one state would be:

Mood, Hunger, Food, Time, PosX, PosY, Money
.5, .3, 67.4, 4.5, 0, 0, 5

This would mean that our agent has a neutral mood, is kind of hungry, has 67.4% food, the time is 4:30 am, they are in the center of the city (0,0), and have 5 dollars. Which has happened to me once or twice before too so this is reasonable. If we we still wanted to generate a transition matrix and a reward matrix we would have to represent this state uniquely from other possibly similar states such as if the time was 5 pm with all other things equal.

So now how would we approach this?
First lets discretize or breakup each range into little chunks. Then we could have an array of 0s representing our chunks and assign a 1 to the index of the chunk we are in. Lets call this array our feature array.
If we did this for each dimension of our state information, we would end up with some state similar to what you originally proposed. I've chosen some arbitrary numbers to break up the state space and lets see what it does to the example state.

If we broke the Mood range up into 8 chunks of .25 we would have these ranges:

m = Mood

-1.0 ≤ m < -.75
-.75 ≤ m < -.50
-.50 ≤ m < -.25
-.25 ≤ m < 0.0
0.0 ≤ m < .25
.25 ≤ m < .50
.50 ≤ m < .75
.75 ≤ m < 1.0

If we took the mood from the example state (.5), it would land in the 7th range, so then our feature vector for mood would look like:

[0,0,0,0,0,0,1,0] <- represents a mood of .5

Lets do the same for hunger by splitting it into 8 chunks of .125:

[0,0,1,0,0,0,0,0] <- represents a hunger of .3

We could then do something similar for each other state variable and break each state variable's range up into 8 chunks of equal sizes as well. Then after calculating their feature vectors, we would concatenate them all together and have a feature vector that is 56 elements long (7 state variables * 8 chunks each).

Although this would give us 2^56 different combinations of feature vectors, it actually doesn't represent what we wanted to represent. We need features that activate (are 1) when events happen together. We need a feature that is 1 when we are hungry AND when we have food. We could do this by doing a cross product of our 56 element feature vector with itself.

Now we have a 3136 element feature vector that has features like:
  1 if its 3pm and am happy
  1 if am full and out of food
  1 if at coordinate -3, 4 (Position x, Position y)

This is a step in the right direction but still isn't enough to represent or original example state. So lets keep going with the cross products!

If we do 6 cross products we would have 30,840,979,456 features and only 1 of them would be on when our agent has a neutral mood, is kind of hungry, has 67.4% food, the time is 4:30 am, they are in the center of the city (0,0), and have 5 dollars.

At this point you are probably like "Well.. thats a bit much", to which I would agree. This would be the curse of dimensionality, and is the reason transition tables stop being fun (if they ever were).

Instead lets take a different approach, rather than trying to represent an individual state with an individual feature and saying whether that feature is good, lets go back to our nice 56 element feature vector. From here lets instead give a weight (w_i) to each of our 56 features. We have now entered the world of linear function approximation. Although I could explain it here, I think its better explained in Chapter 9 of the Introduction of Reinforcement Learning.


multi vs one prediction using Regression I was trying to build a prediction system where I have the input data arranged in multiple columns. The input data would be of the type where I have

weather,
service type (bronze, silver, gold),
size (xs, s, m, l, xl, xxl),
time,
availability,
pin code, and
the result (target).

Each of the data types is arranged in columns with a specific code. I have read this, this, this , this, and this.
They are helpful but do not give me a clear picture. I would like to achieve a multi-vs-one prediction. Most of the schemes available are one-vs-one where the data is a 1*1 entity.
Here is a sample code that I was working with:
regressionModel = linear_model.LinearRegression()
    """ 3. Processing is not necessary for current concept """
    y = pd.DataFrame(modifiedDFSet['Code'])
    print(y.shape)
    drop2 = ['Code']
    X = modifiedDFSet.drop(drop2)
    print(X.shape)
    """ 4. Data Scaling, Data Imputation is not necessary. Training and Test data is prepared using train-test-split """
    train_data, test_data = train_test_split(X, test_size=0.20, random_state=42)
    """ 5. the Regression Model """
    # h = .02  # step size in the mesh
    # logreg = linear_model.LinearRegression()
    # we create an instance of Neighbours Classifier and fit the data.
    regressionModel.fit(X, y)
    d_predictions = regressionModel.predict(y)

X.shape and y.shape would yield (500, 6) and (500, 1), respectively, which would obviously cause a dimensional error in the d_predictions, meaning the regression model does not take multiple column inputs.
I have a hypothesis that I can create a scoring scheme that will take into account the importance of each of the columns and create a scheme that creates a score and the end result would be a one-vs-one regression problem. Looking for some direction with respect to my hypothesis. Is it correct, wrong or halfway?

I think the model will have no problem taking a multicolumn input. In fact, from your code, this is exactly how you trained it. It expects an input of size [k, 6], where k is k>=1.

Instead you are feeding it with [k, 1] sized data, which are the dimensions of y. So you if you run it like this it should work:

regressionModel.predict(X)


Defining formula for fuzzy equation I'm learning fuzzy logic and more or less understand the basic concept, but i'm having a hard time understanding how to apply it to a method. I tried browsing online for explanation on how to use it, but only found some implementation and test case using the basic form of 4 rules and 3 variables, and 2 rules per variable. Anyway this is an example case, i will use Tsukamoto method.

In this case actually i have 6 rules and 3 variables with 3 rules ver variable, but i will only explain 1 of the variables because i think the rest will have the same solution. I have 3 variables one of them is "size", the range is for small it's 0-2 and for large it's 7-10. The current condition is size = 6.5. The rules is as follow(simplified to only use this variable):

[R1] size = small
[R2] size = medium
[R3] size = large

What i want to know is:

how do i define the formula for medium(the middle rule if the case is different)?
what if the rule is more than 3 (i.e. small, medium, large, ex-large)?

What i understands if the rule is only 2 i can use this formula

small[x]=(max-x)/(max-min)
large[x]=(x-min)/(max-min)

My current approach to this problem is as follow:

small[x]=1; x<=2

medium[x]=(max-x)/(max-min); 2 < x < 7

large[x]=0; x>=7

Is this correct? Also can you refer me to some source to study this? As i mentioned before i can only find some implementation and basic explanation, it's either there is no online source for this or i don't know what to search for. Sorry if it's hard to understand i can edit and post the whole problem if you want, thanks in advance.

Extra question: what is the name of algorithm which can be used to solve the crossing bridge puzzle(the one with timer, max person, and stuff)?i forgot the name.

After i did a bit more study on this, i found this solution.

μsmall[x]

1; -> x≤0
(20-x)/(20-0); -> 0
0; -> x≥20

μmedium[x]

0; -> x≤20 or x≥70
(x-20)/(45-20); -> 20
(70-x)/(70-45); -> 45

μlarge[x]

0; -> x≤70
(x-70)/(100-70); -> 70
1; -> x≥100

For the final result i need to add 3 more rules for the function to work. So it become 9 rules and 3 variables, with 3 rules for each input variables.

To answer both of my question:

Add a triangle graph and use the formula.
Add a triangle graph for each rules.

What i learn from this is the total rules needed to determine the output is the number of input variables rules multiplied(i.e. v1 = 3 rules, v2 3 rules, total rules = 3x3=9 rules). And i'm not really sure about this one but you can combine multiple kind of graphs.


In Faster R-CNN, how can I get the predicted bounding box given the neural network's output? The RPN loss in Faster RCNN paper is
$$
L({p_i}, {t_i}) = \frac{1}{N_{cls}} \sum_{i} L_{cls}(p_i,p_i^*) + \lambda \frac{1}{N_{reg}} \sum_i p_i^* L_{reg}(t_i, t_i^*)
$$
For regression problems, we have the following parametrization
$$t_x=\frac{x - x_a}{w_a}, \\ t_y=\frac{y−y_a}{h_a}, \\ t_w= \log \left( \frac{w}{w_a} \right),\\ t_h= \log \left(\frac{h}{h_a} \right)$$
and the ground-truth labels are
$$t_x^*=\frac{x^* - x_a}{w_a},\\ t_y^*=\frac{y^*−y_a}{h_a}, \\ t_w^*= \log \left( \frac{w^*}{w_a} \right), \\ t_h^*= \log \left(\frac{h^*}{h_a} \right)$$
where

$x$ and $y$ are the two coordinates of the center, $w$ the width, and $h$ the height of the predicted box.

$x$ and $y$ are the two coordinates of the center, $w$ the width, and $h$ the height of the anchor box.

$L_{reg}(t_i, t_i^*) = R(t_i − t_i^*)$, where $R$ is a robust loss function (smooth $L_1$)

These equations are unclear to me, so here are my two questions.

How can I get the predicted bounding box given the neural network's output?

What exactly is the smooth $L_1$ here? How is it defined?

If I understood well you have 2 questions.

How to get the bounding box given the network output
What Smooth L1 loss is

The answer to your first question lies in the equation (2) in the section 3.2.1 from the Faster R-CNN paper. As all anchor based object detector (Faster RCNN, YOLOv3, EfficientNets, FPN...) the regression output from the network are not the bounding box coordinates. The regression output predicts the shift of the predicted bounding box with respect to the selected anchor (all of these networks, use more than 1 anchor per location, check section 3.1.1 from the paper).
So basically what your network predict is $t_x, t_y, t_w, t_h$:

And the bounding box coordinates are $x, y, w, h$, and the anchor coordinates are $x_a, y_a, w_a, h_a$. So in order to compute $x, y, w, h$ from $t_x, t_y, t_w, t_h$, you just have to invert the equations above. However I think you could gain more intuition about it if you take your time and read the whole section 3.1 from the paper. I know sometimes is a pain, but you will grasp the high level concept.
With regard to your second question. Yes the loss is computed with the output from the network and the "coded" ground truth, meaning you compute the loss with the paramters $t$ (predicted) against $t^*$ (coded ground truth) instead for computing loss with the real coordinates of the bounding boxes (decoded output from the network). For the equation on Smooth L1 loss check this wonderful documentation.


How to detect multiple playing cards of the same class with a neural network? I want to train an AI to detect the class (i.e. suit and rank) of playing cards. Playing cards from different decks may use slightly different shapes or colors to represent these attributes, and I want the system to work across many decks. I bought many different decks, scanned and labeled them. Next up would be to create training data with an augmentation library. I found two examples of how other people did that:

Image detection using YOLO algorithm and poker cards
Playing card detection with YOLO

The problem is that I want my AI to be able to detect multiple cards of the same class in one picture. In the examples above, they put a label on the top left corner of a card. This makes sense since it is a very good indicator of what class the card is in. Unfortunately, every card has two of these labels.
I think their solution returns "detected" if any instance of a given label is found. But many cards with the same label could be in the same picture, and I am not sure if I can detect the quantity of the class with this solution. For example, if there are 2 Ace of Clubs in a picture, I would like my system to output "2", rather than "detected".
Do you think it is feasible to mark the whole card, prepare the data accordingly, and train an AI that detects the count as well?

Although it was not crystal clear, we'll assume that by, "Multiple cards of the same class in one picture," is meant that multiple cards of identical suit and rank will be grouped together in the same example image but each card in the image will be selected from a unique deck.

That arrangement would only be fruitful if the objective of training was to classify, flag, or otherwise analyze the same kind of groupings after training. Otherwise, it would likely be most productive and efficient to first devise a way to divide up the images by deck so that the focus of learning is detection of the three dimensions or perhaps just the first two, depending on the intended use of the trained network.

Rank
Suit
Style


How can the derivative of a neural network be calculated, given no mathematical expression? Neural networks (NNs) are used as approximators in reinforcement learning (RL). To update the policy in RL, the actor network's gradients w.r.t its weights are needed. Since NN doesn't have a mathematical expression to work with, how can its derivatives be calculated?

I think what you mean to ask is how can differentiation occur when there's no obvious neural network function to differentiate?

Don't worry - lots of people get confused about this, because it seems like an obvious hole in the puzzle. As mentioned by @AtillaOzgur, neural networks use partial differentiation through backpropagation.

First, take the output of all the neurons (except the one you're about to differentiate by) as a function:

The above diagram represents the output of one neuron. Do this for every neuron in your network until you have a set. Let's call this set function NN. The output of NN (given all your neuron outputs) is what you'd normally plug into your RL policy.

You then differentiate NN by a single neuron (n) as shown:

$$\frac{\partial NN}{\partial n} = \lim_{h\to0} \left(\frac{NN(\text{all other neuron outputs}, n + h) - NN(\text{all other neuron outputs}, n)}{h} \right)$$

In reality however, it's the partial derivative of the activation function (A) with respect to the output of a single neuron (n):

$$\frac{\partial A}{\partial n}$$

So, depending on your activation function, you just plug in your neuron output to a certain expression and you've found the value by which to update your neural network.

I hope this helps. Deep learning is definitely a field with a learning curve, but places like StackExchange are great resources to help you out.


How to resolve lexical ambiguity in natural language processing? I'm interested in implementing a program for natural language processing (aka ELIZA).

Assuming that I'm already storing semantic-lexical connections between the words and its strength.

What are the methods of dealing with words which have very distinct meaning?

Few examples:

'Are we on the same page?'

The 'page' in this context isn't a document page, but it's part of the phrase.
'I'm living in Reading.'

The 'Reading' is a city (noun), so it's not a verb. Otherwise it doesn't make any sense. Checking for the capital letter would work in that specific example, but it won't work for other (like 'make' can be either verb or noun).
'I've read something on the Facebook wall, do you want to know what?'

The 'Facebook wall' has nothing to do with wall at all.

In general, how algorithm should distinguish the word meaning and recognise the word within the context?

For example:

Detecting the word for different type of speech, so it should recognise whether it's a verb or noun.
Detecting whether the word is part of phrase.
Detecting word for multiple meaning.

What are the possible approaches to solve that problem in order to  identify the correct sense of a word with the context?

In general, how algorithm should distinguish the word meaning and recognise the word within the context?

I don't think anybody knows how to answer this for the general case. If they did, they'd have basically solved AGI.  But we can certainly talk about techniques that get part-of-the-way there, and approaches that could work.

One thing I would consider trying (and I don't know off-hand if anybody has tried this exact approach) is to model the disambiguation of each word as a discrete problem for a Bayesian Belief Network where your priors (for any given word) are based on both stored "knowledge" as well as the previously encountered words in the (sentence|paragraph|document|whatever).  So if you "know", for example, that "Reading is a city in the UK" and that "place names are usually capitalized", your network should be strongly biased towards interpreting "Reading" as the city, since nothing in the word position in the sentence strongly contradicts that.

Of course I'm hand-waving around some tricky problems in saying that, as knowledge representation isn't exactly a solved problem either.  But there are at least approaches out there that you could use.  For example, you could use the RDF / triple based approach from the Semantic Web world.  Finding a good way to merge that stuff with a Bayesian framework could yield some interesting results.

There has been a bit of research on "probabilistic RDF" that you could possibly use as a starting point.  For example:

http://om.umiacs.umd.edu/material/papers/prdf.pdf

http://ceur-ws.org/Vol-173/pos_paper5.pdf

https://www.w3.org/2005/03/07-yoshio-UMBC/

http://ebiquity.umbc.edu/paper/html/id/271/BayesOWL-Uncertainty-Modeling-in-Semantic-Web-Ontologies

http://www.pr-owl.org/


Based on the Turing test, what would be the criteria for an agent to be considered smart? Based on the Turing test, what would be the criteria for an agent to be considered smart?

If an artificial agent (AA) passes the (standard) Turing test (i.e. where you have to imitate a human that speaks), then, on average, the AA should be able to imitate any human in any situation that mainly requires the conversation abilities and common-sense knowledge of a human, without being ever recognized as an AA.
For example, if you want to talk about football, you don't expect the AA only to say "I don't know" or to clearly avoid a topic (e.g. by redirecting you to a search engine) when it doesn't know something (although some humans behave in this way), but you expect it to have common-sense knowledge, such as that Messi, Cristiano Ronaldo, Pelé, Maradona, etc., are among the best players of all time, and this should be precious information that the AA should have in any case, even if it doesn't know much about football.
You also expect it to be emotional and have a personality, given that humans are emotional and have personalities. So, in the example above, maybe the AA could say that Maradona is its favorite player, and then it could explain why in an emotional way (e.g. by changing the tone of the voice).
The AA should also be able to keep track of (almost) everything you and it said, and it should be able to contextualize very well, as humans do. Some personal assistants already take context into account, but they don't do this very well or just do it to a little extent.
The AA should also be able to reason given the current situation. For example, if you explain something to the AA, you expect it to infer or predict something based on the information it has acquired and the common-sense knowledge.
Moreover, when you speak or write to the AA, you don't expect it to regularly hear badly or not understand what you say or ask (and ask you to repeat), but you expect it to understand well what you say almost always, provided you don't talk or write trash. You also don't expect big delays and you expect the AA to at least say something while it searches for a more appropriate answer, although not all humans behave in this way, but I think that interjections or words such as "hm", "well", "let me think...", etc., will be very important to make the AA look or sound like a human. In general, the AA should be as interactive as a human.
These are some traits that the AA absolutely needs to have in order to pass the Turing test (and be considered intelligent according to that test), but there are probably many others.


Does summing up word vectors destroy their meaning? For example, I have a paragraph which I want to classify in a binary manner. But because the inputs have to have a fixed length, I need to ensure that every paragraph is represented by a uniform quantity.

One thing I've done is taken every word in the paragraph, vectorized it using GloVe word2vec and then summed up all of the vectors to create a "paragraph" vector, which I've then fed in as an input for my model. In doing so, have I destroyed any meaning the words might have possessed? Considering these two sentences would have the same vector:
"My dog bit Dave" & "Dave bit my dog", how do I get around this? Am I approaching this wrong?

What other way can I train my model? If I take every word and feed that into my model, how do I know how many words I should take? How do I input these words? In the form of a 2D array, where each word vector is a column?

I want to be able to train a model that can classify text accurately.
Surprisingly, I'm getting a high (>90%) for a relatively simple model like RandomForestClassifier just by using this summing up method. Any insights?

Summing up a sequence of word vector maybe used in practice sometimes. However, the operation of addition is non-reversible, meaning that once you sum up a few numbers, you cannot get the original numbers back. However summing up a sequence of word vectors may work depending on your task. You should also normalize the values, or just use average value.

For details: https://towardsdatascience.com/document-embedding-techniques-fed3e7a6a25d#ecd3

To feed data with different lengths, you can also try padding and trimming it. Set a constant L for length of paragraph and trim/pad all list of word vectors to this length. Padding adds 0 vectors to the begining of the list and trimming trims the first part of a text until it is equal to L. Even in LSTM networks padding and trimming is still used as even though you can feed as long of a text to a LSTM as you want, you still have to process the word vectors in batches, which requires them to be the same length.

Example code in python on padding/trimming vector:

def pad_trim(list_vec, L):
    //list vec: {vec1,vec2,vec3......}
    //Assuming vecN have a size of 200
    return list_vec[-L:0] if len(list_vec) > L else [[0] * 200] * (L-len(list_vec)) + list_vec

However in inference, you can ignore maximum length if you used a RNN based method, although as the network has not been trained on lengths more than L, it may perform better or worse.

Generally speaking, you should go for concatenating if possible, so you can keep all information in the sentence. However both may work just fine depending on your task.

For RNN based and CNN based model example, you should check this out:
https://medium.com/jatana/report-on-text-classification-using-cnn-rnn-han-f0e887214d5f


Using UMAP, PCA or t-SNE to find the separating hyperplane? Is it possible to use t-SNE, PCA or UMAP to find separating hyperplane?

Assume we have data points in high dimensional space and we want to phase separate it into two sets of points?

Is there a way to use t-SNE, PCA or UMAP for such a task?

All the methods are basically manifold methods which are used to squeeze hyper dimensions to two or three dimensions with a certain amount of information loss. So whatever you "see" with these methods are not real and deceiving. You may see a separation of data points in 3D but when they are mapped onto the actual dimensions it may be complete rubbish.

By definition, these methods do not provide a way to find separating hyperplane. They just tell you some condensed information about the whole data. It is generally advised to use methods like t-SNE with caution.


Trading off "Memory" vs "Optimization" I've been researching the following topic. Or rather, I would like to but I can't find anything because I'm not sure what to look for.

I am interested weather there are some concepts or models that explain how humans (or cognitive systems in general) trade off remembering what to do in a particular situation versus thinking about the board state and devising a new solution. Consider the following example:

A person plays chess. On their turn they must decide what to do. They could act according to one of the following extreme strategies:

Remembering all possible board configurations and how the game ended (we'll call that Memory)
Use the known set of rules to simulate the game in their mind and choose the optimal path (Optimization)

I'd say the best choice is a trade off between these two. But how? When to choose one of the strategies?

Is there a research field that is working on these kind of questions? What would I look for?

If you can remember everything and there's no randomisation in your outcome like chess, there is absolutely no reason not to do that. Anybody who can remember all the possible board configurations in chess, by definition plays perfect chess. A perfect player would never lose.

Unfortunately, most practical problems can't be solved by brute-force, and that includes chess. All chess players learn general chess principles. Nobody could memorize all possible opening variations.

If you can somehow memorize every possibility, why not? But can you?


How is the state-value function expressed as a product of sums? The state-value function for a given policy $\pi$ is given by
$$\begin{align}
V^{\pi}(s) &=E_{\pi}\left\{r_{t+1}+\gamma r_{t+2}+\gamma^{2} r_{t+3}+\cdots \mid s_{t}=s\right\} \\
&=E_{\pi}\left\{r_{t+1}+\gamma V^{\pi}\left(s_{t+1}\right) \mid s_{t}=s\right\} \tag{4.3}\label{4.3} \\
&=\sum_{a} \pi(s, a) \sum_{s^{\prime}} \mathcal{P}_{s s^{\prime}}^{a}\left[\mathcal{R}_{s s^{\prime}}^{a}+\gamma V^{\pi}\left(s^{\prime}\right)\right] \tag{4.4}\label{4.4}
\end{align}$$
It is given in section 4.1 of the first edition of Sutton and Barto's book (equations 4.3 and 4.4).
I don't understand how equation \ref{4.4} derives from equation \ref{4.3}. How can I get the product in equation \ref{4.4} from the expectation in equation \ref{4.3}?

A quick review of resolving expectations: If you know that a discrete random variable $X$, drawn from set $\mathcal{X}$ has probability distribution $p(x) = \mathbf{Pr}\{X=x \}$, then
$$\mathbb{E}[X] = \sum_{x \in \mathcal{X}} xp(x)$$
This equation is the core of what is going on when resolving the expectation in your quoted equation.
Resolving the expectation to show how the value function of a state relates to the possible next rewards and future states means summing up all possible rewards and next states. There are two components to the distribution over the single step involved - the policy $\pi(a|s)$, and the state progression $P^a_{ss'}$. As they are both independent probabilities, they need to be multiplied to establish the combined probability of any specific trajectory.
So, looking at only a single trajectory starting from state $s$, the trajectory of selecting action $a$ and ending up in state $s'$ has a probability of:
$$p_{\pi}(a,s'|s) = \pi(a|s) P^a_{ss'}$$
Iterating over all possible trajectories to get the expected value of some function of the end of the trajectory $f(s,a,s')$ looks like this:
$$\mathbb{E}[f(S_t, A_t, S_{t+1})|S_t=s] = \sum_a \pi(a|s)\sum_{s'}P_{ss'}^a f(s,a,s')$$
It is important to note that the sums are nested here, not separately resolved then multiplied. This is standard notation, but you could add some brackets to show it:
$$\mathbb{E}[f(S_t, A_t, S_{t+1})|S_t=s] = \sum_a \pi(a|s)\left(\sum_{s'}\left(P_{ss'}^a f(s,a,s')\right)\right)$$
In the equation from the book, $f(s,a,s') = R_{ss'}^a + \gamma v_{\pi}(s')$


Is AGI likely to be developed in the next decade? AI experts like Ben Goertzel and Ray Kurzweil say that AGI   will be developed in the coming decade. Are they credible?

As a riff on my answer to this question, which is about the broader concern of the development of the singularity, rather than the narrower concern of the development of AGI:

I can say that among AI researchers I interact with, it far more common to view the development of AGI in the next decade as speculation (or even wild speculation) than as settled fact.

This is borne out by surveys of AI researchers, with 80% thinking "The earliest that machines will be able to simulate learning and every other
aspect of human intelligence" is in "more than 50 years" or "never", and just a few percent thinking that such forms of AI are "near". It's possible to quibble over what exactly is meant by AGI, but it seems likely that for us to reach AGI, we'd need to simulate human-level intelligence in at least most of its aspects. The fact that AI researchers think this is very far off suggests that they also think AGI is not right around the corner.

I suspect that the reasons AI researchers are less optimistic about AGI than Kurzweil or others in tech (but not in AI), are rooted in the fact that we still don't have a good understanding of what human intelligence is. It's difficult to simulate something that we can't pin down. Another factor is that most AI researchers have been working in AI for a long time. There are countless past proposals for AGI frameworks, and all of them have been not just wrong, but in the end, more or less hopelessly wrong. I think this creates an innate skepticism of AGI, which may perhaps be unfair. Nonetheless, expert opinion on this one is pretty well settled: no AGI this decade, and maybe not ever!


How to perform binary classification when one class is more predominant than the other? Assuming we have big $m \times n$ input dataset, with $m \times 1$ output vector. It's a classification problem with only two possible values: either $1$ or $0$.
Now, the problem is that almost all elements of the output vector are $0$s with a very few $1$s (i.e. it's a sparse vector), such that if the neural network would "learn" to give always 0 as output, this would produce high accuracy, while I'm also interested in learning when the 1s occurs.
I thought one possible approach could be to write a custom loss function giving more weight to the 1s, but I'm not completely sure if this would be a good solution.
What kind of strategy can be applied to detect such outliers?

As described in this post, this problem is known as "unbalanced dataset" problem, which can have different solution approaches. If you use supervised learning, augmentation approaches
could  help. Otherwise, unsupervised approaches need some proper distance measure for outliers detection.


How to perform PCA in the validation/test set? I was using PCA on my whole dataset (and, after that, I would split it into training, validation, and test datasets). However, after a little bit of research, I found out that this is the wrong way to do it.
I have few questions:

Are there some articles/references that explain why is the wrong way?

How can I transform the validation/test set?

Steps to do PCA (from https://www.sciencedirect.com/science/article/pii/S0022460X0093390X):

zero mean

$$\mu = \frac{1}{M}\sum_{i=1}^{M} x_{i}$$
where x is my training set

centering (variance)

$$S^{2} = \frac{1}{M}\sum_{i=1}^{M} (x_{i}-\mu)^{T}(x_{i}-\mu)$$

use (1) and (2) to transform my original training dataset

$$x_{new} = \frac{1}{\sqrt{M}} \frac{(x_{i} - \mu)}{S}$$

calculate covariance matrix (actually correlation matrix)

$$C= x_{new}^T x_{new}$$

take the k-eigenvectors (/phi) from the covariance matrix and defined the new space for my new dimension training set (where k are the principal components that I choose according to my variance)

$$ x_{new dim} = x_{new}\phi$$
Ok, then I have my new dimensional training dataset after PCA (till here it's right, according to other papers that I have read).
The question is: *What I have to do now for my validation/testing set? Just the equation below?
$$y_{new dim} = y\phi $$
where y is my (for example) validation original dataset?
Can someone explain the right thing to do?

for the first point I'm very sorry that I cannot give you any literature on this, but I might be able to explain you, why you don't take PCA on both datasets independently.

Principal components analysis is simply a transformation of your data into another (less dimensional) coordinate system. The axis for your new coordinate system are defined by the principal components (i.e. eigen-vectors) of your covariance matrix.

Since you will train your machine learning algorithm in the domain that is generated by the PCA, your test data must be exactly in the same domain. So as you said, you use exactly the same transformation for the test data as for the training data, i.e.
$y_{newdim} = y \phi $.
Of course if you applied standardization to your training data, you have to apply the same standardization to your test data. So you need to store the mean $\mu_x$ and the standard deviation $S_x$ and also standardize your test data y to
\begin{equation}
y_{standardized} = \dfrac{y_i - \mu_x}{S_x}
\end{equation}

Note here that you have an error in your standardization formula (you do not need to divide by $M$).

The point is that the principal components of your test data $\phi_{y}$ would not match the principal components of your training data $\phi_{x}$. Thus the transformations from original space into PCA-space $\Phi_x(u)$ and $\Phi_y(u)$ would diverge and similar data points in the original space might be far away in the PCA-representation and vice versa. This is why you generate the mapping $x_{newdim} = \Phi_x(x) = x\phi$ and apply it also on the test data.

I hope I could make it clear to you.

Best


How large should the corpus be to optimally retrain the GPT-2 model? I just started working with the GPT-2 models and want to retrain one on a pretty narrow topic, so I have problems finding training material.
How large should the corpus be to optimally retrain the GPT-2 model? And what is the bare minimum size? Should it simply be as large as possible or can it flip over and make the model worse in some way?
I am also not certain how many steps you should let the retraining run. I have been using 6000 steps when testing, and it seems not much happens after that, loss only moved from 0.2 to 0.18 last 1000 steps.

After doing many runs with GPT-2 355M I come to the conclusion that anything below 20k tokens yield worse and worse results the less tokens you have regardless of how many steps you train for. For my application the sweet spot is around 30-35k tokens.


Why do knowledge-based agents only add a sentence to the knowledge base when it is 100% sure the sentence is true? According to Russell and Norvig, a knowledge-based agent will only add a sentence to its knowledge base if it follows logically from what it previously knows, or directly observes. To follow logically essentially means that if the premises are true, then the conclusions are guaranteed to be true. So the agent will only add the sentence if it 100% sure the sentence is true.

Is this hyperskepticism in logic justified? Couldn't the agent be more efficient if it added a sentence if it were 99% sure it were true? It could potentially add a lot more true sentences, and only occasionally add false ones. There would need to be a mechanism for unlearning sentences, but as long as the vast majority of sentences added are true, why couldn't that be done?

I essentially asked this question here, and someone suggested that I post it here.

Mostly because traditional "knowledge based systems" are based purely on deductive logic and that's just the way deduction works.  It only deals with what consequences must follow from the premises.  Traditionally these systems didn't deal with probabilistic knowledge or other less strict forms of reasoning, like abduction.

That said, it is possible to build a knowledge based system that includes probabilistic reasoning, and other reasoning systems (like abduction).

I'll add some more on that point when I have a little more time.


Pose estimation using CNNs on Point clouds In the case of single shot detection of point clouds, that is the point cloud of an object is taken only from one camera view without any registration. Can a Convolutional Network estimate the 6d pose of objects (initially primitive 3D objects -- cylinders, spheres, cuboids)?
The dataset will be generated by simulating a depth sensor using a physics engine (ex:gazebo) and primitive 3D objects are spawned with known 6d pose as ground truth. The resulting training data will be the single viewed point cloud of the object with the ground truth label (6d pose)?

The answer is yes this is possible and here are the papers where they do almost exactly the same project you are describing above. Although none of the bellow combine gazebo, single point/single shot, 6D-pose and CNNs. In order to use synthetic data to train a model that works on real data.

Pose Estimation by Key Points Registration in Point Cloud
 (2019) By Weiyi ZHANG, Chenkun QI This paper uses gazebo to test validate the model they made.
Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics (2017) by Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, Ken Goldberg
Real-Time Seamless Single Shot 6D Object Pose Prediction (2018) by Bugra Tekin, Sudipta N. Sinha, Pascal Fua
This paper sounds like exactly what you want to do.
SSD-6D: Making RGB-Based 3D Detection and 6D Pose Estimation Great Again (2017) by Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, Nassir Navab
This paper is also referenced by them.

The model will be able to be trained but how effect a model trained on the synthetic data will be able to properly function on real data will be the challenge.


Difference between softmax and Logistic Regression? So I read about softmax from this article. Apparently to me these 2 are almost similar, except that the probability of all classes in softmax add to 1. According to their last paragraph for number of classes = 2, softmax reduces to LR. What I want to know is other than number of classes = 2 what are the essential differences between LR and softmax. Like in terms of:

Performance.
Computational Requirements.
Ease of calculation of derivatives.
Ease of visualization.
Number of minimas in the convex cost function, etc.

Other differences are also welcome!

EDIT : I am asking for relative comparisons only, so that at the time of implementation I have no difficulty in selecting which method of implementation to use.

As written, SoftMax is a generalization of Logistic Regression.
Hence:

Performance
If the model has more than 2 classes then you can't compare. Given K = 2 they are the same.
Computation Requirements
Please explain as the computational requirements require the data, enough memory to hold it and enough time to let run.
Ease of Calculation of Derivatives
The cost function is summation hence once you do it for one element you do it fol all.
Ease of Visualization
Well, it is easy to visualize the Confusion Matrix even for K = 10 classes. So no issue here.
Cost Function
The cost function is convex. Yet not Strictly Convex hence there infinite number of minimas.


Understanding the node information score in the paper "Hierarchical Graph Pooling with Structure Learning" The paper Hierarchical Graph Pooling with Structure Learning (2019) introduces a distance measure between:

a graph's node-representation matrix $\text{H}$, and
an approximation of this constructed from each node's neighbours' information $\text{D}^{-1}\text{A}\text{H}$:

Here, we formally define the node information score as the Manhattan distance between the node representation itself and the one constructed from its neighbors:
$$\mathbb{p} = \gamma(\mathcal{G}_i) = ||(\text{I}^{k}_{i} - (\text{D}^{k}_{i})^{-1}\text{A}^{k}_{i})\text{H}^{k}_{i}||  $$

(where $\text{A}$ and $\text{D}$ are the Adjacency and Diagonal matrices of the graph, respectively)
Expanding the product on the RHS we get (ignoring index notation for simplicity):
$$||\text{H} - (\text{D}^{-1}\text{A}\text{H})||$$
Problem: I don't see how $\text{D}^{-1}\text{A}\text{H}$ is a "node representation... constructed from its neighbors".
$\text{I} - \text{D}^{-1}\text{A}$ is clearly equivalent to the Random Walk Laplacian, but it's not immediately obvious to me how multiplying this by $\text{H}$ provides per-node information on how well one can reconstruct a node from its neighbours.

Here, $H$ is a $n * d$ matrix where $n$ is the number of total nodes in the graph and $d$ is the dimension of embedding of each node.

Using the notation in the question, the basic GNN formulation without self loop is: $\text{D}^{-1}\text{A}\text{H}$. If you study this equation closer then you will find that the $i^{th}$ row of $\text{A}\text{H}$ generates the $i^{th}$ node's representation by summing the node representation of its neighboring nodes. Multiplying it with $\text{D}^{-1}$ makes the aggregated representation normalized with respect to the degree of a node (number of neighbors).

By defining a metric called information score:
$$||\text{H} - (\text{D}^{-1}\text{A}\text{H})||$$
we will be getting low values for nodes that are well represented by their local neighborhood nodes and high values for nodes that are having a hard time being represented/summarized by its neighboring nodes.
To approximate the graph information, the authors choose to preserve the nodes that can not be well represented by their neighbors, i.e., the nodes with relatively larger node information score will be preserved in the construction of the pooled graph, because the authors believe it can provide more information.


Extract features with CNN and pass as sequence to RNN I read an article about captioning videos and I want to use solution number 4 (extract features with a CNN, pass the sequence to a separate RNN) in my own project.
But for me, it seems really strange that in this method we use the Inception model without any retraining or something like that. Every project has different requirements and even if you use pretrained model instead of your own, you should do some training.
And I wonder how to do this? For example, I created a project where I use the network with CNN layers and then LSTM and Dense layers. And in every epoch, there is feed-forward and backpropagation through the whole network, all layers. But what if you have CNN network to extract features and LSTM network that takes sequences as inputs. How to train CNN network if there is no defined output? This network should only extract features but the network doesn't know what features. So the question is: How to train CNN to extract relevant features and then passing these features to LSTM?

The approach that you don't train the whole net, but just the latter part of it (all starting with lstm in our case), can actually work. The idea is that the inception was already pretrained a very large dataset (imagenet for instance). And it's capable of extracting some useful information from it. Actually there are different domains of images in the imagenet and the inception net needed to capture a vast variety of input information to classify images well. The idea is that the pretrained inception is already capable to extract almost everything what could possibly be useful (unless your images aren't something completely different from imagenet, but that a rare case). Then you adapt the lstm layers and the fully connected layers to correctly process that information. Maybe you aren't going to get the perfect score with this approach and maybe it's better to train the whole large net including the inception part on the new data to lower the distributional shift and that's what people usually do in fact, but it takes more time to train and if you don't have enough data you won't be able to achieve results that are significantly better than those with a frozen CNN part.


Is my backpropagation code correct? I am trying to implement the back-propagation algorithm for the following neural network.

2 input -> 3 input layer -> 1 output
Activation f(x) -> sigmoid
Loss f(x) -> Yexpected-Yresult

This is the Matlab/Octave function for backpropagating an XOR ANN. However, it doesn't seem to work, and it converges to 0.5. Is it correct?

sigmoid = @(z) 1./(1 + exp(-z));
sig_der = @(y) sigmoid(y).*(1-sigmoid(y));
    function [w1, w2, b1, b2] = backprop(w1,w2,b1,b2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i)
      delta2 = (sig_der(last_layer,sigmoid)).*cost;
      delta1 = (sig_der(mid_layer,sigmoid)).*sum(delta2.*w2);
      w2 = w2 + 0.001 .* mid_layer .* delta2;
      w1 = w1 + 0.001 .* data(1:2,i) .* delta1.';
      b1 = b1 + 0.001 .* delta1';
      b2 = b2 + 0.001 .* delta2;
    end

Here is the code:

clear

graphics_toolkit("gnuplot")
sigmoid = @(z) 1./(1 + exp(-z));
sig_der = @(y) sigmoid(y).*(1-sigmoid(y));

function [cost, mid_layer, last_layer] = forward(w1,w2,b1,b2,data,sigmoid,i)
  mid_layer = sigmoid(sum(data(1:2,i).*w1)'-b1);
  last_layer = sigmoid(sum(mid_layer.*w2)-b2);
  cost = data(3,i)-last_layer;
end

function [w1, w2, b1, b2] = backprop(w1,w2,b1,b2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i)
  delta2 = (sig_der(last_layer,sigmoid)).*cost;
  delta1 = (sig_der(mid_layer,sigmoid)).*sum(delta2.*w2);
  w2 = w2 + 0.00001 .* mid_layer .* delta2;
  w1 = w1 + 0.00001 .* data(1:2,i) .* delta1';
  b1 = b1 + 0.00001 .* delta1;
  b2 = b2 + 0.00001 .* delta2;
end

data(:,1)=[0; 0; 0];
data(:,2)=[1; 0; 1];
data(:,3)=[0; 1; 1];
data(:,4)=[1; 1; 0];

w1=rand(2,3)./2.*(rand(2,3).*-2+1);
w2=rand(3,1)./2.*(rand(3,1).*-2+1);
b1=rand(1,3)./2;
b2=rand(1,1)./2;

for j=1:20000
  for i=1:4
    [cost, mid_layer, last_layer] = forward(w1,w2,b1,b2,data,sigmoid,i);
    [w1, w2, b1, b2] = backprop(w1,w2,b1,b2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i);
    cost_mem(j,i)=cost;
  end
end
toc

Thanks for everyone's help. I have now solved the problem. I hadn't quite understood the back propagation algorithm. I invite people to take a look at this link: backpropagation @ AGH UST which has solved my problem.

Code:

clear
pkg load image
graphics_toolkit("gnuplot")
tic
sigmoid = @(z) 1./(1 + exp(-z));
sig_der = @(y) sigmoid(y).*(1-sigmoid(y));

function [cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i)
  mid_layer(:,1)=sum(w1.*data(1:2,i));
  mid_layer(:,2)=sigmoid(mid_layer(:,1));
  last_layer(:,1)=sum(mid_layer(:,2).*w2);
  last_layer(:,2)=sigmoid(last_layer(:,1));
  cost = data(3,i)-last_layer(:,2);
end

function [w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i)
  delta(1) = cost;
  delta(2:4) = cost .* w2;
  w2 = w2 + 0.99 .* delta(1) .* mid_layer(:,2) .* sig_der(last_layer(:,1));
  w1 = w1 + 0.99 .* delta(2:4) .* sig_der(mid_layer(:,1))' .* data(1:2,i);
  %b2 = b2 + 0.01 .*
  %b1 = b1 + 0.01 .*
end

tic

data(:,1)=[0; 0; 0];
data(:,2)=[1; 0; 1];
data(:,3)=[0; 1; 1];
data(:,4)=[1; 1; 0];

w1=rand(2,3)./2.*(rand(2,3).*-2+1);
w2=rand(3,1)./2.*(rand(3,1).*-2+1);
%b1=ones(1,3);%rand(1,3)./2;
%b2=ones(1,1);%rand(1,1)./2;

for j=1:10000
  for i=[randperm(4)]
    [cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i);
    [w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i);
    cost_mem(j,i)=cost;
  end
end

toc

%{
dlmwrite("/tmp/weights_1",w1);
dlmwrite("/tmp/weights_2",w2);
%}


How would the performance of federated learning compare to the performance of centralized machine learning when the data is i.i.d.? How would the performance of federated learning (FL) compare to the performance of centralized machine learning (ML), when the data is independent and identically distributed (i.i.d.)?
Moreover, what is the difference in the performance of FL when the data is i.i.d. as compared to non-i.i.d?

There are some works that do this comparison. Briefly, it's been observed that the performance of models trained via FL drops as data distributions between participating agents differ. When data is IID-like though, performance is comparable to centralized training. Some works that I'm aware of are as follows:

Overcoming Forgetting in Federated Learning on Non-IID Data
Improving Accuracy of Federated Learning in Non-IID Settings
Federated Learning with Non-IID Data

There are probably many more around. It's an active area of research.