How would I compute the optimal state-action value for a certain state and action? I am currently trying to learn reinforcement learning and I started with the basic gridworld application. I tried Q-learning with the following parameters:

Learning rate = 0.1
Discount factor = 0.95
Exploration rate = 0.1
Default reward = 0
The final reward (for reaching the trophy) = 1

After 500 episodes I got the following results:

How would I compute the optimal state-action value, for example, for state 2, where the agent is standing, and action south?
My intuition was to use the following update rule of the $q$ function:
$$Q[s, a] = Q[s, a] + \alpha (r + \gamma  \max_{a'}Q[s', a'] — Q[s, a])$$
But I am not sure of it. The math doesn't add up for me (when using the update rule).
I am also wondering either I should use the backup diagram to find the optimal state-action q value by propagating the reward (gained from reaching the trophy) to the state in question.
For reference, this is where I learned about the backup diagram.

It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values.
Remember that to simply obtain an optimal Q-value for a given state-action pair we can evaluate
$$Q(s, a) = r + \gamma \max_{a'} Q(s', a)\;;$$
where $s'$ is the state we transitioned into (note that this only holds when obtaining the optimal Q-value, if we were using a stochastic policy then we would have to introduce expectations).
Now, this assumes that we have been given/obtained the optimal Q-values. To obtain them, we have to use the update rule (or any other learning algorithm) that you mentioned in your question.


Why do we need 10 bits to represent the 1000 classes in AlexNet? I'm reading the AlexNet paper. In section 4, where the authors explain how they prevent overfitting, they mention

  Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label".

What does this mean?

You need 10-bits ($2^{10} = 1024$) to represent 1000 classes.


Deep learning for bitmap tracing: extract simple svg paths from bitmaps I need an algorithm to trace simple bitmaps which only contain paths with a given stroke width.

It is obviously very easy to generate bitmaps from vector paths, so creating data for a machine learning algorithm is simple.

Is any existing attempt to create a deep learning model which extracts vector paths from bitmaps?

The model could be trained by giving both the vector and bitmap representation.

Once trained, it would be able to generate the vector paths from the given bitmap.

This seems simple but I could not find any work on this particular task; so I suppose this problem is not fitted for current deep learning architectures, why?

The goal is to trace this kind of image, which would be drawn by hand with a thick felt pen and scanned:

Update

Is there a deep learning architecture fitted for this problem? I believe this question could help me understand what is possible to do with deep learning and what is not, and why. Tracing bitmaps is a perfect example of converting sparse data to a dense abstract representation ; I have the intuition one can learn a lot from this problem.

If we seek proven working source code to plug into a GPLv2-licence compatible solution, we should at least consider autotrace. Its source code is open for review. It can be tested against the example images we have and, if it works fine, called by our GPLv2 software. We can even use the calling code in Inkscape's plug-in image tracing implementation as a good starting point for design and implementation of our calling program, whether it be C, C++, Java, Python, or ECMA (JS).

The trace algorithm in Adobe Illustrator is comparable but is not open source.

If we seek theory, there are several academic publications discussing some of the theory, the last being most aligned with machine learning ideology. I would not dismiss earlier work simply because it doesn't connect with the current machine learning idioms. Investigating what is fully implemented and successfully used by many follows a wise old business proverb: The bird in the hand is worth two in the bush.

Potrace: a polygon-based tracing algorithm, Peter Selinger, 2003
Vector Representation of Binary Images Containing Halftone Dots, Kei Kawamura, Hiroshi Watanabe, Hideyoshi Tominaga, 2004
Testing AutoTrace:
A Machine-learning Approach to Automated
Tongue Contour Data Extraction, Jae-Hyun Sung, Jeff Berry, Marissa Cooper, Gustave Hahn-Powell, and Diana Archangeli, 2013

Many of the online drawing programs collect data.  It would not be surprising if, behind the gracious give-away of online bandwidth, they are establishing a continuously improving data set for training a new breed of autotracers.  None have published AI designs admitting as much, but they would not be legally obligated to do so because a single input example is indeterminable from the autotrace service that could resulting from the training.


Ensure convergence of DDQN if true Q-values are very close I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future reward from either of these actions in the current state is very close). How can I still ensure that the algorithm gets these values (and their relative ranking) right?

EDIT 1

To provide a little more info: What happens currently is that the loss function on the Q-estimator decreases rapidly in the beginning, but then starts evening out. The Q-values also first converge quickly, but then start fluctuating around.

I've tried increasing the batch size which I feel has helped a bit. What did not really help, however, was decreasing the learning rate parameter in the loss function optimizer.

Which other steps might be helpful in this situation?

EDIT 2

If this question is misleading (since reinforcement learning is an approximate dynamic programming method), please just let me know. I mean the algorithm usually does find an only slightly sub-optimal solution to the MDP.

EDIT 3

To answer some questions brought up in the comments:
(1) I do indeed have full control over the MDP including the reward function which in my case is sparse (0 until the terminal episode). The "true" Q-values I know from an analytical solution to the problem I am trying to solve using reinforcement learning.
(2) The rewards are the same for identical transitions. However, the rewards vary for any given state and action taken therein.
(3) The environment is only stochastic for a part of the actions in the action space. I.e. the action chosen by the agent influences the stochasticity of the rewards.

Let $Q^*(s, a)$ denote the "true" $Q$-value for a state-action pair $(s, a)$, i.e. the values that we're hoping to learn to approximate using a neural network that outputs $Q(s, a)$ values.

The problem you describe is basically that you have situations where $Q^*(s, a_1) = Q^*(s, a_2) + \epsilon$ for some very small value $\epsilon$, where $a_1 \neq a_2$.

An important thing to note is that the "true" $Q^*$ values are directly determined by the reward function, which essentially encodes your objective. If your reward function is constructed in such a way that it results in the situation described above, that means the reward function is essentially saying: "it doesn't really matter much whether we pick action $a_1$ or $a_2$, there is some difference in returns but it's a tiny difference, we hardly care." The objective implied by the reward function in some sense contradicts what you describe in the question that you want. You say that you care (quite a lot) about a tiny difference in $Q^*$-values, but normally the "extent to which we care" is proportional to the difference in $Q^*$-values. So, if there is only a tiny difference in $Q^*$-values, we should really only care a tiny bit.

Now, with tabular RL algorithms (like standard $Q$-learning), we do expect to be able to eventually converge to the truly correct solutions, we expect to eventually be capable of correctly ranking different actions even when the differences in $Q^*$-values are small. When you enter function approximation (especially neural networks), this story changes. It's pretty much in the name already; function approximation, we cannot guarantee being able to do better than just approximating. So, 100% ensuring that you'll be able to learn the correct rankings is not going to be feasible. There may be some things that can help though:

If you are "in control" of your environment and its reward function (meaning, you implemented it yourself and/or can modify it if you like), modifying the reward function would probably be the single most effective solution to your problem. As I described above, a reward function that results in tiny differences in $Q^*$-values essentially encodes that you only care about those difference to a tiny (negligible) extent. If that's not the case, if you actually do care about those tiny differences... well, you can try changing the reward function to accentuate those tiny differences. For example, you could try simply multiplying all rewards by $100$, then all differences in returns are also multipled by $100$. Note that this does mean you get larger values everywhere (larger losses, larger gradients, etc.), so it would require changing hyperparameters (like learning rate) accordingly, and could destabilize things. But at least it will emphasize those small differences in $Q^*$-values. (probably won't work, see Neil's comment)
Increasing batch size. You already mentioned this yourself, but I'll also explain why it can help. A major issue in your problem is that you described the environment to be highly stochastic. This means you get a large amount of variance in your observed returns, and therefore also in your updates. If the true differences in $Q^*$-values are very small, these differences will get dominated and become completely invisible due to high variance in observations. You can smooth out this variance by increasing the batch size.
Decreasing learning rate. You also mentioned this in the question, and that it didn't really help. I suppose a high learning rate early on can help to learn more quickly, but decreasing it later on can help to take more "careful", small update steps that don't take "jumps" of a magnitude greater than the true difference in $Q^*$-values.
Distributional Reinforcement Learning: algorithms like $Q$-learning, DQN, Double DQN, etc., they all learn what we can call values, or scalars, or point estimates. Given a state-action pair $(s, a)$, such a $Q$-value represents the expected returns obtained by executing $a$ in $s$ and continuing with a greedy policy afterwards. In stochastic environments, it is possible that the true expected value $Q^*(s, a)$ never coincides with any conrete observation at the end of an actual trajectory. For example, suppose that in some state $s$, an action $a$ has a probability of $0.5$ of giving a reward of $1$ and instantly terminating afterwards, but also has a probability of $0.5$ of giving a reward of $0$ and instantly terminating. Then, we have $Q^*(s, a) = 0.5$, but we never actually observe a return of $0.5$; we always observe returns of either $0$ or $1$. When comparing the output of the learned $Q$-function (which should hopefully be around $0.5$ in such a situation) to any observed trajectories, we'll almost always have a non-zero error and take update steps that keep bouncing around the optimal solution, rather than converging to the optimal solution. Since you mentioned that you see $Q$-values fluctuating around, you may have observed this. Excessive fluctuations around $Q$-value estimates will likely make it difficult to get a consistent, stable ranking when the optimal $Q$-values that you are fluctuating around are themselves very close to each other. There is a class of algorithms (extensions of DQN mostly) referred to as distributional RL, which aim to address this issue by learning a full distribution of returns we expect to be capable of observing, rather than the single-point estimates that $Q$-values essentially are. I believe the first major paper on this topic was the following: https://arxiv.org/abs/1707.06887. There have been multiple follow-up papers too, the most recent of which I believe (but may be wrong here) to be this one: https://arxiv.org/abs/1806.06923. In general, I do know Marc Bellemare and Rémi Munos at least have done quite some work on that, so you could search for those names too.


What makes learned feature detectors specialize in CNN? It has been shown that it is possible to use unsupervised learning techniques to produce good feature detectors in CNNs. I can't understand what drives specialization of those feature detectors. In publications https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf (page 6) they show a very reasonable set of edge and blob detectors with little or no overlap. It goes against intuition - in absence of incentive to specialize, one would expect at least some duplication of learned kernels.

You are thinking along the correct line when you consider intuitivity when examining the strange self-specialization aspect of the layers in any deep network that has been tuned, through toil, mental machinations, and late nights, to work.

Do AI experimenters manipulated parameters, topology, and algorithmic hierarchies to force circuit evolution that produces data flows through specialized functions like this one.

Noisy, normalized, cubes (horizontal, vertical, depth) of integers straight from video hardware or the video channel of a multimedia file
Indications of edges
Indications of corners, ends, and bends
Indications of relative angles (in the case of size independent recognition)
Indications of shapes
Indications of 2D topology
Indications of object forms
Indication of scene
Indication of action (when a hypercube is used and frame is another dimension)

Of course, one could design a solution and then tune layers, one by one, to follow the design, but it is not necessary because of the concept of entropy removal, the elimination of information that is redundant or irrelevant.

The principle in information science is this, in general: As redundancy is removed from the information, the number of bits representing the information decreases and the level of abstraction increases.

In minimalist art, a single word an identical twin might say to the other twin as the train rolls away, or a good Pictionary player, given a small window of information passing opportunity, adaptations can be made to pass large amounts of information base on previously agreed upon conventions.

When you think of it this way, it makes sense that by narrowing the bit width of the aggregate data representation from layer to layer that only certain parametric optimizations will provide an end-to-end indication of convergence, given a goal such as fully categorizing objects based on a small number of features.

When Leibniz envisioned a world tied together by mathematical certainty, he was overly optimistic, however he was correct in at least one important respect, mathematics had a reality all its own.  The success of Newtonian mechanics is one early example of a huge reality that has become apparent and has been so widely applicable, deduced from a thought experiment involving a cannon ball and the moon.

There are few demonstrations of discoveries that originally appear to be wild inventions than the amazingly accurate prediction of the previously anomalous orbit of Mercury.  But Einstein was not surprised by his success.  He had deduced general relativity from the challenge of Ernst Mach and the undeniable results from light and gravity experiments.  He discovered how it must be before it was found.

In the same sense, the above sequence of events can be proven mathematically to be the efficient strategy for vision.  The sequence of processes to transform incoming visual signals to collision avoidance control was there in mathematics before there were visual receptors on microbes.  That is why the narrowing of the visual pathways of a Pterodactyl have similarities to those of a shark shark even though their common ancestor may not have been able to see.

The incentive to specialize is the incentive to converge on the objective even after redundancy is stripped by force by limiting the information channel aperture.  Even if the aperture is in multiple dimensions, the same principle applies.


Clarification on definition of convolution: Is adding the Frobenius inner products between filter and input part of convolution or a separate step? From the literature I have read so far, it is not clear how exactly the convolution operation is defined. It seems people use two different definitions:

Let us assume we are given an $n_w \times n_h \times d$ input tensor $I$ and an $m_w \times m_h \times d$ filter $F$ of $d$ kernels (I use the convention of referring to the depth-slices of filters as kernels. I also will call the depth slices of the input tensor channels). Let us also assume $F$ is the $j$th filter of $J$ filters.

Now to the definitions.

Option 1:

The convolution of $I$ with $F$ is obtained by sliding $F$ across $I$ and computing the Frobenius inner product between channel $k$ and kernerl $k$ at each position, adding the products and storing them in an output matrix. That matrix is the result of the convolution. It is also the $j$th feature map in the output tensor of the convolution layer.

Let $I \in \mathbb{R}^{n_w \times n_h \times d}$ and $F \in \mathbb{R}^{m_w \times m_h \times d}$. Let $s \in \mathbb{N}$ be the stride. The operation will only be defined if the smaller tensor fits within the larger tensor along its width and height a positive integer number of times when shifting by $s$, that is if and only if $k_w = (n_w - m_w) / s + 1\in \mathbb{N}$ and $k_h = (n_h - m_h) / s + 1\in \mathbb{N}$, where $k_w \times k_h \times d$ is the shape of the output tensor. Furthermore let ${f_x : i \mapsto (x - 1)s + i}$ be a function that returns the absolute index in the input tensor, given an index $x$ in the output tensor, the stride length $s$ and a relative index $i$.
\begin{equation*}
    \begin{split}
        (I * F)_{x y} =
            & \sum_{k=1}^d \sum_{i = 1}^{m_w} \sum_{j = 1}^{m_h} I_{f_x(i) f_y(j) k} \cdot F_{i j k}
    \end{split}
\end{equation*}

Option 2:

The convolutions (plural) of $I$ with $F$ are obtained by sliding $F$ across $I$ and computing the Frobenius inner product between channel $k$ and kernel $i$ at each position. Each product is stored in a matrix associated with the channel $k$. There is no adding of the products yet. The convolutions are the result matrices. The step where the matrices are added component wise to obtain the $j$th feature map of the output tensor of the convolution layer is not part of the convolution operation, but an independent step.

Let $I \in \mathbb{R}^{n_w \times n_h}$ and $F \in \mathbb{R}^{m_w \times m_h}$. Let $s \in \mathbb{N}$ be the stride. The operation will only be defined if the smaller matrix fits within the larger one along its width and height a positive integer number of times when shifting by $s$, that is, if and only if $k_w = (n_w - m_w) / s + 1\in \mathbb{N}$ and $k_h = (n_h - m_h) / s + 1\in \mathbb{N}$, where $k_w \times k_h$ is the shape of the output matrix. Furthermore let ${f_x : i \mapsto (x - 1)s + i}$ be a function that returns the absolute index in the input matrix, given an index $x$ in the output matrix, the stride length $s$ and a relative index $i$.
\begin{equation*}
    \begin{split}
        (I * F)_{x y} =
            &\sum_{i = 1}^{m_w} \sum_{j = 1}^{m_h} I_{f_x(i) f_y(j)} \cdot F_{i j}
    \end{split}
\end{equation*}

Which of these two definitions is the common one?

Both are incorrect.

using your notation
You do not take a sliding frobenius inner product of a singular channel of $I$ with $F$, but with all the channels at once. This may be easier to understand if you do not assume the number of channels of the input and output are the same (ie different number of input channels then filters). So lets say your input has $k_1$ channels and you have $k_2$ filters.

This means $shape(I) = (N, M, k_1)$ and each of the $k_2$ filters is of shape $(n, m,k_2)$ and your output shape is $(N-n+1, M-m+1, k_2)$ assuming your not using padding.

So i guess trying to put it in the manner you use: The convolution's output's $i^{th}$ channel is taking the sliding Frobenius inner product between all $n \times m$ cross-section of $I$ (including all input channels) with filter $F_i$.


A neural network with 2 or more hidden layers is a DNN? I just learnt the math behind neural networks so please bear with my ignorance. I wonder if there is a precise definition for DNN.
Is it true that any neural network with more than 2 hidden layer can be named as a DNN, and training a NN with 2 hidden layers using Q-learning we are technically doing a type of deep reinforcement learning?
PS: If it is conceptually that simple why do common people regard deep learning like something done by archmages in ivory towers.

I don't think there is a fixed threshold that differentiates between Shallow and Deep Learning, but I would say that a 2 layer NN should not be considered deep. But now-a-days, almost all NN architectures are studied under the umbrella of Deep Learning.
And yes, training a 2 hidden layers NN using Q-learning would technically mean doing deep RL.
I guess it is conceptually simple but making NN perform optimally is an art. Tuning hyperparameters or debugging NN can be tough and one learns with experience. I guess others in the community would be much more suited to answer this question. But these were my 2 cents.


Which unsupervised anomaly detection algorithms are there? I need to create model which will find suspicious entries or anomalies in a network, whose characteristics or features are the asset_id, user_id, IP accessed from and time_stamp.

Which unsupervised anomaly detection algorithms or models should I use to solve this task?

If you are OK to use python, thy novelty-detection with sklearn:

https://scikit-learn.org/stable/modules/outlier_detection.html


Is it appropriate to use a softmax activation with a categorical crossentropy loss? I have a binary classification problem where I have 2 classes. A sample is either class 1 or class 2 - For simplicity, lets say they are exclusive from one another so it is definitely one or the other.
For this reason, in my neural network, I have specified a softmax activation in the last layer with 2 outputs and a categorical crossentropy for the loss. Using tensorflow:
model=tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(units=64, input_shape=(100,), activation='relu'))
model.add(tf.keras.layers.Dropout(0.4))
model.add(tf.keras.layers.Dense(units=32, activation='relu'))
model.add(tf.keras.layers.Dropout(0.4))
model.add(tf.keras.layers.Dense(units=2, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Here are my questions.

If the sigmoid is equivalent to the softmax, firstly is it valid to specify 2 units with a softmax and categorical_crossentropy?

Is it the same as using binary_crossentropy (in this particular use case) with 2 classes and a sigmoid activation, and if so why?

I know that for non-exclusive multi-label problems with more than 2 classes, a binary_crossentropy with a sigmoid activation is used, why is the non-exclusivity about the multi-label case uniquely different from a binary classification with 2 classes only, with 1 (class 0 or class 1) output and a sigmoid with binary_crossentropy loss.

Let's first recap the definition of the binary cross-entropy (BCE) and the categorical cross-entropy (CCE).
Here's the BCE (equation 4.90 from this book)
$$-\sum_{n=1}^{N}\left( t_{n} \ln y_{n}+\left(1-t_{n}\right) \ln \left(1-y_{n}\right)\right) \label{1}\tag{1},$$
where

$t_{n} \in\{0,1\}$ is the target
$y_n \in [0, 1]$ is the prediction (as produced by the sigmoid), so $1 - y_n$ is the probability that $n$ belongs to the other class

Here's the CCE (equation 4.108)
$$
-\sum_{n=1}^{N} \sum_{k=1}^{K} t_{n k} \ln y_{n k}\label{2}\tag{2},
$$
where

$t_{n k} = \{0, 1\}$ is the target of input $n$ for class $k$, i.e. it's $1$ when $n$ is labelled as $k$ and $0$ otherwise (so it's $0$ for all $K$ except for one of them)
$y_{n k}$ is the probability that $n$ belongs to the class $k$, as produced by the softmax function

Let $K=2$. Then equation \ref{2} becomes
$$
-\sum_{n=1}^{N} \sum_{k=1}^{2} t_{n k} \ln y_{n k} =
-\sum_{n=1}^{N} \left( t_{n 1} \ln y_{n 1} + t_{n 2} \ln y_{n 2} \right)
\label{3}\tag{3}
$$
So, if $[y_{n 1},  y_{n 2}]$ is a probability vector (which is the case if you use the softmax as the activation function of the last layer), then, in theory, the BCE and CCE are equivalent in the case of binary classification. In practice, if you are using TensorFlow, to choose the most suitable loss function for your problem, you could take a look at this answer.


Is superintelligence a function of strength or a category? Super comes from the Latin and means "above".

University of Oxford philosopher Nick Bostrom defines superintelligence as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest". (wiki)

Bostrom's definition could be taken to imply this is a quantitative measure of degrees as a numeric relationship. (Under this definition, we have achieved narrow superintelligence, reduced to competency in a single task.)
Gibson, famously, sheds light on an another aspect via Wintermute & Neuromancer, where, once superintelligence is achieved, the AI just f-'s off and does it's own thing, motivations beyond human comprehensions.  (Essentially, "next-level" thinking.) The second measure is discrete and ordinal.
Is superintelligence a function of strength or a category?

I’ll have a stab at this.

Cognitive performance in narrow domains is determined by competency, efficiency and speed. Take calculating numbers, extremely narrow domain but compared to humans the ability of a calculator to calculate numbers exceeds normal human performance, it is much competent in terms of speed. In a bit broader domain, AlphaGo has defeated Go players, which is more difficult than chess, and requires intuition. In fact, there is an instance where the AlphaGo makes a long-term move that was previously unimagined. In all domains however Humans are well rounded, therefore Human Intelligence is called general intelligence. An AlphaGo or Calculator cannot speak eloquently or make music, but AIs are gaining pace in these areas too.

I agree with @nbro that Bostrom wants to keep the interpretation of Superintelligence open. But if there is a rough category, these are-

ANI- Artificial Narrow Intelligence
AGI- Artificial General Intelligence: Where the AI’s performance is at par with humans. After AGI, it quickly takes off to SI.
SI- Superintelligence: Superintelligence is beyond our imagination, we have not figured out yet what will a SI do, think or want.

While these categories are discrete, the functions of strength are not. I’d say they are rather discrete to continuous, because if you look at the computing power plots that follow Moore’s Law, a similar exponential graph can be drawn for AI’s performance towards general intelligence. In that graph, it seems the AI’s performance starts with discrete performance points, and then as it takes off, it becomes continuous.

This is why the term Singularity is often associated with Superintelligence. I hope this answers your question.


What Constitutes Messages in Junction Tree Algorithm? I'm currently studying the Junction Tree Algorithm: I'm referring to the process of transforming a Bayesian Network into a Junction Tree in order to apply inference. I understand how you build the Junction Tree, but I'm stuck on the idea of message passing.
What exactly are these messages? Are they numbers, or vectors?
If any of you could direct me to a numerical example that would be very appreciated.

I think an example could make you understand better.
Suppose you want to calculate $P(X|S)$, you now need to put evidence on $S$(so you have to change all tables where $S$ appears making probability zero where $S$ does not appear).
At this point, you can proceed with the collect and distribute method in order to propagate the evidence throughout the graph, in order to maintain the global consistency property. Now you can choose any $cluster\space V$ that contains $X$, marginalize with respect to $S$, and the job is done.


What else can boost iterative deepening with alpha-beta pruning? I read about minimax, then alpha-beta pruning, and then about iterative deepening. Iterative deepening coupled with alpha-beta pruning proves to quite efficient as compared to alpha-beta alone.
I have implemented a game agent that uses iterative deepening with alpha-beta pruning. Now, I want to beat myself. What can I do to go deeper? Like alpha-beta pruning cut the moves, what other small change could be implemented that can beat my older AI?
My aim to go deeper than my current AI. If you want to know about the game, here is a brief summary:
There are 2 players, 4 game pieces, and a 7-by-7 grid of squares. At the beginning of the game, the first player places both the pieces on any two different squares. From that point on, the players alternate turns moving both the pieces like a queen in chess (any number of open squares vertically, horizontally, or diagonally). When the piece is moved, the square that was previously occupied is blocked. That square can not be used for the remainder of the game. The piece can not move through blocked squares. The first player who is unable to move any one of the queens loses.
So my aim is to cut the unwanted nodes and search deeper.

First thing you're going to want to add is probably a Transposition Table, as also suggested by SmallChess.

Afterwards, I'd look into Aspiration Search and/or Principal Variation Search (also see this page).

Then I'd look into things like the Killer Move Heuristic, and maybe also see if you can simply implement existing parts of your engine more efficiently (e.g. use bitboards for your state representation).

Other than all of that, the chess programming wiki probably has lots of other interesting pages as well.


How can fuzzy logic be used in creating AI? Fuzzy logic is the logic where every statement can have any real truth value between 0 and 1.

How can fuzzy logic be used in creating AI? Is it useful for certain decision problems involving multiple inputs? Can you give an example of an AI that uses it?

A classical example of fuzzy logic in an AI is the expert system Mycin.

Fuzzy logic can be used to deal with probabilities and uncertainties.

If one looks at, for example, predicate logic, then every statement is either true or false. In reality, we don't have this mathematical certainty.

For example, let's say a physician (or expert system) sees a symptom that can be attributed to a few different diseases (say A, B and C). The physician will now attribute a higher likelihood to the possibility of the patient having any of these three diseases. There is no definite true or false statement, but there is a change of weights. This can be reflected in fuzzy logic, but not so easily in symbolic logic.


Is it expected that during self-play reinforcement learning that player 1 or player 2 wins the majority of games? I'm testing various learning rates and neural network configurations. I'm testing over 10000 games, with the first 2000 having random starting moves and general randomness throughout of about 20%, i.e. 20% of moves are random.

In all configurations, I initiate the weights to random values.

What I've found is that in all configurations, Player 1 will win the majority of games, or Player 2 will. There's no 50/50 split.

Is this expected or normal? I am training on the Connect Four game.

From wikipedia article on  Connect Four:

  Connect Four is a solved game. The first player can always win by
  playing the right moves.

It's pretty reasonable that even without training 1st player win more often by small margin.
With some training  1st player should mostly win.


What would happen to an agent trained using Markov Decision Process if the goal node changes? I was reading up a paper that did routing based on an MDP, and I was wondering because, in routing, there is a sender node and a receiver node, so if the receiver node changes (sending a message to someone else), would we have to train the MDP algorithm all over again?
This also got me thinking about what would happen even if one node in the process of transmission changes. Does using an MDP for training the agent mean that the obstacle and goals should never change?

It is possible, at design time for a reinforcement learning problem, to allow for changes within an environment. You can make any element into a variable property of the state, that the agent can realistically be told at the start or sense from the environment.
If you do add new variable to model the possibility of change:

It allows the agent to learn to solve a more general problem where the chosen property can vary.

It increases the size of the state space.

It requires training to include variations of the new variable.

Usually this also increases the time taken to train.
It is not always possible to use a state variable for the task - perhaps a goal state is effectively hidden from the agent and the purpose of training is for it to be discovered. In which case, you will require at least some re-training. It may be faster to start with the existing trained agent if the difference is not large.
If you cannot simply extend the state representation, and the environment changes in a small enough way, then it may also be possible to use an agent which continuously explores which will re-train itself over time in repsonse to changes in the environment. The DynaQ+ algorithm is an example of a method which is designed to explore and find changes in the agent's environment to allow for this kind of online retraining when things change.


How can I train a neural network to give probability of a random event? Let's say I have an adjustable loaded die, and I want to train a neural network to give me the probability of each face, depending on the settings of the loaded die.

I can't mesure its performance on individual die roll, since it does not give me a probability.

I could batch a lot of roll to calculate a probability and use this batch as an individual test case, but my problem does not allow this (let's say the settings are complex and randomized between each roll).

I have 2 ideas:

train it as a classification problem which output confidence, and hope that the confidence will reflect the actual probability. Sometimes the network would output the correct probability and fail the test, but on average it would tend to the correct probability. However it may require a lot of training and data.
batch random rolls together and compare the mean/median/standard deviation of the measured result vs the predictions. It could work but I don't know the good batch size.

Thank you.

The starting point is that for a fair dice thrown fairly the p(n) is 1/n where n is the number of sides.
You said both
and

there are too many variables (up 40 dimensions with value range 1-100) in input, I don't know how these properties relate and an empirical approach would require too much data.

It seems that this problem has 2 solutions:

Don't use a neural net and create a 'std' statistical model.
It may be possible since you said:

I know there is some underlying rule that simplify a lot the problem (ie. reduce the actual number of dimension of the input)

Use a neural network (with softmax at the end) - for a fair dice; with enough training data the classifier should arrive as 1/n as the approximating function for a fair dice. The other 40 dimensions/settings your mentioned are the inputs. I think a 'basic' neural network with dense layers only could work for your task.


How to create a custom environment for reinforcement learning I am a newbie in reinforcement learning working on a college project. The project is related to optimizing the hardware power. I am running proprietary software in Linux distribution (16.04). The goal is to use reinforcement learning and optimize the power of the System (keeping the performance degradation of the software as minimum as possible).

For this, I need to create a custom environment for my reinforcement learning. From reading different materials, I could understand that I need to make my software as a custom environment from where I can retrieve the state features. Action space may include instructions to Linux to change the power (I can use some predefined set of power options).

The proprietary software is a cellular network and the state variables include latency or throughput. To control the power action space, rapl-tools can be used to control CPU power.

I just started working on this project and everything seems blurry. What is the best way to make this work? Is there some tutorials or materials that would help me make things clear. Is my understanding of creating a custom environment for reinforcement learning true?

This answer assumes that your "proprietary software" is a simulation of, or controller for a real environment.

Yes you will very likely need to write software to represent your environment in some standard way as a Reinforcement Learning (RL) environment. Depending on details, this may be trivially easy or it might be quite involved.

An environment in RL must have the following traits in general, in order to interface with RL agent software:

A state representation. This will typically be an object or array of data that matches sensor readings from the real environment. It is important to RL that the state has the Markov property so that predictions of value can be accurate. For some environments that will mean calculating derived values from observations, or representing a combined history of last few observations from sensors as the state.

The state can either be held inside an internal representation of the environment, which is a typical object-oriented approach, or it can be passed around as a parameter to other functions.
A simple state might just be a fixed size array of numbers representing important traits of the environment, scaled between -1 and 1 for convenience when using it with neural networks.

An action representation.

A simple action representation could just be an integer which identifies which of N actions has been chosen, starting from 0. This allows for a basic index lookup when checking value function estimates.

A reward function. This is part of a problem definition, and you may want to have that code as part of the environment or part of the agent or somewhere in-between depending on how likely it is to change - e.g. if you want to run multiple experiments that optimise different aspects of control but in the same environment, you may make a totally separate reward calculation module that you combine at a high level with the agent and environment code.

Reward function design is a complex subject. However, the basics are that this should always be a single real valued number (i.e. a floating point number in most programming languages).

A time step function. This should take an action choice, and should update the state for a time step - returning the next state, and the immediate reward. If the environment is real, then the code will make actual changes (e.g. move robot arm), potentially wait for the time step to elapse, then read sensors to get the next state and calculate reward. If the environment is simulated, then the code should call some internal model to calculate the next state. This function should call the proprietary software you have been provided for your task.

If actions available depend on the current state, then code for that could live in the environment simulation or the agent, or be some helper function that the agent can call, so it can filter the actions before choosing one.

If you are working in Python, to help make this more concrete, and follow an existing design, see "How to create a new gym environment in OpenAI?" on Stack Overflow. The Open AI environments all follow the same conventions for environment definitions, which helps when writing agents to solve them. I also recommend finding an Open AI Gym environment that seems similar to your problem, seeing how that works and trying to train an agent to solve it.

There may still be work to match your environment to an agent that can solve it. That depends on what agent software you are using.

Even if you write your own agent software, it helps to separate out the environment code like this. The environment is the problem to solve, and the RL agent is one way to search for a solution to it. Keeping those parts separate is a useful design that will allow you to try different types of agent and compare their performance for example.


How robust are deep networks to class imbalance? Before deep learning, I worked with machine learning problems where the data had a large class imbalance (30:1 or worse ratios). At that time, all the classifiers struggled, even after under-sampling the represented classes and creating synthetic examples of the underrepresented classes -- except Random Forest, which was a bit more robust than the others, but still not great.
What are guidelines for class distribution when it comes to deep learning (CNNs, ResNets, transformers, etc)? Must the representation of each class be 1:1? Or maybe it's "good enough" as long as it is under some ratio like 2:1? Or is deep learning completely immune to class imbalance as long as we have enough training data?
Furthermore, as a general guideline, should each class have a certain minimum number of training examples (maybe some multiple of the number of weights of the network)?

@nbro pointed out the paper A systematic study of the class imbalance problem in convolutional neural networks, which tested class imbalance LeNet for MNIST, on a custom CNN for CIFAR-10, and on ResNet for ImageNet. The paper found that by artificially creating class imbalance on those data sets, the neural networks are significantly deteriorated. The ROC AUC drops by 5-10%, and accuracy decreases by 20-30%. These effects are worsened on more complex tasks.
There are 3 noteworthy class imbalance approaches to partially alleviate this:

Undersampling: sampling the over-represented class less often
Oversampling: sampling the under-represented class more often
Thresholding: after the neural network learns the weights, during inference, multiply the output class probability by a weight (the class prior, which is different for each class). The weight is the inverse of the class representation of the dataset (i.e. the inverse of $\frac{numInstances(c)}{\displaystyle \sum_i numInstances(i)}$, where $c$ is the current class and $numInstances(i)$ is the number of unique instances of class $i$ in the training set.

The paper concludes with the following best practices:

Regarding the choice of a method to handle CNN training on imbalanced dataset we conclude the following.

The method that in most of the cases outperforms all others with respect to multi-class ROC AUC was oversampling.

For extreme ratio of imbalance and large portion of classes being minority, undersampling performs on a par with oversampling. If training time is an issue, undersampling is a better choice in such a scenario since it dramatically reduces the size of the training set

To achieve the best accuracy, one should apply thresholding to compensate for prior class probabilities. A combination of thresholding with baseline and oversampling is the most preferable, whereas it should not be combined with undersampling.

Oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance. The higher a fraction of minority classes in the imbalanced training set, the more imbalance ratio should be reduced.

Oversampling does not cause overfitting of convolutional neural networks, as opposed to some classical machine learning models.

The last point is very interesting, because oversampling is known to cause overfitting in classical machine learning models and many have advised against doing it.


How could logistic loss be used as loss function for an ANN? Normally, in practice, people use those loss functions with minima, e.g. $L_1$ mean absolute loss, $L_2$ mean squared error, etc. All those come with a minimum to optimize to.

However, there's another thing, logistic loss, I'm reading about, but don't get it why the logistic function could be used as a loss function, given that it has the so-called minimum at infinity, but that isn't a normal minimum. Logistic loss function (black curve):

How can an optimizer minimize the logistic loss?

I see why you might be confused. First, the logistic-loss or log-loss is technically called cross-entropy loss. This function is very simple:
$CE = -[y \log(p) + (1 - y) \log(1 - p)]$
This tells basically if the predicted class $y$ was right $y=1$ then the loss is $CE=-\log(p)$, if the predicted class was not the right one then the loss is $CE=-\log(1-p)$.
If we look at the function as a pure math concept we see that:
$CE = f(x) = - \log(x)$
And as you point out, that function is minima-unbounded as its domain is $D(f(x)) = [0, +\infty]$. You can check that in here:

However the trick is that the inputs must be bounded, meaning, the inputs to the loss function must be in range $[0, 1]$. This bounding is achieved by applying a sigmoid activation function as the final "layer" of the network. Then if the inputs to the loss function are bounded the function has a clear minima.
Check how the function looks in reality from one of the most important papers in loss function in the AI world: Focal Loss (I really encourage you to read it as the first section explains in detail the cross-entropy loss). The blue curve is the one you are looking for.

Finally, you might want to review your log-loss/CE function since it should have an asymptote for $f(x=0) = \infty$


IQN bellman target: using Z vs using Q IQN paper (https://arxiv.org/abs/1806.06923) uses distributional bellman target:
$$ \delta^{\tau,\tau'}_t = r_t + \gamma Z_{\tau'}(x_{t+1}, \pi_{\beta}(x_{t+1})) - Z_{\tau}(x_t, a_t) $$
And optimizes:
$$ L = \frac{1}{N'} \sum^{N}_i \sum^{N'}_j \rho^\kappa_{\tau_i} \delta^{\tau_i,\tau_j}_t $$

But similar quantiles can be got just from Q values, when doing so:
$$ \delta^\tau_t = r_t + \gamma \frac{1}{N'} \sum_{j}^{N'} Z_{\tau_j}(x_{t+1}, \pi_{\beta}(x_{t+1})) - Z_\tau(x_t, a_t) \\ = r_t + \gamma Q (x_{t+1}, \pi_\beta(x_{t+1})) - Z_\tau(x_t, a_t) $$
optimizing:
$$ L = \sum^N_i \rho^{\kappa}_{\tau_i} \delta^{\tau_i}_t $$

Both lead to similar performance on CartPole env. The loss function of the 2nd one is more simpler and intuitive (atleast to me). So i was thinking if there are any obvious reason why authors didin't use it?

Replacement you suggest is replacement of random variable by its expectation in forward part of TD. It would make IQN into modification of C51 with randomly sampled function approximator instead of discrete distribution. Both distribution produced and especially exploration behavior with your replacement would be very different. The authors of paper explicitly said that "more randomness" in their opinion benefit training, so reducing randomness would go aginst spirit of the paiper.  That they produce similar results on single toy test mean very little. IQN could be better then C51 or it could be worse then C51 but single toy example is not enough to say they are close. Nevertheless I agree that IQN looks overly complex and may require more training time, C51 approach could be more practical.


How to implement a contextual reinforcement learning model? In a reinforcement learning model, states depend on the previous actions chosen. In the case in which some of the states -but not all- are fully independent of the actions -but still obviously determine the optimal actions-, how could we take these state variables into account?

If the problem was a multiarmed bandit problem (where none of the actions influence the states), the solution would be a contextual multiarmed bandit problem. Though, if we need a "contextual reinforcement learning problem", how can we approach it?

I can think of separating a continuous context into steps, and creating a reinforcement learning model for each of these steps. Then, is there any solution where these multiple RL models are used together, where each model is used for prediction and feedback proportionally to the closeness between the actual context and the context assigned to the RL model? Is this even a good approach?

In the case in which some of the states -but not all- are fully independent of the actions -but still obviously determine the optimal actions-, how could we take these state variables into account?

I think the key thing here is the caveat but not all. What you have is a fully-featured MDP (states, actions, rewards, timesteps where next reward and next state depend on current state and action). The fact that next state is only marginally affected by the current action does not prevent it being an MDP.

It would be a problem if the current state did not adequately describe this limitation e.g. if some other data, outside of observable state, decided whether action had any effect. Assuming that is not the case, then you still have a full reinforcement learning problem, but one with some unusual qualities.

You can mitigate against problems caused by algorithms that will over-estimate likely reward (caused by algorithm associating lucky state trajectories with action choice), by using "double learning" - one estimate of return is used to select maximising action, and another used to estimate the actual return from the next state. You probably will also prefer single-step learning over learning based on trajectories as most of the time your state trajectories will not contain learnable data. So Double Deep Q Networks might be a good starting algorithm to try.

If you know absolutely from inspecting a current state, that the next state is independent of the current action, and the next state could in theory be almost anything (from either the whole state space or some large subset) then you may be able to adapt the algorithm to allow for that knowledge. You would do this by altering the TD target and replacing the terms for bootstrap estimate of next state with a rolling mean over all reachable next states. In concept this is similar to Expected SARSA - and in practice if it is possible, it would go a long way to reducing variance during learning process, and may speed up learning significantly. If you know the distribution of next states, you could maybe use that, but basing it purely on samples seen should also be fine, provided you can allocate the means to correct group of states (your question implies you have some understanding of how the states will group). Note that transitions from states with no action effect on next state to states which do have an effect from the action will need careful handling - they should not be assigned to the same "mean group", but should instead affect the TD target normally when they occur.

If it is not possible for the agent to know whether it is in a state where its action affects the next state, except through experience, then you really have to use a standard RL solver.

Finally, if you have a situation where the current action does not affect next state, but the current state definitely does in all cases -  the state "evolves", maybe stochastically, independently of the action taken in most cases, and can only change to a relatively small subset of next states from any given current state - then this is best solved with a normal RL solver - again double Q Learning might be a reasonable starting algorithm in that case.


A* is similar to Dijkstra with reduced cost According to this Wikipedia article

If the heuristic $h$ satisfies the additional condition $h(x) \leq d(x, y) + h(y)$ for every edge $(x, y)$ of the graph (where $d$ denotes the length of that edge), then $h$ is called monotone, or consistent. In such a case, $A^*$ can be implemented more efficiently — roughly speaking, no node needs to be processed more than once (see closed set below) — and $A^*$ is equivalent to running Dijkstra's algorithm with the reduced cost $d'(x, y) = d(x, y) + h(y) − h(x).$

Can someone intuitively explain why the reduced cost is of this form ?

What you are doing when calculating $d'(x,y)$:

$d(x,y)$: calculating the original edge distance from $x$ to $y$
$h(y)$: plus the heuristic from $y$ to the goal
$h(x)$: minus the heuristic from $x$ to the goal

So, using this recalculation of the original edge-values ($1.$) in Dijkstra's algorithm you are inherently accounting for the heuristic component of A* by incorporating it ($2.$) into the value of the edge traversed, and discarding ($3.$) the accumulated previous heuristic values of previous nodes in the path.

The additional condition $h(x) ≤ d(x, y) + h(y)$ ensures the new edge-values are strictly positive.


Can training a model on a dataset composed by real images and drawings hurt the training process of a real-world application model? I'm training a multi-label classifier that's supposed to be tested on underwater images. I'm wondering if feeding the model drawings of a certain class plus real images can affect the results badly. Was there a study on this? Or are there any past experiences anyone could share to help?

To my knowledge the deployment model (that you will test on underwater images) as inference will not have a negative effect. Yet drawings may even help differentiate some classes at training and inference. Provided that you won't use drawings in inference, adding them in training phase will not necessarily hurt the accuracy. Note that a drawing of a particular class should not be in the search domains of other classes, namely, a drawing of a particular class should not be the same with the other classes.


What is the difference between memetic algorithms and genetic algorithms? What is the difference between memetic algorithms and genetic algorithms? Is an individual's lifetime a learning part of memetic algorithms?

A genetic algorithm is an algorithm, based on natural selection (the process that drives biological evolution), for solving both constrained and unconstrained optimization problems.
A memetic algorithm is an extension of the concept of a genetic algorithm that uses a local search technique to reduce the likelihood of premature convergence.
The paper A Comparison between Memetic algorithm and Genetic algorithm for the cryptanalysis of Simplified Data Encryption Standard algorithm compares both approaches.
To answer your last question, yes, an individual's lifetime still plays a part in memetic algorithms because the objective here is to avoid premature convergence.


What is meant by "reproduction" in the description of this exercise? In the following exercise, the word reproduction is mentioned.

Your task is to design a simple genetic algorithm, with binary-coded chromosomes, in order to solve a pattern-finding problem in 16-bit strings.
...
You may use the following operators:

Reproduction: You can use either one of the following reproduction types: Proportional, Ranking, Tournament.

Crossing over: In order to perform this operation the individuals must be grouped in pairs (randomly), and, with certain probability p, cross information from their chromosomes must be exchanged.

Mutation: This operator changes the value of each bit in the chromosome to the opposite one with a very small probability

So, apparently, crossover and reproduction do not refer to the same concept. What does the word reproduction refer to in a genetic algorithm and in this exercise?

The terminology of this exercise is not standard. What is referred to as "reproduction" in the exercise is usually referred to as "selection".

The term "reproduction" does indeed seem conceptually closer to the notion of crossover or recombination (these two are the same thing), which is probably where your confusion has arisen.

See the excellent (and freely-downloadable) 'Essentials of Metaheuristics' for an introduction to the usual terminology for evolutionary algorithms.


How is it that AI can become biased, and what are the proposals to mitigate this? This is not meant to be negative or a joke but rather looking for a productive solution on AI development, engineering and its impact on human life:

Lately with my Google searches, the AI model keeps auto filling the ending of my searches with:

“...in Vietnamese”

And

“...in a Vietnamese home”

The issue is I have never searched for that but because of my last name the model is creating this context.

The other issue is that I’m a halfy and my dad is actually third generation, I grew up mainstream American and don’t even speak Vietnamese. I’m not even sure what a Vietnamese home means.

My buddy in a similar situation of South Asian and noticed the same exact thing more so with YouTube recommended videos.

We already have enough issues in the US with racism, projections of who others expect us to be based on any number of things, stereotyping and putting people in boxes to limit them - I truly believe AI is adding to the problem, not helping.

How can we fix this. Moreover, how can we use AI to bring out peoples true self, talents and empower and free them them to create their life how they like ?

There is huge potential here to harness AI in ways that can bring us more freedom, joy and beauty so people can be the whole of themselves and with who they really are. Then meet peoples needs, wishes, dreams and hope. Given them shoulders to stand on to create their reality, not live someone else's projection of themselves.

Lately with my Google searches, the AI model keeps auto filling the ending of my searches with:
“...in Vietnamese”

I can see how this would be annoying.
I don't think Google's auto-complete algorithm and training data is publicly available. Also it changes frequently as they work to improve the service. As such, it is hard to tell what exactly is leading it to come up with this less-than-useful suggestion.
Your suspicion that it has something to do with Google's service detecting your heritage seems plausible.
The whole thing is based around statistical inference. At no point does any machine "know" what Vietnamese - or in fact any of the words in your query  - actually means. This is a weakness of pretty much all core NLP work in AI, and is called the grounding problem. It is why, for instance, that samples of computer generated text produce such surreal and comic material. The rules of grammar are followed, but semantics and longer term coherence are a mess.
Commercial chatbot systems work around this with a lot of bespoke coding around some subject area, such as booking tickets, shopping etc. These smaller domains are possible for human developers to "police", connecting them back to reality, and avoiding the open-ended nature of the whole of human language. Search engine text autocomplete however, cannot realistically use this approach.
Your best bets are probably:

Wait it out. The service will improve. Whatever language use statistics are at work here are likely change over time. Your own normal use of the system without using the suggestions will be part of that data stream of corrections.

Send a complaint to Google. Someone, somewhere in Google will care about these results, and view them as errors to be fixed.

Neither of these approaches guarantee results in any time frame sadly.

We already have enough issues in the US with racism, projections of who others expect us to be based on any number of things, stereotyping and putting people in boxes to limit them - I truly believe AI is adding to the problem, not helping.

You are not alone in having these worries. The statistics-driven nature of machine learning algorithms and use of "big data" to train them means that machines are exposing bias and prejudice that are long buried in our language. These biases are picked up by machinery then used by companies that don't necessarily want to reflect those attitudes.
A similar example occurs in natural language processing models with word embeddings. A very interesting feature of LSTM neural networks that learn statistical language models is that you can look at word embeddings, mathematical representations of words, and do "word math":
$$W(king) - W(man) + W(woman) \approx W(queen)$$
$$W(he) - W(male) + W(female) \approx W(she)$$
This is very cool, and implies that the learned embeddings really are capturing semantics up to some depth. However, the same model can produce results like this:
$$W(doctor) - W(male) + W(female) \approx W(nurse)$$
This doesn't reflect modern sensibilities of gender equality. There is obviously a deep set reason for this, as it has appeared from non-prejudiced statistical analysis of billions of words of text from all sorts of sources. Regardless of this though, engineers responsible for these systems would prefer that their models did not have these flaws.

How can we fix this. Moreover, how can we use AI to bring out peoples true self, talents and empower and free them them to create their life how they like ?

Primarily by recognising that statistical ML and AI doesn't inherently have prejudice or any agenda at all. It is reflecting back ugliness already in the world. The root problem is to fix people (beyond scope of this answer, if I had solid ideas about this I would not be working in software engineering, but in something more people-focussed).
However, we can remove some of the unwanted bias from AI systems. Broadly the steps toward this go:

Recognise that a particular AI system has captured and is using unwanted gender, racial, religious etc bias.

Reach a consensus about how an unbiased model should behave. It must still be useful for purpose.

Add the desired model behaviour into the training and assessment routines of the AI.

For instance in your case, there are possibly some users of Google's system who would prefer to read articles in Vietnamese, or have English translated into Vietnamese, and are finding it awkward that the default assumption is that everything should be presented in English. These users don't necessarily need to use the search text for this, but presumably are for some reason. A reasonable approach is to figure out how their needs could be met without spamming "in Vietnamese" on the end of every autocomplete suggestion, and perhaps in general move suggestions to localise searches by cultural differences out of autocomplete into a different part of the system.
For the case of gender bias in NLP systems, Andrew Ng's Coursera course on RNNs shows how this can be achieved using the embeddings themselves. Essentially it can be done by identifying a bias direction from a set of words (e.g. "he/she", "male/female"), and removing deviations in that direction for most other words, preserving it only for words where it is inherently OK to reflect the differences (such as "king" and "queen" for gender bias).
Each case of unwanted bias though needs to be discovered by people and oversight of this as a political and social issue, not primarily a technical one.


What is the time complexity of the forward pass algorithm of a feedforward neural network? How do I determine the time complexity of the forward pass algorithm of a feedforward neural network? How many multiplications are done to generate the output?

Let's suppose that we have an MLP with $15$ inputs, $20$ hidden neurons and $2$ output neurons. The operations performed are only in the hidden and output neurons, given that the input neurons only represent the inputs (so they do not perform any operation).

Each hidden neuron performs a linear combination of its inputs followed by the application of a non-linear (or activation) function. So, each hidden neuron $j$ performs the following operation

\begin{align}
o_j = \sigma \left(\sum_{i}^{15} w_{ij}x_i \right),\tag{1}\label{1}
\end{align}

where $i$ is the input coming from the input neuron $i$, $w_{ij}$ is the weight of the connection from the input neuron $i$ to the hidden neuron $j$, and $o_j$ is used to denote the output of neuron $j$.

There are $20$ hidden neurons and, for each of them, according to equation $\ref{1}$, we perform $15$ multiplications (ignoring any multiplications that might be associated with the activation function), so $15*20 = 300$ multiplications are performed at the (only) hidden layer. In general, if there are $n$ inputs and $m$ hidden neurons, then $n*m$ multiplications will be performed in the first hidden layer.

Now, each neuron $j$ at the next layer (in this case, the output layer), also performs a linear combination followed by the application of an activation function

\begin{align}
o_j = \tau \left(\sum_{i}^{20} w_{ij}x_i \right),\tag{2}\label{2}
\end{align}

where $\tau$ is another activation function which might or not be equal to $\sigma$, but we ignore all multiplications that might involve the application of the activation functions (we just want to count the ones in the linear combinations). Of course, in this case, $x_i$ corresponds to the activation of neuron $i$ (of the hidden layer).

Similarly to the previous reasoning, there are $2$ output neurons and, to compute the output of each of them, $20$ multiplications are performed (in the linear combination), so there are a total of $2*20 = 40$ multiplications at the output layer.

So, an MLP with $15$ inputs, $20$ hidden neurons and $2$ output neurons will perform $15*20 + 20*2 = 340$ multiplications (excluding activation functions). Of course, in this case, the number of multiplication depends not only on the number of neurons but also on the input size.

In general, an MLP with $n$ inputs, $M$ hidden layers, where the $i$th hidden layer contains $m_i$ hidden neurons, and $k$ output neurons will perform the following number of multiplications (excluding activation functions)

\begin{align}
nm_{1} + m_{1}m_{2} + m_{2}m_{3} + \dots + m_{M-1}m_{M} + m_{M}k = nm_{1} + m_{M}k + \sum_{i=1}^{M-1} m_{i}m_{i+1}
\end{align}

which, in big-O notation, can be written as

\begin{align}
\Theta\left(nm_{1} + m_{M}k + \sum_{i=1}^{M-1} m_{i}m_{i+1} \right)
\end{align}

where $\Theta(\cdot)$ is used (as opposed to $\mathcal{O}(\cdot)$) because this is a strict bound. If you have just one hidden layer, the number of multiplications becomes

\begin{align}
\Theta\left(nm_{1} + m_{1}k \right)
\end{align}

Of course, at each layer, the number of multiplications can be computed independently of the multiplications of the other layers (you can think of each layer as a perceptron), hence we sum (and not e.g. multiply) the multiplications of each layer when computing the total number of multiplications of the whole MLP.

In general, when analyzing the time complexity of an algorithm, we do it with respect to the size of the input. However, in this case, the time complexity (more precisely, the number of multiplications involved in the linear combinations) also depends on the number of layers and the size of each layer. The time complexity of a forward pass of a trained MLP thus is architecture-dependent (which is a similar concept to an output-sensitive algorithm).

You can easily include other operations (sums, etc.) in this reasoning to calculate the actual time complexity of a trained MLP.


How to detect that the fitness landscape of a genetic algorithm is changing over time? I understand that in each generation of a genetic algorithm, that generation must re-prove it's fitness (and then the fittest of that population is taken for the next population).
In this case, I guess it's a presumption that if you take the fittest of each generation, and use them to form the basis of the next generation, that your population as a whole is getting fitter with time.
But algorithmically, how can I detect this? If there's no end goal known, then I can't measure the error/distance from goal? So how can you tell how much each generation is becoming fitter by?

There is no exact way to assess that a genetic algorithm has located a global optima. Indeed there may be multiple global optima. You must fall back to heuristic methods. The fitness of a population is the maximum fitness of any individual. Unless specific measures are taken to maintain diversity the population will converge to an optima, local or global. At that point all individuals will, except for mutation, be identical. You could take the fittest individual of such a population as your solution, but you will not know if the solution is a global or local optima.
Two reasonable heuristics are these. First, run the algorithm till it converges and maintains its fitness for a number of further generations. Or second run the algorithm multiple times, and take the fittest of all the located solutions. Neither is exact.


Detect street and sidewalk surface in aerial imagery (neural network) I would like to detect street and sidewalk surface in a very detailed (0.075m/pix) USGS High Resolution Orthoimagery which basically means image segmentation with two classes. Places in question are residential areas similar to this one. I will download uncompressed raw imagery in GeoTIFF from USGS for the detection.

I read that neural networks can perform very good in image segmentation and I would like to try them. I am a developer by day so I can code but am a beginner to neural networks only knowing the basic principles about architecture, weighting and backpropagation etc. Is it possible to jump right in into my task or do I need to start with something simpler? I would prefer jumping right in if it can save time.

I skimmed though few papers dealing with similar thing and they seem quite complicated. Is there some simple way I can get started? I mean maybe an open source project in neural networks that deals with image segmentation that is similar to my task and I could make use of it?

I see neural networks need to be trained first and I am prepared to do manual segmentation first to have data for training. However, I have no idea about neural network design/architecture, how to design the layers, how many layers do I need etc. I also would like to use the fact that the network would learn some basics on how streets and sidewalks are built - that they are (not sure if my term is correct) "linear structures" which usually run many meters in length and may not even end in the image, also that sidewalks usually run alongside streets, streets have intersections etc.

Yes, in fact neural networks (NNs) are very efficient at segmentation and it seems to me that your problem matches the capabilities of neural networks very well.

I think it best for you to truly understand what a NN is before using it. First, let's start with the architecture. A NN has 3 regions, the input layer, the hidden layers and the output layer. The input layer depends on the number of features in your dataset. The hidden layers, you can have multiple layers all of different breadth (number of nodes per layer). The output layer depends on the number of classes in your dataset.

An easy example is applying a NN to the MNIST datatset. This is a dataset which contains handwritten digits between 0-9. Let's assume each of these images is 16*16=256 pixels. Thus, you will need 256 input nodes. And you will need 10 output nodes, one for each possible output. The hidden layers can be set in any way you can creatively imagine. There are however ways to optimize your hidden layer to get the best performance possible while not spending too much computational power.

This is always how a NN works. The beauty of a NN is that you only need to code it once and it can learn any function. All you need to do is change your architecture, but the underlining principles will always be the same.

In your case, you want to do segmentation. This is often done using a window around the pixel you want to classify. Popular choices are 3*3 or 5*5 pixels. The choice of your considered window will determine the number of nodes in your input layer. Then you want to classify them as one of two classes, thus you need 2 output nodes. You can also use just 1, but I don't recommend it, I can expand on this if you care.

One caveat of NN is that you will need quite a bit of training data. So get ready to classify a lot of pixels manually.

How to know how many layers in hidden layer? How to know how many nodes per layer in the hidden layer?

In general, for simple operations like the one you are trying to learn you do not need to have multiple layers. One hidden layer should be enough, at most 2. But, how do you determine the number of nodes you should use? You need to use some model validation techniques to do this. One way is through grid search and cross-validation. Train, and re-train your model with multiple number of nodes and then compare their performances to identify the optimal number of nodes. To get good results this does require a large dataset.

Rule of thumb: 1 hidden layer for NN! Don't get dragged into deep models if you don't need them.


How is trajectory sampling different than normal (importance) sampling in reinforcement learning? I am using Sutton and Barto's book for Reinforcement Learning.
In Chapter 8, I am having difficulty in understanding the Trajectory Sampling.
I have read the particular section on trajectory sampling (Sec 8.6) two times (plus 3rd time partially) but still, I do not get how it is different from the normal sampling update, and what are its benefits.

Here is my understanding:
In trajectory sampling as the book describes it, we use the current policy on the simulator to get (next-state, action) pairs. The advantage is that if some states occur more frequently than others in that environment, and if we take enough samples, the distribution among the samples would be similar to the actual distribution.
On the another hand, you can sample in a different manner if you have access to the model. Suppose you have access to the transition distribution. Then you can sample your start state uniformly, and use the transition distribution to get the (next-state, action) pairs. This can be useful if you want to force your algorithm to look at all states evenly, even if that's not the actual case.


NEAT - Managing species across generations I (mis?)understood the NEAT algorithm has the following steps:

Create a genome pool with N random genomes
Calculate each genome fitness
Assign each genome to a species
Calculate the adjusted fitness and the number of offspring of each species
Breed each species through mutation/crossover from the stronger genomes
go to step 2.

Step 3 is tricky: speciation is made placing each genome G in the first species in which it is compatible with the representative genome of that species, or in a new species if G is not compatible with any existing species. Compatible is meant as having compabilitity distance below a certain threshold. Regarding representative genome NEAT paper says:

  Each existing species is represented by a random genome inside the
  species from the previous generation

Somewhere I've found that keeping the number of species stable is good, and this is achieved automatically with dynamic thresholding. However, dynamic thresholding makes hard to evaluate species behaviour across generations.

Let me give one example:
Assume that in Generation 20, Species 1 has Genome A as representative and Species 2 has Genome B as representative. Assume elitism is implemented.

As the representative genome is taken from previous generation, assume that in Generation 21, Genome A and B are still representatives for Species 1 and 2, however assume compatibility threshold has changed (i.e. bigger) in order to reach the target species number. With this change, A and B have now a compatibility distance lower than threshold and should be placed in the same Species, however they are representatives of different species.

How to solve this issue?

More in general, with dynamic thresholding, how to make sure species management across generations is consistent? E.g. NEAT paper also says:

  If the maximum fitness of a species did not improve in 15 generations,
  the networks in the stagnant species were not allowed to reproduce.

How to make sure that across all 15 generations, we are still considering that same single species and this has not drastically changed (so that they are actually different 'objects'?). E.g. in the example above, if A and B are both placed in Species 1 in Generation 21, Species 2 no longer represents what it represented in Generation 20.

In some implementations, the specie representatives change each generation. This allows for a dynamic definition of what a specie is. If you speciate from scratch(meaning each specie is assigned a new representative) every generation, you won't have the problem where a representative leaves its specie.


Why does Deep Q Network outputs multiple Q values? I am learning Deep RL following this tutorial: https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8

I understand everything but one detail:

This image shows the difference between a classic Q learning table and a DNN.
It states that a Q table needs a state-action pair as input and outputs the corresponding Q value whereas a Deep Q network needs the state as feature input and outputs the Q value for each action that can be made in that state.

But shouldn't the state AND the action together be the input to the network and the network just outputs a single Q value?

I think this was just a "clever" design choice. You can actually design a neural network (NN), to represent your Q function, which receives as input the state and an action and outputs the corresponding Q value. However, to obtain $\max_aQ(s', a)$ (which is a term of the update rule of the Q-learning algorithm) you would need a "forward pass" of this network for each possible action from $s'$. By having a NN that outputs the Q value for each possible action from a given $s'$, you will just need one forward pass of the NN to obtain $\max_aQ(s', a)$, that is, you pick the highest Q value among the outputs of your NN.

In the paper A Brief Survey of Deep Reinforcement Learning (by Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage and Anil Anthony Bharath), at page 7, section "Value functions" (and subsection "Function Approximation and the DQN"), it's written

  It was designed such that the final fully connected layer outputs $Q^\pi(s,\cdot)$ for all action values in a discrete set of actions — in this case, the various directions of the joystick and the fire button. This not only enables the best action, $\text{argmax}_a Q^\pi(s, a)$, to be chosen after a single forward pass of the network, but also allows the network to more easily encode action-independent knowledge in the lower, convolutional layers.


Constraints of embedded Artificial Intelligence for a robot So I was wondering about the actual constraints in the field of Artificially Intelligent agent/robot. We have been able to replicate most of human sensory functions like:

Vision - Computer Algorithms can perform better image classification tasks.
Touch - A physical object more sensitive than skin can be modelled easily.
Taste and smell - Although there are electro-mechanical tongues, since taste and smell and related it is quite complex to model (i.e. no satisfactory devices exist).
Hearing - All types of high fidelity audio recording devices are available.

We have also got processors for these sensors:

Memory - We have got memory running into exabytes probably.
Processing power - We have got a high processing power, albeit it falls somewhat short of human brain according to this answer.
Algorithms - We have got highly developed purely logical algorithms, or we can easily Machine Learn and approximate results of unknown algorithms.
We can also model the human anatomy of various physical functions like walking/running/exercising, etc, like this mechanical cheetah my MIT students.

So my question is what is stopping us from stringing together all this to actually device a quite good Artificially Intelligent agent? (Some people would like to call a good AI as AGI, but I am not defining a good agent as an AGI as I feel the definition of ntelligence is not clear enough, to make comments about AGI, hence feel free to use your own definition of agood AI). What are the constraining factors and how are we trying to overcome it?
My wording maybe little bit off, you are free to edit the question keeping the general theme same.

I think a big problem with intelligent robots is that the world is very dynamic and always changing and our techniques right now are still quite static and not really flexible.

I myself come from a Computer Vision background and in this field I often see some limitations of one of the most promising approaches for AI right now (Deep Learning). For example in Computer Vision we are now able to teach image classification systems that are on the level of human performance (or better) thanks to Deep Learning. But still these classifiers are trained supervised with annotated data and should more be viewed as very smart filters instead of true AI.

In a dynamic world a supervised trained model often fails in specific situations. For example you cannot always train to classify specific objects to specific predefined classes. There are corner cases or no clear boundaries between different classes in some situations that depend on the very specific situation.

An Example:

Consider an autonomous car that classifies between a road and no-road to see were it is able to drive. Most of the time this task is quite straight forward but in some sitations the road border gently transforms into a field of grass or something different that is not driveable. So where would you set the border between both classes i.e. how would you annotate your training data in such a situation? There is no unique solution to this problem.

A supervised trained model would finally probably learn to set the border to somewhere in-between but it doesn't really tell you that there is a corner case or slight transition with a soft border or that it's not really sure about it at all.

So there is problem with the encoding/decoding of information in a neural network. We can apply a full image as an input into the network and that is fine but then we only get a static output of predefined classes or something similar which often is not sufficient information to describe the world to make proper decisions in every situation. I am pretty sure a neural net would be able to learn such rich information but right now there is no real good approach to encode or represent this wanted (dynamic) information in a static predefined network architecture (as far as I know).

And this also brings me to the next problem: there is no real good way to represent uncertainity. There are some approaches like Bayesian Deep Learning which tries to tackle this problem but (as far as I know) this is not really practical or good enough yet. And an intelligent robot should really know its own limits and capabilities to not becoming a serious danger for anyone or itself.

And finally, general learning is usually an ongoing process that means a really intelligent agent should keep learning while it is exploring the world and that best without a supervisor. There are some approaches to this like Deep Reinforcement Learning (where just a reward function is needed) which is also very promising but right now it is very sample inefficient and not easily generally applicable to every task so that it could update its behaviour in an online manner.

Conclusion:

The hard part about it is actually to develop a reliable robot that handles these rare or special cases realliably and/or is able to identify these with its own reliable uncertainity to fail savely. This state is not achieved yet.


How do I recognise a bandit problem? I'm having difficulty understanding the distinction between a bandit problem and a non-bandit problem.

An example of the bandit problem is an agent playing $n$ slot machines with the goal of discovering which slot machine is the most probable to return a reward. The agent learns to find the best strategy of playing and is allowed to pull the lever of one slot machine per time step. Each slot machine obeys a distinct probability of winning.

In my interpretation of this problem, there is no notion of state. The agent potentially can utilise the slot results to determine a state-action value? For example, if a slot machine pays when three apples are displayed, this is a higher state value than a state value where three apples are not displayed.

Why is there just one state in the formulation of this bandit problem? As there is only one action ("pulling the slot machine lever" ), then there is one action. The slot machine action is to pull the lever, which starts the game.

I am taking this a step further now. An RL agent purchases $n$ shares of an asset and its not observable if the purchase will influence the price. The next state is the price of the asset after the purchase of the shares. If $n$ is sufficiently large, then the price will be affected otherwise there is a minuscule if any effect on the share price. Depending on the number of shares purchased at each time step, it's either a bandit problem or not.

It is not a bandit problem if $n$ is large and the share price is affected? It is a bandit problem if $n$ is small and the share price is not affected?

Does it make sense to have a mix of a bandit and non-bandit states for a given RL problem? If so, then the approach to solving should be to consider the issue in its entirety as not being a bandit problem?

The bandit problem has one state, in which you are allowed to choose one lever among $n$ levers to pull.

  Why is there just one state in the formulation of this bandit problem?

There is one state because the state does not change over time. Two notable consequences are that (i) pulling a lever does not change the internals of any slot machine (e.g. the distribution of rewards) and (ii) you are allowed to choose any lever without restrictions.  More generally, there is no sequential aspect of the state in this problem, as future states are unaffected by past states, actions, and rewards.

  It is not a bandit problem if $n$ is large and the share price is affected?

Correct! If the share price is affected, then future states would be influenced by past actions. This is because the price per share is affected, which is one aspect of the state. Thus, you would need to plan a sequential strategy for your purchases.

  It is a bandit problem if $n$ is small and the share price is not affected?

It all depends on the problem: as long as the state before buying shares remains completely the same after you purchase some shares, then yes. Share price being unaffected is only one of the requirements; another example requirement is that the maximum number of shares purchased is fixed at each time step, independent of the shares purchased previously.

  Does it make sense to have a mix of a bandit and non-bandit states for a given RL problem? If so, then the approach to solving should be to consider the issue in its entirety as not being a bandit problem?

It makes sense to allow the share price to either be affected or unaffected based on $n$ in the same problem. Since some actions (large $n$) change the state, then there are multiple states, and actions sequentially affect the next state. Hence it is not a bandit problem as a whole, as you correctly stated.

  The agent potentially can utilise the slot results to determine a state-action value?

Correct! I suggest reading Chapter 2 of Sutton and Barto to learn some fundamental algorithms of developing such strategies.

Nice work on analyzing this problem! To help solidify your understanding and formalize the arguments above, I suggest that you rewrite the variants of this problem as MDPs and determine which variants have multiple states (non-bandit) and which variants have a single state (bandit).


How powerful is OpenAI's Gym and Universe in board games area? I'm a big fan of computer board games and would like to make Python chess/go/shogi/mancala programs. Having heard of reinforcement learning, I decided to look at OpenAI Gym.

But first of all, I would like to know, is it possible using OpenAI Gym/Universe to create a chess bot that will be nearly as strong as Stockfish and create a go bot that will play as good as AlphaGo?

Is it worth learning OpenAI?

OpenAI's Gym is a standardised API, useful for reinforcement learning, applied to a range of interesting environments many of which you can then access for free with little effort. It is very simple to use, and IMO worth learning if you want to practice RL using Python to any depth at all. You could use it to ensure you have good understanding of basic algorithms such as Q learning, independently of and before you look at using RL in a board game context.

There are limitations for Gym and Universe when dealing with multiple agents. The API is not really designed with that in mind. For instance, there is no simple way to add two agents to an environment, you would have to write a new environment and attach an opposing agent inside of it. This is still possible, and not necessarily a terrible idea (it depends on the training setup you want to investigate).

If you want to look into classic two-player games, and write bots like AlphaGo and Stockfish, then I would point out that:

Game-playing bots often make extensive use of planning that can interrogate potential future game states. OpenAI's Gym doesn't prevent you doing that, but it doesn't help in any way.
Algorithms for AlphaGo are public, with many nice tutorials. It would be quicker to follow one of these and develop your own bot training code in most cases, than to try and adapt an OpenAI solution for single agent play.
Probably the biggest time-saver you could find for any game is a rules engine that implements the board, pieces and game rules for you. If Gym already has a game environment for the game you want your bot to play, it might be worth checking the Gym code to see what it is integrating, then try to use the same library yourself, but not the Gym environment directly.
Many decent game-playing algorithms don't use RL at all. You can frame most of them as search (finding best moves) plus heuristics (rating moves or positions), and can usually make independent choices for algorithms that perform each sub-task. You can apply RL so that a bot learns game heuristics, then use a more traditional search e.g. negamax in order to make decisions during play. Or you can use any analysis of the game you like in order to generate heuristics. Very simple games usch as tic-tac-toe (noughts and crosses in UK) can just have heuristic of +1 if X has won, -1 if O has won and 0 otherwise, and still be quickly solved with a minimax search for perfect play.
DeepMind's AlphaGo uses a variant of MCTS for the search algorithm, which can be considered a RL technique, but the lines are a bit blurry around definitions there - it is safer to say that AlphaGo incorporates MCTS as the chosen search technique for both self-play training and active play against any other opponent.


Why I have a different number of terms in word2vec and TFIDF? How I can fix it? I need multiply the weigths of terms in TFIDF matrix by the word-embeddings of word2vec matrix but I can't do it because each matrix have a different number of terms. I am using the same corpus for get both matrix, I don't know why each matrix have a different number of terms
.

My problem is that I have a matrix TFIDF with the shape (56096, 15500) (corresponding to: number of terms, number of documents) and matrix Word2vec with the shape (300, 56184) (corresponding to : number of word-embeddings, number of terms).
And I need the same numbers of terms in both matrix.

I use this code for get the matrix of word-embeddings Word2vec:

def w2vec_gensim(norm_corpus):
    wpt = nltk.WordPunctTokenizer()
    tokenized_corpus = [wpt.tokenize(document) for document in norm_corpus]
    # Set values for various parameters
    feature_size = 300
    # Word vector dimensionality
    window_context = 10
    # Context window size
    min_word_count = 1
    # Minimum word count
    sample = 1e-3
    # Downsample setting for frequent words
    w2v_model = word2vec.Word2Vec(tokenized_corpus, size=feature_size, window=window_context, min_count =  min_word_count, sample=sample, iter=100)
    words = list(w2v_model.wv.vocab)
    vectors=[]
    for w in words:
        vectors.append(w2v_model[w].tolist())
    embedding_matrix= np.array(vectors)
    embedding_matrix= embedding_matrix.T
    print(embedding_matrix.shape)

    return embedding_matrix

And this code for get the TFIDF matrix:

tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True)

def matriz_tf_idf(datos, tv):
    tv_matrix = tv.fit_transform(datos)
    tv_matrix = tv_matrix.toarray()
    tv_matrix = tv_matrix.T
    return tv_matrix

And I need the same number of terms in each matrix. For example, if I have 56096 terms in TFIDF, I need the same number in embeddings matrix, I mean matrix TFIDF with the shape (56096, 1550) and matrix of embeddings Word2vec with the shape (300, 56096). How I can get the same number of terms in both matrix?
Because I can't delete without more data, due to I need the multiplication to make sense because my goal is to get the embeddings from the documents.

Thank you very much in advance.

Your problem is that TFIDF is cutting out around 90 terms.

Since you aren't using the min_df or max_df parameters and as far as I can tell you aren't doing any stemming/lemmatization, the only difference I can see between the two methods is the tokenizer.

There are two things I'd try out if I were you:

Try explicitly converting the word2vec corpus to lowercase. TfidfVectorizer does this by default and I can't see where you're doing it in the word2vec pipeline. Ignore this if your corpus is already lowercased.
Try using the nltk.WordPunctTokenizer() with the TfidfVectorizer. You can do this like this:

wpt = nltk.WordPunctTokenizer()
tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True,
                     tokenizer=wpt.tokenize)


How is it possible to teach a neural network to perform addition? I am trying to understant how it works.  How do you teach it say, to add 1 to each number it gets.  I am pretty new to the subject and I learned how it works when you teach it to identify a picture of a number.  I can understand how it identifies a number but I cant get it how would it study to perform addition? I can understand that it can  identify a number or picture using the pixels and assigning weights and then learning to measure whether a picture of a number resembling the weight is assigned to each pixel. But i can't logically understand how would it learn the concept  of adding a number by one.  Suppose I showed it thousands of examples of 7 turning to 8 152 turning into 153 would it get it that every number in the world has to be added by one? How would it get it having no such operation of + ? Since addition does not exist to its proposal then how can it realize that it has to add one in every number? Even by seeing thousands of examples but having no such operation of plus I cant understand it.    I could understand identifying pixels and such but such an operation I cant get the theoretical logic behind it. Can you explain the logic in layman terms?

Welcome to AI.SE @bilanush.

Here's an example approach that might make things clearer. There are other ways to train a neural network to do this however.

In your earlier example with an image, you probably noticed that the network receives the image as a series of values, representing each pixel in the image. The network then learns which of a series of output neurons should be active in response to a given set of pixel values. Those output neurons, when read in an appropriate way, correspond to the correct label for the image. The difference between the set of outputs that should have been active, and the set that were active, forms the basis of the error signal that allows the network to learn.

You've probably heard that computers represent numbers with binary digits. So you could think of the number 16 as being: 00010000 in "8-bit binary". In 16-bit binary, this number would be 0000000000010000, and so on.

So one way of viewing your problem is a function mapping binary inputs to binary outputs (very similar to labelling a black-and-white image). For instance, the input 00010000 (16) should produce the output 00010001 (17). The input 00100011 (35) should produce the output 00100100 (36), and so on.

As before, you will have a set of output neurons. In this case, it should be wide as the set of input neurons. As before, the error signal is the difference between the expected inputs and outputs.

As to the question of how they can learn this function "without plus", in fact the individual neurons in a network perform just two operations: addition of their inputs, and a non-linear transformation of the sum. It has been proven that these are sufficient to learn any function from inputs to outputs, as long as the network contains 3 layers or more, and the as long as the middle layer is wide enough, but here it should be easy to see how addition might emerge.