Is there any situation in which breadth-first search is preferable over A*? Is there any situation in which breadth-first search is preferable over A*?

The only general situation that comes to my mind where BFS could be preferred over A* is when your graph is unweighted and the heuristic function is $h(n) = 0, \forall n \in V$. However, in that case, A* (which is equivalent to UCS) behaves like BFS (except for the goal test: see section 3.4.2 of this book), i.e. it will first expand nodes at level $l$, then at level $l+1$, etc., this is because nodes at level $l+1$ are farther away from the initial node than nodes at level $l$, i.e. $f(n') = g(n') > f(n) = g(n)$, for all $n' \in V_{l+1}$ and $n \in V_{l}$, where $g(n)$ is the cost of the shortest path from the initial node to $n$ and $V_{l}$ is the subset of nodes of the search space that belong to level/layer $l$ of the search tree.
So, if you think that your goal is close to the initial node in terms of levels/layers and each node has a small branching factor (i.e. not many children), then BFS may be a good idea (but you could also use A*).
BFS can also be used as a sub-routine in other algorithms, such as the Ford–Fulkerson algorithm. So, in these cases, BFS may also be preferable.


How to overcome overfitting to single player styles in reinforcement learning? I am implementing an actor-critic reinforcement learning algorithm for winning a two player tic-tac-toe like game. The agent is trained against a min-max player and after a number of episodes is able to learn a set of rules which lead it to winning a good majority of games.

However, as soon as I play against the trained agent by using even a slightly different playing style, it looses miserably. In other words, it is evident the agent overfitted with respect to the deterministic behaviour of the min-max player. It is clear to me what are the roots of the problem, but I would like to get an overview of the different methodologies which can be applied to overcome (or mitigate) this issue.

The two solutions I would like to try are the following:
1. Training the agent with different opponents for fixed amounts of episodes (or time) each. So that for example I train the agent by using a depth 2 min-max player for the first 10000 episodes, then I use a random playing agent for the next 10000 episodes, then I use a depth 4 min-max player for other 10000 episodes and repeat the process.
2. Starting episodes from different initial configurations. In this way the agents will play a much wider set of sampled games and will be more difficult for the agent to overfit.

Are these two reasonable approaches? Are there other tricks/good practices to try out?

Both of the solutions you suggest seem to be built around the intuition that it's good to ensure that there is sufficient variety in the experiences that you provide to your RL algorithm.

That intuition is good, but it should not come at (too much of) a cost in playing strength of the opponent. I'm afraid that your first solution may break down because of this; Tic Tac Toe is such a simple game, any agent that doesn't play optimally can really be viewed as a very poor agent... and I think minimax agents with very low limits on search depth may end up playing suboptimally. Your second solution seems better in this regard, that could help.

For this particular case of Tic Tac Toe, I suspect you should be able to train just fine against only optimal minimax agents, as long as you make sure that those optimal minimax agents break ties randomly. In some situations, there may be multiple different actions that are all "equally optimal". Then, you'll want to make sure that your minimax agents break ties randomly, rather than always deterministically picking the same action. This can be done, for example, by making sure to always shuffle your lists of legal moves in minimax after they are generated.


What is a heatmap in the CornerNet paper? I have been working on understanding how CornerNet works, but I couldn't figure out a few parts about the architecture.
First, the authors mention that there are 3 distinct parts to be predicted as a heatmap, embedding, and offset.
Also, in the paper, it is stated that the network was trained on the COCO dataset, which has bounding box and class annotations.
As far as I am concerned, since CornerNet is based on detecting the top-left and bottom-right corners, the ground-truth labels for heatmap should be composed of top-left and bottom-right pixel locations of bounding boxes with the class score (but I might be wrong). What is the heatmap used for?
Moreover, for the embedding part, authors used the pull&push loss at the ground-truth pixel locations to find out which corner pairs belong to which object, but I don't understand how to backpropagate this loss. How do I back-propagate the embedding loss?

Heatmap in the sense of Corner net is the heatmap of the pooled corner values. As discussed in the paper, there is a corner pooling operation that gives you the vector values for a pixel point being a corner(it may or not). The output from the corner pooling is CxHxW. Then to generate a heat map you train the network similar to the Grad-CAM method. Training for heatmap generation means that we are going to tell how much a given pixel point has a weight to be a corner point. i.e if it has a high weight it will generate more colored values on that pixel area.
The heatmap is used for the prediction of corner points. That means if you know which pixel has a higher weight then that pixel will have a higher probability to hold the top-right or bottom-left corner of the given bounding box.


What is a "trajectory" in reinforcement learning? I'm now learning about reinforcement learning, but I just found the word "trajectory" in this answer.

However, I'm not sure what it means. I read a few books on the Reinforcement Learning but none of them mentioned it. Usually these introductionary books mention agent, environment, action, policy, and reward, but not "trajectory".

So, what does it mean?
According to this answer over Quora:

  In reinforcement learning terminology, a trajectory $\tau$ is the path of the agent through the state space up until the horizon $H$. The goal of an on-policy algorithm is to maximize the expected reward of the agent over trajectories.

Does it mean that the "trajectory" is the total path from the current state the agent is in to the final state (terminal state) that the episode finishes at? Or is it something else? (I'm not sure what the "horizon" mean, either).

In answer that you linked, I may have used an informal definition of "trajectory", but essentially the same thing as the quote. A "trajectory" is the sequence of what has happened (in terms of state, action, reward) over a set of contiguous timestamps, from a single episode, or a single part of a continuous problem.

So $(s_3, a_3, r_4, s_4, a_4, r_5, s_5, a_5, r_6, s_6)$ taken from any scenario where an agent was used in the problem environment would be a trajectory - at least as I intended it in the answer. This could be from real-world data, or a simulation. It could involve a totally random or untrained agent, or a fully-optimised policy.

In the other definition that you have found, the focus on states and a horizon could make it slightly different, but actually I suspect that it is the same thing, as it is not that useful to only know the states. The Quora answer is probably just using "path of the agent through the state space" as shorthand to describe the same data.

A "horizon" in reinforcement learning is a future point relative to a time step, beyond which you do not care about reward (so you sum the rewards from time $t$ to $t+H$). Fixed horizons can be used as an alternative to a discount factor for limiting sums of reward in continuous problems. They may also be used in other approaches, but basically mean the same thing - a time step beyond which you don't account for what happens.


Recognizing Set CARDs Set is a card game and is Nicely described here.

Each set-card has 4 properties:

The number(1,2 or 3)
the color (Red, Green or Purple)
Fill (Full, Stripes, None)
Form (Wave, Oval or Diamond)

converts to 2 Purple Waves No fill (code: 2PWN)

 and

convert to codes 1RON and 3GDN

For every combination there is one card so in total there are 3^4 = 81 cards. The goal is to identify 3 cards (set) out of collection of 12 displayed randomly chosen set cards where all properties occur 0,1 or 3 times.

As a hobby project I want to create an android app which can -with the camera- capture the 12 (less or more) set cards and indicate the sets present in the collection of 12. I'm looking for ways to leverage image recognition as efficient as possible.

I've been thinking of taking multiple pictures of all the individual cards, label them and feeding them to a trainer (firebase ML KIT AutoML Vision Edge) But I have the feeling that this a bit of brute force and takes a lot of time and effort photographing and labeling. I could also take pictures of multiple set cards and provide the different codes as labels.

What would be the best (most efficient) approach to have a model for labelling all cards?

Since you only have fixed types.

For colour, I think it is fairly straight forward.
For number, simplest way is to plot a projection histogram and count the points of discontinuity.

An example of the projection histogram

For fill, You can find the number of islands. Islands of background colour.
For shape, Like Clement Hui suggested you can use shape detection


Are there any good ways of simultaneously incorporating object detection with speech recognition? Are there any good ways of simultaneously incorporating object detection with speech recognition? For example, if you want to identify whether an animal is a dog or cat, we can obviously use visual features (e.g. YOLO, CNNs, etc.). But how would you incorporate speech and sound in this model?

Check out this paper. It deals with the problem of mixing input modalities! Here's the abstract.

This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.


Is there a reason evolutionary algorithms are language-bound in research material? I've been working on genetic algorithms & evolutionary strategies for a while now in a research context. Across the vast majority of the articles and content I've read, every single one of them will either use Python, Matlab, or Java/C++ to build & benchmark their algorithms.

Is there an objective reason for these languages to be the single ones used in a research environment? Mainly in contrast with other languages like C#, or Javascript, that are almost never used (despite being some of the most used programming languages in other areas), whereas it would definitely be possible to code in practice all current algorithms in them.

I would say there are quite a few different reasons for this, with the proportion of each dependent on a given researcher.

For example, I use python for the vast majority of what I do. And for me, it is due to a few different factors:

I was already familiar with python, and it is a simple, high-level abstraction. This is probably the reason a lot of people set it and forget it, particularly with python. It allows them to focus on the ideas and implementation of said ideas, without worrying about all the junk that comes with trying to write a program in a faster language like C++
The vast majority of ML/DS packages are only or primarily supported via python. I think this is probably the main reason for most in the field, as even if one can implement the architecture in a faster language, the time to do so would likely even out when taking into account the time required to prototype a given model. Tensorflow and others are supported for other languages but do not see the same level of dev support.
The ability to deploy models to multiple platforms without headache. When working in an environment where the work is also applied, the ability to deploy a given model without too much debugging cannot be understated.

These are just a few of the main ones, and as I said the reasons for a particular language over another is primarily a preferential one, and can even vary by requirements(i.e speed)


Why AI is (or not) a good option for the generation of random numbers? Why AI is (or not) a good option for the generation of random numbers? Would GANs be suited for this purpose?

Why AI is (or not) a good option for the generation of random numbers?

AI approaches are generally not good for generating random numbers, for these reasons:

Similar to why they are not good for adding numbers, there already exist many strong pseudo-random and "true" random sources, possible without using any AI approach, and demonstrably good enough for purposes such as simulations and cryptography. These existing algorithms and devices perform much more efficiently at the task than any AI could.
Generating random numbers with good statistical properties is a very precise task, that the approximations typically used in machine learning cannot cope with (a pseudo-random function is inherently noisy and difficult to learn). AI search techniques are not helpful either, because the amount of output that needs to be generated in order to validate a random data source is very large, making things like tree searches for solutions intractable.

One possible relationship between AI and random number generation that could be explored is in code generation and code analysis for random number generators. In other words, an AI that is sophisticated enough to understand the complexity of code used in a typical random number generator (RNG) might help with making random number generation more efficient, or less easy to hack, or expose flaws in existing implementations. This is beyond the capabilities of AI techniques at the moment - or put another way, human intelligence has taken RNG technology quite far already, so there is very little practical that current AIs can do to improve on it.

  Would GANs be suited for this purpose?

No. The input to a GAN is typically a vector of random numbers that have been generated, by an already-existing RNG. The purpose the the GAN is convert this simply-distributed random vector input $\mathbf{x}$ - which might be suitable for many other purposes, such as running simulations of stochastic processes - into a $\mathbf{y}$ sampled from a distribution of some target population $\mathbb{Y}$, which is typically being expressed as a subset of a large space $\mathbb{R}^d$. The values that $\mathbf{y}$ will take are biased to a subspace (or manifold) of all possible space within the representations possible in that vector space. For instance, you might input a random vector with 100 variables and the GAN will generate a picture of a dog. It cannot generate anything other than attempts at pictures of dogs, whilst the space of all possible images that it might create if completely random is far larger (and usually not interesting to look at).

So a GAN would take an already-good random source and turn it into a biased set of numbers that would be generally unusable as a RNG without further processing. The kind of processing you would do (e.g. "noise whitening") is something that some current "secure" random generators do - meaning to use the GAN to generate quality random numbers you would literally put existing RNG code on its input and output layers. The GAN would add nothing but wasted computation.

You cannot even train a GAN to do this task, since a truly random space contains all possible outputs, there will be no negative examples for the discriminator. Although a random picture most often looks like static noise, and a picture of the Mona Lisa is clearly considered "not random" by most people, in fact it is just as likely to be seen as any other image.


How are artificial neural networks different from normal computer programs? How are artificial neural networks different from normal computer programs (or software)?

Well, a computer consists of the hardware (CPU, GPU, memory, etc.), the operational system (OS) and software/programs (OS is also a software, but it is a very special software).

A neural network is simply a program that runs on a computer. Like a calculator app. In fact, the most simple NNs are indeed just some sort of a calculator that performs matrix multiplications.


What is the difference between AI architecture and AI model? What is the difference between AI architecture and AI models? Are both of them the same? If not, please distinguish both of them and give example of each.

Well simply put AI model can be seen just as a flowchart showing how the control flow moves where it moves how it moves why it moves etc.
However AI architecture refers to the next step after building an AI model AI architecture involves representation of the functions that you use in your program. It also involved declaration of the variables you're going to use, the functions that you are going to use, etc.


Clarifying when a loss/cost function calculation and backprop take place I read different articles and keep getting confused on this point. Not sure if the literature is giving mixed information or I'm interpreting it incorrectly.

So from reading articles my understanding (loosely) for the following terms are as follows:

Epoch:
One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE.

Batch Size:
Total number of training examples present in a single batch. In real life scenarios of utilising neural nets, the dataset needs to be as large as possible, for the network to learn better. So you can’t pass the entire dataset into the neural net at once (due to computation power limitation). So, you divide dataset into Number of Batches.

Iterations:
Iterations is the number of batches needed to complete one epoch. We can divide the dataset of 2000 examples into batches of 500 then it will take 4 iterations to complete 1 epoch.

So if all is correct, then my question is, at what point does the loss/cost function and the subsequent backprop processes take place (assuming from my understanding that backprop takes place straight after the loss/cost is calculated)? Does the cost/loss function gets calculated:

At the end of each batch where the data samples in that batch have been forward-fed to the network (i.e. at each "Iteration, not each Epoch")? If so, then the loss/cost functions gets the average loss of all losses of all data samples in that batch, correct?
At the end of each Epoch? Meaning all the data samples of all the batches are forward-fed first, before the a cost/loss function is calculated.

My understanding is that it's the first point, i.e. at the end of each batch (passed to the network), hence at each iteration (not Epoch). At least when it comes to SGD optimisation. My understanding is - the whole point is that you calculate loss/cost and backprop for each batch. That way you're not calculating the average loss of the entire data samples. Otherwise you would get a very universal minima value in the cost graph, rather than local minima with lower cost from each batch you train on separately. Once all iterations have taken place, then that would count as 1 Epoch.
But then I was watching a YouTube video explaining Neural Nets, which mentioned that the cost/loss function is calculated at the end of each Epoch, which confused me. Any clarification would be really appreciated.

Epoch: One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE.

  Batch Size: Total number of training examples present in a single
  batch. In real life scenarios of utilising neural nets, the dataset
  needs to be as large as possible, for the network to learn better. So
  you can’t pass the entire dataset into the neural net at once (due to
  computation power limitation). So, you divide dataset into Number of
  Batches.

  Iterations: Iterations is the number of batches needed to complete one
  epoch. We can divide the dataset of 2000 examples into batches of 500
  then it will take 4 iterations to complete 1 epoch.

This is for the most part correct, except there are other reasons you would sometimes want to use batches (even if you could fit the whole thing in memory). One is that its less likely to overfit in a stochastic setting then in the full setting. Another is that it can achieve similar extrema with faster convergence.

Now regarding your question, yes you apply the gradient descent step at the end of each batch or desired batch (what I mean by desired batch, is if you want to use a batch-size of 24 but your device can only process 8, you may use gradient accumulation of 3 pseudo-batches to achieve an emulated batch of 24).

Though I do think its worth mentioning, your goal is still to find the global minimum. Even if your batch size is the entire dataset, that does not mean you will not fall into a local minimum, its actually in most cases still the most probable outcome


How should we choose the dimensions of the encoding layer in auto-encoders? How should we choose the dimensions of the encoding layer in auto-encoders?

The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so it should be small enough to be trainable in a reasonable time.


Is Expected SARSA an off-policy or on-policy algorithm? I understand that SARSA is an On-policy algorithm, and Q-learning an off-policy one.
Sutton and Barto's textbook describes Expected Sarsa thusly:

  In these cliff walking results Expected Sarsa was used on-policy, but
  in general it might use a policy different from the target policy to
  generate behavior, in which case it becomes an off-policy algorithm.

I am fundamentally confused by this - specifically, how do we define when Expected SARSA adopts or disregards policy. The Coursera Course states that it is On-Policy, further confusing me.

My confusions became realized when tackling the Udacity course, specifically a section visualizing Expected SARSA for simple a gridworld (See section 1.11 and 1.12 in link below). Note that the course defines Expected Sarsa as on-policy.
https://www.zhenhantom.com/2019/10/27/Deep-Reinforcement-Learning-Part-1/

You'll notice the calculation for the new state value Q(s0,a0) as

  Q(s0, a0) <— 6 + 0.1( -1 + [0.1 x 8] + [0.1 x 7] + [0.7 x 9] + [0.1
  x 8] - 6) = 6.16.

This is also the official answer. But this would mean that it is running off policy, given that it is stated that the action taken at S1 corresponds to a shift right, and hence expected SARSA (On policy) should yield you.

  Q(s0, a0) <— 6 + 0.1( -1 + [0.1 x 8] + [0.1 x 7] + [0.1 x 9] + [0.7 x
  8] - 6) = 6.1

The question does state

  (Suppose that when selecting the actions for the first two timesteps
  in the 100th episode, the agent was following the epsilon-greedy
  policy with respect to the Q-table, with epsilon = 0.4.)

But as this same statement existed for the regular SARSA example (which also yields 6.1 as A1 is shift right, as before), I disregarded it.

Any advice is welcome.

Expected SARSA can be used either on-policy or off-policy.

The policy that you use in the update step determines which it is. If the update step uses a different weighting for action choices than the policy that actually took the action, then you are using Expected SARSA in an off-policy way.

Q-learning is a special case of Expected SARSA, where the target policy is greedy with respect to the action values, so there is only ever one $r_{t+1} + \gamma \text{max}_{a'} Q(s_{t+1}, a')$ term to add with a probability $1$.

You can also use Expected SARSA, similarly to SARSA, where the behaviour policy and target policy are identical. It is not identical to SARSA though, because it calculates the TD Target over all possible actions $r_{t+1} + \gamma \sum_{a'} \pi(a'|s_{t+1}) Q(s_{t+1}, a')$

You can construct Expected SARSA updates where $\pi(a|s)$ is different when selecting which action to explore in the environment (behaviour) and when updating the Q values (target). For instance, you can decide to explore using $\epsilon$-greedy with $\epsilon=0.1$ and update the value function with $\epsilon=0.01$.


How is the policy gradient calculated in REINFORCE? Reading Sutton and Barto, I see the following in describing policy gradients:

How is the gradient calculated with respect to an action (taken at time t)? I've read implementations of the algorithm, but conceptually I'm not sure I understand how the gradient is computed, since we need some loss function to compute the gradient.

I've seen a good PyTorch article, but I still don't understand the meaning of this gradient conceptually, and I don't know what I'm looking to implement. Any intuition that you could provide would be helpful.

The first part of this answer is a little background that might bolster your intuition for what's going on. The second part is the more practical and direct answer to your question.

The gradient is just the generalization of the derivative to multivariable functions. The gradient of a function at a certain point is a vector that points in the direction of the steepest increase of that function.

Usually, we take a derivative/gradient of some loss function $\mathcal{L}$ because we want to minimize that loss. So we update our parameters in the direction opposite the direction of the gradient.

$$\theta_{t+1} = \theta_{t} - \alpha\nabla_{\theta_t} \mathcal{L} \tag{1}$$

In policy gradient methods, we're not trying to minimize a loss function. Actually, we're trying to maximize some measure $J$ of the performance of our agent. So now we want to update parameters in the same direction as the gradient.

$$\theta_{t+1} = \theta_{t} + \alpha\nabla_{\theta_t} J \tag{2}$$

In the episodic case, $J$ is the value of the starting state. In the continuing case, $J$ is the average reward. It just so happens that a nice theorem called the Policy Gradient Theorem applies to both cases. This theorem states that

$$\begin{align}
\nabla_{\theta_t}J(\theta_t) &\propto \sum_s \mu(s)\sum_a q_\pi (s,a) \nabla_{\theta_t} \pi (a|s,\theta_t)\\
&=\mathbb{E}_\mu \left[ \sum_a q_\pi (s,a) \nabla_{\theta_t} \pi (a|s,\theta_t)\right].
\end{align}\tag{3}
$$

The rest of the derivation is in your question, so let's skip to the end.

$$\begin{align}
\theta_{t+1} &= \theta_{t} + \alpha G_t \frac{\nabla_{\theta_t}\pi(A_t|S_t,\theta_t)}{\pi(A_t|S_t,\theta_t)}\\
&= \theta_{t} + \alpha G_t \nabla_{\theta_t} \ln \pi(A_t|S_t,\theta_t)
\end{align}\tag{4}$$

Remember, $(4)$ says exactly the same thing as $(2)$, so REINFORCE just updates parameters in the direction that will most increase $J$. (Because we sample from an expectation in the derivation, the parameter step in REINFORCE is actually an unbiased estimate of the maximizing step.)

Alright, but how do we actually get this gradient? Well, you use the chain rule of derivatives (backpropagation). Practically, though, both Tensorflow and PyTorch can take all the derivatives for you.

Tensorflow, for example, has a minimize() method in its Optimizer class that takes a loss function as an input. Given a function of the parameters of the network, it will do the calculus for you to determine which way to update the parameters in order to minimize that function. But we don't want to minimize. We want to maximize! So just include a negative sign.

In our case, the function we want to minimize is
$$-G_t\ln \pi(A_t|S_t,\theta_t).$$

This corresponds to stochastic gradient descent ($G_t$ is not a function of $\theta_t$).

You might want to do minibatch gradient descent on each episode of experience in order to get a better (lower variance) estimate of $\nabla_{\theta_t} J$. If so, you would instead minimize
$$-\sum_t G_t\ln \pi(A_t|S_t,\theta_t),$$
where $\theta_t$ would be constant for different values of $t$ within the same episode. Technically, minibatch gradient descent updates parameters in the average estimated maximizing direction, but the scaling factor $1/N$ can be absorbed into the learning rate.


When is max pooling exactly applied in convolutional neural networks? When using convolutional networks on images with multiple channels, do we max pool after we sum the feature map from each channel, or do we max pool each feature map separately and then sum?

What's the intuition behind this, or is there a difference between the two?

The pooling operation is applied to the output of the convolution layer. More precisely, it is applied separately for each of the input channels (or slices). So, if the pooling layer receives an input volume of $H_i \times W_i \times D$, then it will produce an output volume $H_o \times W_o \times D$, so the depth of the output volume is equal to the depth of the input volume. There is no sum involved in the pooling operation. For example, in the case of max pooling, you will choose the maximum number of a certain 2D window of values. You do this for each of the input channels.

In the article (which is part of some course notes) Convolutional Neural Networks: Architectures, Convolution / Pooling Layers, Andrej Karpathy says

  Pooling layer downsamples the volume spatially, independently in each depth slice of the input volume.

The following diagram (a screenshot of one of the figures from the mentioned article) should provide some intuition behind the pooling operation.


How to use MOPSO to align characters vertically? I need to efficiently align characters vertically using Multi Objective PSO. Alignment is achieved by adding spaces in between a given set of characters.

a b c d e f
b b d h g
c a b f

Might be

- a b - c d e f - -
- - b b - d - - h g
c a b - - - - f - -

Now this is a multi objective solution.
I need to maximize the characters that get aligned vertically and minimize the amount of spaces in between the characters.

I wanted to focus firstly on how to get a set of characters to represent a position of a particle. This would mean that I need to somehow transform a possible set of characters into a position of a particle. If I can somehow achieve this then the rest should fall into place.

How do I transform these set of characters into a position of a particle?
Also is this the best approach or are there better ways to approach this problem?

Your problem appears to be multiple sequence alignment. That problem is well studied, and depending on your application, there are fast algorithms that can perform quite well without recourse to AI or optimization, and which are detailed in the provided link.

If your problem has a lot of local minima, then a PSO might be a good approach. Certainly local search has been widely used for this kind of problem in the past.

Your state should be represented by a list of sequences, each with some blanks inserted. Movement between states can consist of adding or deleting blanks from one or more sequences. The distance between two states can be computed as the sum of the number of operations needed to transform each sequence to its corresponding sequence in the other state, which you can compute in linear time (just sum up the needed additions or deletions between each pair of letters).

Hope that gives you a good starting place!


What makes neural networks so good at predictions? I am new to neural-network and I am trying to understand mathematically what makes neural networks so good at classification problems.

By taking the example of a small neural network (for example, one with 2 inputs, 2 nodes in a hidden layer and 2 nodes for the output), all you have is a complex function at the output which is mostly sigmoid over a linear combination of the sigmoid.

So, how does that make them good at prediction? Does the final function lead to some sort of curve fitting?

Neural networks are good at classifying. In some situations that comes down to prediction, but not necessarily.

The mathematical reason for the neural networks prowess at classifying is the universal approximation theorem. Which states that a neural network can approximate any continuous real-valued function on a compact subset. The quality of the approximation depends on the number of neurons. It has also been shown that adding the neurons in additional layers instead of adding them to existing layers improves the quality of the approximation faster.

Add to that the not well-understood effectiveness of the backpropagation algorithm and you have a setup then can actually learn the function that the UAT promises or something close.


beautify an image with reinforcement learning I am trying to formulate and solve the following problem of image mutation. Suppose I am trying to insert an object image into a "background" image of several objects, and I will need to look for a "sweet spot" to insert the image:

I am tentatively trying to formulate the problem into a reinforcement learning process, with the following elements:

0. initial stage:

a background image where the location of objects within the image has been marked (let's suppose we have a perfect object detector)
another image of a new object, let's say, a human

1. action space:

location (x, y) for the object image to be inserted; in that sense the action space is quite large.

2. environment:

each step I will have a new image to "learn from".
An oracle function F returns 1 or 0 (roughly one computation of F takes 30 seconds).
 This function tells me the latest synthesized image hits the "sweet spot" or not (1 means hit). If so, I will stop the search and return the image.

3. constraint:

the newly inserted object shouldn't overlap with the original objects in the figure.

While my gut feeling is that this problem is somehow similar to the classic "maze escape" problem which can be solved well with reinforcement learning, the action space seems quite large in this problem.

So here are my questions:

In case I would like to formulate this "beautify" image problem into a "deep" reinforcement learning problem, how can I learn from such large action space? Or is it really suitable for a reinforcement learning process?
Can I somehow subsume the "non-overlapping" constraint into the oracle function F? If so, how should I decide the reward score? Any principled or empirical way of deciding so?

The purpose of Reinforcement Learning is to maximize some notion of cumulative reward, leading me to the point (1) :
as far as I understand, there is no timesteps in your problem and the "reward" is immediate.
Thus, I don't think reinforcement is suitable here.

On an other hand, in supervised learning, linear regression is the task of approximating a mapping function (f) from input variables (X) to a continuous output variable (y).
It has much in common with your case. If I am not wrong, you are trying to approximate a function that maps image data to (x,y) coordinates
(the "sweet spot"). So, I think regression would be a better way to go for you.

You could either generate a dataset at first, with images data and associated (x,y) coordinates for only spots validated by your function F, and then train a regression predictive model. Or you could train your model with online learning, by generating batch of images and sweet spot coordinates at each step.

Concerning the point (2), it will highly depend on how is made your F function. Since overlapping spots cannot be sweet spots, the simplest would
be to make your F function return 0 for those spots.


Making decision based on users' reports I have users' reports about an accident. I want to know how to make sure that the number of reports is big enough to take that accident as a true accident and not spam.

My idea is to consider a minimum number of reports in a specific time interval, for example 4 reports in 20 minutes are good enough to believe the existence of that accident.

My question is how can I choose the minimum number of reports and that time interval? Is there some logic to make that decision?

If the only feature you're classifying on is the number of users making a given report, then this isn't really much to do with AI/ML. Just pick a number based on your subjective judgment and go with it.

OTOH, if you can include details of the report itself (as well as the number of reporters), I think you might be able to build a bayesian classifier that would be useful.  If you could consider location, weather, time of day, number of reporters, etc., it seems like you might be able to get something useful put together.


Why is dialogue a hard problem in natural language processing? I've just started learning natural language processing from Dan Jurafsky's videos lectures. In that video, minute 4:56, he is stating that dialogue is a hard problem in natural language processing (NLP). Why?

Dialogue is a hard problem because it requires pretty advanced cognitive functions. Leaving aside all the lower levels of language analysis (phonology if dealing with speech, morphology and syntax), you quickly run into interpretation problems that require a lot of world knowledge.

Simple question and answer is fine, and restricted domains are somewhat easier as well. As soon as you get into a normal conversation, you will refer back to things you said before, so an NLP system would have to recognise that and resolve the reference accordingly. Typically in a conversation you would use variations of reference terms: the first time you mention an object you might describe it fully, but subsequently you will use shorter terms to refer to it.

There is also a structure to conversation. This is typically modelled as conversational moves, and usually moves will have corresponding response-moves. For example, a common sequence would be greeting - greeting. Then you might have question - response - feedback. This sounds fairly easy, but once you try to annotate a dialogue with such moves you will find that it is pretty hard. As far as I am aware, there is no 'grammar' equivalent of describing the structure of conversations.

Often, pragmatic meaning is interfering with the 'surface' meaning of utterances. A statement can actually function as a question, or a question can be a command (or a statement). The pragmatics of utterances depend on the context and also the relationship between the interlocutors. If I talk to my manager, I will use language differently than when I talk to my children.

Dialogue/conversations are hard to analyse. Because of that, descriptive frameworks are still fairly limited. You need to keep track of what has said before, as that can change the way an utterance has to be interpreted. Grammatical analysis is a fairly solved problem, and word sense disambiguation as well. But pragmatics and conversational structure are still on the bleeding edge of linguistic research; at least they were when I was still teaching Discourse Analysis at university a few years ago.

For that reason, chatbots are generally not very good. Sometimes they can fool people into believing they are human speakers, but this is usually done through trickery ("smoke and mirrors") rather than competent handling of conversational structures. It's all in the little box in nbro's answer labelled "DM"...


Does the policy iteration convergence hold for finite-horizon MDP? Most RL books (Sutton & Barto, Bertsekas, etc.) talk about policy iteration for infinite-horizon MDPs. Does the policy iteration convergence hold for finite-horizon MDP? If yes, how can we derive the algorithm?

In the discussion about Neil Slater's answer (that he, sadly, deleted) it was pointed out that the policy $\pi$ should also depend on the horizon $h$. The decision of action $a$ can be influenced by how many steps are left. So, the "policy" in that case is actually a collection of policies $\pi_h(a|s)$ indexed by $h$ - the distance to horizon.
Alternatively, one can look at it as if our state space $\mathcal{S}$ is now extended with that integer. So we can "lift" our original finite-horizon MDP $(\mathcal{S},\mathcal{A},P,R)$ into a new infinite-horizon MDP by substituting:
$$\mathcal{S} \to \left(\mathcal{S}\times\{0,1,\dots,h\}\right) \cup \{\epsilon\}\\
\mathcal{A} \to \mathcal{A},\; R \to \tilde R,\; P \to \tilde P
$$
The new reward and transition functions make sure that the horizon is decreasing, and that we are ending up in a capture state $\epsilon$ that has no future effects:
$$ \tilde P(s'_{n-1}|s_n,a) = P(s'|s,a)\quad\quad \tilde P(\epsilon|s_0,a) =\tilde P(\epsilon|\epsilon,a) = 1\\
\tilde R(s_n,a,s'_{n-1}) = R(s,a,s')\quad\quad
\tilde R(s_0,a,\epsilon) =\tilde  R(\epsilon,a,\epsilon) = 0
$$
This way I've reduced the finite-horizon MDP to an infinite-horizon one. Thus I can reuse the result for the policy iteration convergence of infinite MDPs.
Couple of notes:

At first this feels like a huge increase in the state space making the whole problem unnecessary complex. But this complexity is intrinsic to the problem: both the policy and the value function depend on the distance to horizon. So it is necessary to consider the extended number of unknowns in a single self-consistent manner.
The infinite-horizon policy iteration convergence relies on a discounting factor $\gamma < 1$. The finite-horizon doesn't need $\gamma$ for convergence. That's where I feel like I've cheated a bit.
I came up with this approach myself. It feels quite obvious though. I'd expect this approach to either be mistaken or be already mentioned somewhere in the literature - comments pointing out one or the other are welcome.


Why does the output shape of a Dense layer contain a batch size? I understand that the batch size is the number of examples you pass into the neural network (NN). If the batch size is 10, it means you feed the NN 10 examples at once.
Assuming I have an NN with a single Dense layer. This Dense layer of 20 units has an input shape (10, 3). This means that I am feeding the NN 10 examples at once, with every example being represented by 3 values. This Dense layer will have an output shape of (10, 20).
I understand that the 20 in the 2nd dimension comes from the number of units in the Dense layer. However, what does the 10 (Batch Size) in the first dimension mean? Does this mean that the NN learns 10 separate sets of weights (with each set of weights corresponding to one example, and one set of weights being a matrix of 60 values:3 features x 20 units)?

The Dense layers outputs 20 values per example. And since you have 10 examples in the batch the output is (10, 20) (one set of 20 values per example in the batch).
The nn doesn't learn 10 separate sets of weights. Each set of 20 values is computed with the same weight (and bias if you have any). So if say example 2 and 5 had the same input values, they'll always have the same output values.


Is my proof of equation 0.6 in the book "Reinforcement Learning: Theory and Algorithms" correct? In Sham Kakade's Reinforcement Learning: Theory and Algorithms, this equation (page 17) is used preceding the proof of performance difference lemma.

I am attempting to prove equation 0.6. Here is my current attempt:
\begin{align*}
        \mathbb{E}_{\tau \sim \rho^\pi}\left[\sum\limits_{t=0}^\infty \gamma^t f(s_t,a_t)\right] &= \sum\limits_{t=0}^\infty \gamma^t \mathbb{E}_{\tau \sim \rho^\pi} [f(s_t,a_t)]\\
        &= \sum\limits_{t=0}^\infty \gamma^t \mathbb{E}_{s_t, a_t} [f(s_t,a_t)]\\
        &= \sum\limits_{t=0}^\infty \gamma^t \sum\limits_{s, a} \mathbb{P}(s_t = s, a_t = a) f(s,a)\\
        &= \sum\limits_{t=0}^\infty \gamma^t \sum\limits_{s} \mathbb{P}(s_t = s) \sum\limits_{a}\pi(a_t = a|s_t = s) f(s,a)\\
        &= \frac{1 - \gamma}{1 - \gamma}\sum\limits_{t=0}^\infty \gamma^t \sum\limits_{s} \mathbb{P}(s_t = s) \mathbb{E}_{a \sim \pi(s)} [f(s,a)]\\
        &= \frac{1}{1 - \gamma} \sum\limits_{s} (1-\gamma) \sum\limits_{t=0}^\infty \gamma^t \mathbb{P}(s_t = s) \mathbb{E}_{a \sim \pi(s)} [f(s,a)]\\
        &=\frac{1}{(1-\gamma)} \mathbb{E}_{s \sim d^\pi}\left[\mathbb{E}_{a \sim \pi(s)}\left[f(s,a)\right]\right] \\
\end{align*}
Is the swapping of expectation and summation in this way allowed (given that the series converges)?
Note that this is not the proof of the performance difference lemma, but just an attempt to show equation 0.6, which is used but not proved in the book.

The expectation of a sum is equal to the sum of the expectation this just follows from the linearity property of expectations
$$ \begin{aligned} E[\sum_{t} f(s_t,a_t)] &= \sum_{\tau} p(\tau)\left(\sum_t f(s_t,a_t)\right)
 \\ &= \sum_\tau\sum_{t}p(\tau)f(s_t,a_t)
 \\ &= \sum_t\sum_\tau p(\tau)f(s_{t,\tau},a_{t,\tau})
 \\ &= \sum_tE[f(s_t,a_t)]
\end{aligned} $$
note that in the penultimate line I swapped the sums around which can be understood as looking at all trajectories for a single time-step instead of looking at time-steps within  a single trajectory (the second expression). I append subscript for variables in the penultimate line for clarity since they also depend on which trajectory they were drawn from.

As mentioned by nboro in the comments, the linearity of property only holds for infinite sums if
$$\sum_{i=0}^\infty E[|X|]< \infty  \quad or \quad E\left[\sum_{i=0}^{\infty}|X|\right]. <\infty$$
More details can be found here. Using this theory alongside knowledge that since we know that value functions in RL are always bounded (in continuing tasks discounting factors are introduced) then we can say there exists an $M$ such that
$$ |f(s_{t,\tau},a_{t,\tau})| \leq M \qquad \forall t,\tau $$
Then
$$ \begin{aligned}\sum_t\sum_\tau \lambda^t p(\tau)|f(s_{t,\tau},a_{t,\tau})| &\leq \sum_t\sum_\tau \lambda^t p(\tau)M
\\ &=M\sum_t\lambda^t\sum p(\tau)
\\ &=M\sum\lambda^t
\\ &< \infty
\end{aligned}$$
As for the rest of the proof is there anything else you need clarity on? it looks good to me. From that step to the next step where you change distribution from $p(\tau)$ to $p(s,a)$ is just because of marginalisation but I think you got that.


In off-policy MC control algorithm by Sutton & Barto, why do we perform a last update when sample action is inconsistent with target policy? I have a question about the $W$ term in the off-policy MC control algorithm on Page 111 of Sutton & Barto. I have also included it in the figure below.

My question: shouldn't the check $A_{t} = \pi(S_{t})$ be made before updating $C(S_{t}, A_{t})$ and $Q(S_{t}, A_{t})$? And, at this point if $A_{t} \neq \pi(S_{t}) $ then the inner loop should exit before updating $Q(\cdot)$. If $A_{t} = \pi(S_{t})$ then shouldn't $W$ be updated to
$W = W \frac{1}{b(A_{t}|S_{t})} $ before updating the $Q(s, a)$ and $C(s, a) functions?
The algorithm as stated seems problematic to me. For example, if say the target policy $\pi$ is deterministic and behavior policy $b$ is stochastic. If in period $T-1$ the behavior policy takes an action that is not consistent with $\pi$ then the importance sampling ratio $\rho_{T-1:T-1} = 0$. However, the algorithm as shown would update $Q(S_{T-1}, A_{T-1})$ since the checks I referred to above don't occur until the end of the inner loop. What am I missing here?

I think that this is an intentional subtle detail of the algorithm that ensures the convergence property.  The claim in the book is that for any $b$ that provides us with "an infinite number of returns for each pair of state and action" the target policy $\pi$ will converge to optimal.
Imagine now that we have such a bad policy $b$ that it never aligns with the target policy action at the last step $t=T-1$ of each generated episode: $A_{T-1} = \pi(S_{T-1})$. In that case the weight value will stay $W=1$ and the the algorithm will be reduced to (ignoring $t$ indices for the last $S,A,R$ triplet):
$$\begin{array}{l} C(S,A) \leftarrow C(S,A) + 1 \\
Q(S,A) \leftarrow Q(S,A) + \frac{1}{C(S,A)}\left[R - Q(S,A)\right]
\end{array}$$
Which is just the tabular incremental averaging for the Q values (see for example eq. (2.3)).
If we bail from the for loop before these updates, then no updates would happen at all. And authors won't be able to claim convergence to optimal policy for all those "good" sampling policies $b$.


Why is the equation $\mathbb{E} \left[ (Y - \hat{Y})^2 \right] = \left(f(X) - \hat{f}(X) \right)^2 + \operatorname{Var} (\epsilon)$ true? In the book An Introduction to Statistical Learning, the authors claim (equation 2.3, p. 19, chapter 2)
$$\mathbb{E} \left[ (Y - \hat{Y})^2 \right] =  \left(f(X) - \hat{f}(X) \right)^2 + \operatorname{Var} (\epsilon) \label{0}\tag{0},$$
where

$Y = f(X) + \epsilon$, where $\epsilon \sim \mathcal{N}(0, \sigma)$ and $f$ is the unknown function we want to estimate
$\hat{Y} = \hat{f}(X)$ is the output of our estimate of $f$, i.e. $\hat{f} \approx f$

They claim that this is easy to prove, but this may not be easy to prove for everyone. So, why is equation \ref{0} true?

Let's say we have $a$ - constant and $\epsilon \sim \mathcal{N}(0,\sigma)$, then:
$$\mathbb{E}\left[(a+\epsilon)^2\right] = \mathbb{E}\left[a^2\right] + 2 \mathbb{E}\left[a\right]\mathbb{E}\left[\epsilon\right] + \mathbb{E}\left[\epsilon^2\right] $$
Expectations of constants are just the constants: $\mathbb{E}[a] = a$ and  $\mathbb{E}[a^2] = a^2$
The mean of $\epsilon$ is zero $\mathbb{E}[\epsilon] = 0$. And the expectation of $\epsilon^2$ is its variance:
$$ \mathop{\mathrm{Var}}(\epsilon)  = \mathbb{E}[\epsilon^2] - \mathbb{E}[\epsilon]^2 = \mathbb{E}[\epsilon^2]$$
Substituting, we get an expression for the original expectation:
$$\mathbb{E}\left[(a+\epsilon)^2\right] = a^2 + \mathop{\mathrm{Var}}(\epsilon) \tag{*}$$
Getting to the expectation in the book, we first substitute the values for $Y$ and $\hat{Y}$:
$$\mathbb{E}\left[(Y - \hat{Y})^2\right] = \mathbb{E}\left[((f(X) - \hat{f}(X)) + \epsilon)^2\right]$$
In the book it is assumed that $X$, $f$ and $\hat{f}$ are constant. So we can use the expression (*) with the constant being $a = f(X) - \hat{f}(X)$:
$$\mathbb{E}\left[(Y - \hat{Y})^2\right] = (f(X) - \hat{f}(X))^2 + \mathop{\mathrm{Var}}(\epsilon)$$


Are there any discount-factors based on branching factors? I recently came across this function:

$$\sum_{t = 0}^{\infty} \gamma^t R_t.$$

It's elegant and looks to be useful in the type of deterministic, perfect-information, finite models I'm working with.

However, it occurs to me that using $\gamma^t$ in this manner might be seen as somewhat arbitrary.

Specifically, the objective is to discount per the added uncertainty/variance of "temporal distance" between the present gamestate and any potential gamestate being evaluated, but that variance would seem to be a function of the branching factors present in a given state, and the sum of the branching factors leading up to the evaluated state.

Are there any defined discount-factors based on the number of branching factors for a given, evaluated node, or the number of branches in the nodes leading to it?

If not, I'd welcome thoughts on how this might be applied.

An initial thought is that I might divide 1 by the number of branches and add that value to the goodness of a given state, which is a technique I'm using for heuristic tie-breaking with no look-ahead, but that's a "value-add" as opposed to a discount.

For context, this is for a form of partisan Sudoku, where an expressed position $p_x$ (value, coordinates) typically removes some number of potential positions $p$ from the gameboard.  (Without the addition of an element displacement mechanic, the number of branches can never increase.)

On a $(3^2)^2$ Sudoku, the first $p_x$ removes $30$ out of $729$ potential positions $p$, including itself.

With each $p_x$, the number of branches diminishes until the game collapses into a tractable state, allowing for perfect play in endgames.  [Even there, a discounting function may have some utility because outcomes sets of ratios. Where the macro metric is territorial (controlled regions at the end of play), the most meaningful metric may ultimately be "efficiency" (loosely, "points_expended to regions_controlled"), which acknowledges a benefit to expending the least amount of points $p_x$, even in a tractable endgame where the ratio of controlled regions cannot be altered. Additionally, zugzwangs are possible in the endgame, and in that case reversing the discount to maximize branches may have utility.]

 $(3^2)^2 = 3x3(3x3) = "9x9"$ but the exponent is preferred so as not to restrict the number of dimensions.

First, an important note on any form of discounting: adding a discount factor can change what the optimal policy is. The optimal policy when a discount factor is present can be different from the optimal policy in the case where a discount factor is absent. This means that "artificially" adding a discount factor is harmful if we expect to be capable of learning / finding an optimal policy. Generally, we don't expect to be capable of doing that though, except for the case where we have an infinite amount of processing time (which is never in practice). In that other answer which you linked to in your question, I describe that it can still be useful, that it may help to find a good (not optimal, just good) policy more quickly, but it does not come "for free".

I'm not aware of any research evaluating ideas like the one in your question. I am not 100% sure why, I suspect it could be an interesting idea in some situations, but would have to be investigated carefully due to my point above; if not evaluated properly, it could also unexpectedly be harmful.

One thing to note is that the use of discounting factors $\gamma < 1$ is extremely common (basically ubiquitous) in Reinforcement Learning (RL) literature, but rather rare in literature on tree search algorithms like MCTS (though not non-existant; for example, it's used in the original UCT paper from 2006). For the concept of "branching factors", we have the opposite; in RL literature it is very common to consistently have the same action space regardless of states ("constant branching factor"), whereas this is very uncommon in literature on tree search algorithms. So, the combination of discount factors + branching factors is actually somewhat rare in existing literature (which of course doesn't mean that the idea couldn't work or be relevant, it just might explain why the idea doesn't appear to have been properly investigated yet).

One important concern I do have with your idea is that it seems like it could be somewhat of an "anti-heuristic" in some situations (with which I mean, a heuristic that is detrimental to performance). In many games, it is advantageous to be in game states where you have many available moves (a large branching factor), this can mean that you are in a strong position. Consider, for example, chess, where a player who is in a convincingly winning position likely has more moves available than their opponent. I suspect your idea, when applied to chess, would simply promote an aggressive playing style where both players capture as many pieces as possible in an effort to quickly reduce the branching factors across the entire tree.

  If not, I'd welcome thoughts on how this might be applied.

  (I might divide 1 by the number of branches and add that value to the goodness of a given state, but that's a "value-add" as opposed to a discount.)

Such an additive change would be more closely related to the idea of reward shaping, rather than discounting (which is again a thing that, if not done carefully, can significantly alter the task that you're optimizing for). Intuitively I also suspect it might not do anything at all since you'd always be adding the same constant value regardless of which move you took (regardless of which move you took, your parent state will always have had the same branching factor). I might be missing out on some details here, but I think you'd have to have a multiplicative effect on your regular observed rewards.

I suppose one example that could be worth a try would be something like maximizing the following return;

$$\sum_{t = 0}^{T} \gamma^t \beta^{b(S_{t}) - 1} R_{t + 1},$$

where:

$0 \leq \gamma \leq 1$ is the normal time-based discount factor (can set it to $1$ if you like)
$0 \leq \beta \leq 1$ is a new branching-factor-based discount factor
$b(S_t)$ is the branching factor (the number of available moves) in state $S_t$
$R_{t + 1}$ is the immediate reward for transitioning from state $S_t$ to $S_{t + 1}$

I put the $-1$ in the power of $\beta$ because I suspect you wouldn't want to do any discounting in situations where only one move is available. Intuitively, I do suspect this $\beta$ would require very careful tuning though. It is already quite common to choose $\gamma = 0.9$ or $\gamma = 0.99$, with $\beta$ you may want to stay even closer to $1$.


Could an AI be sentient? In theory, could an AI become sentient, as in learning and becoming self-aware, all from its source code?

In theory, if one could build a computing device that matched or exceeded the cognitive capabilities of a sentient being, it should be possible.

(Singlarity adherents believe we will one day be able to transfer the human mind into an artificial computing platform, and it logically follows that one could "hack" such a mind, or build from the ground up, to create a truly Artificial Intelligence.)

But this may be like fusion power, where the old adage is that it is "always 20 years away."


What does the notation ${s'\sim T(s,a,\cdot)}$ mean? I have been seeing notations on Expectations with their respective subscripts such as $E_{s_0 \sim D}[V^\pi (s_0)] = \Sigma_{t=0}^\infty[\gamma^t\phi(s_t)]$. This equation is taken from https://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf and $Q^\pi(s,a,R) = R(s) + \gamma E_{s'\sim T(s,a,\cdot)}[V^\pi(s',R)]$ ,in the case of the Bayesian IRL paper.(https://www.aaai.org/Papers/IJCAI/2007/IJCAI07-416.pdf)

I understand that $s_0 \sim D$ means that the starting state $s_0$ is drawn from a distribution of starting states $D$. But how do we understand the latter with subscript ${s'\sim T(s,a,\cdot)}$ ? How is $s'$ drawn from a distribution of transition probabilities?

The dot ($.$) at the end of $T(s,a,.)$ shows all possible states that we can go from state $S$ by doing action $a$. As you know there are some probabilities here for choosing those states, that the sum of these probabilities is equal to 1. Hence, $T(s,a,.)$ is a probability distribution.


Which neural network is appropriate for measuring object dimensions from stereo images? I have stereo pairs (left, right) images of concrete cracks. I want to measure the length of the crack from those image pairs. Which neural network is appropriate for measuring object dimensions from stereo images?

Note: I am insisted to use the NN-based technique only.

Is the image taken from a constant distance?

If yes, you'd need to scale the images to the same dimensions first of all. For few images say 100-500 images (more the better) you'd need to label the dataset by proper scaling.

Once labeled, use it to train a CNN (Although best would be training a ResNet). Once trained with decent accuracy, test it for the rest of your dataset.

I did something similar for one of my projects, check it out if you want to here.


Why neural networks tend to be trained to recognize multiple things instead of just one? I was watching this series: https://www.youtube.com/watch?v=aircAruvnKk
The series demonstrates neural networks by building a simple number recognizing network.
It got me thinking: Why neural networks try to recognize multiple labels instead of just one? In the above example, the network tries to recognize numbers from 0 to 9. What is the benefit of trying to recognize so many thins simultaneously? Wouldn't it make it easier to reason about if there would be 10 different neural networks which would specialize to recognize only one number at a time?

In practice you never want to classify just a single digit rather than series. In such case you have to pass a patch of image to multiple network, which would make it inconvenient. If you built different accurate models, training parameters will not significantly reduced. For example sloppy written 6, in a single model the probability of being 6 and 0 would be close, not same if you consider likelihood you may get closest answer. While with different models the probability may vary in a greater scale and you may not have good generalization as you have in single model. At the end everything boils down to generalization and in my experience neural networks trained with multiple things have good generalization property that a single.


Can Deep Learning be applied to Computational Fluid Dynamics Can Deep Learning be applied to Computational Fluid Dynamics (CFD) to develop turbulence models that are less computationally expensive compared to traditional CFD modeling?

Read the paper Deep learning in fluid dynamics (by J. Nathan Kutz), and you will find your answer.


DQN not learning and step not stepping towards target I am trying to create a simple Deep Q-Network with 2d convolutional layers.
I can't figure out what I am doing wrong, and the only thing I can see that doesn't seem right is when I get the model prediction for a state after the optimizer step it doesn’t seem to get closer to the target.
I am using pixels from pong in OpenAI's gym with single-channel 90x90 images, a batch size of 32, and replay memory.
As an example, if I try with a batch size of 1, and try running self(states) again right after the optimizer step the output is as follows:
current_q_values -> -0.16351485  0.29163417  0.11192469 -0.08969332  0.11081569  0.37215832
q_target ->         -0.16351485  0.5336551   0.11192469 -0.08969332  0.11081569  0.37215832
self(states) ->     -0.8427617   0.6415581   0.44988257 -0.43897176  0.8693738   0.40007943

Does this look as what would be expected for a single step?
The network with loss and optimizer:
    self.in_layer = Conv2d(channels, 32, 8)
    self.hidden_conv_1 = Conv2d(32, 64, 4)
    self.hidden_conv_2 = Conv2d(64, 128, 3)
    self.hidden_fc1 = Linear(128 * 78 * 78, 64)
    self.hidden_fc2 = Linear(64, 32)
    self.output = Linear(32, action_space)

    self.loss = torch.nn.MSELoss()
    self.optimizer = torch.optim.Adam(
        self.parameters(), lr=learning_rate) # lr is 0.001

def forward(self, state):
    in_out = fn.relu(self.in_layer(state))
    in_out = fn.relu(self.hidden_conv_1(in_out))
    in_out = fn.relu(self.hidden_conv_2(in_out))
    in_out = in_out.view(-1, 128 * 78 * 78)
    in_out = fn.relu(self.hidden_fc1(in_out))
    in_out = fn.relu(self.hidden_fc2(in_out))
    return self.output(in_out)

Then the learning block:
        self.optimizer.zero_grad()

        sample = self.sample(self.batch_size)
        states = torch.stack([i[0] for i in sample])
        actions = torch.tensor([i[1] for i in sample], device=device)
        rewards = torch.tensor([i[2] for i in sample], dtype=torch.float32, device=device)
        next_states = torch.stack([i[3] for i in sample])
        dones = torch.tensor([i[4] for i in sample], dtype=torch.uint8, device=device)

        current_q_vals = self(states)
        next_q_vals = self(next_states)
        q_target = current_q_vals.clone()
        q_target[torch.arange(states.size()[0]), actions] = rewards + (self.gamma * next_q_vals.max(dim=1)[0]) * (~dones).float()

        loss = fn.smooth_l1_loss(current_q_vals, q_target)
        loss.backward()

        self.optimizer.step()
```

In my experience, neural networks with convolutional layers take much much longer to train, so try increasing the number of iterations (time steps). After running, save the network model (I dont know how to do it in torch, but in tensorflow it was model.save("filename"+".h5") ).
Then, load this saved model file and do a test run to see if it worked. In this case, you should notice pretty easily if it learned or not).


Feature scaling strategy for many feature with very large variation between them? I was running into a situation in which my input feature experience a very large variation in term of magnitude.
Particularly, consider feature 1 belong to group 1 and feature 2 3 4 belong to group 2,
Like this picture below

I was really worried that in this case feature 1 might dominate feature 2,3,4 (group 2) because its corresponding value is so large (I was trying to train this data set on a neural network).
In this situation, what would be the appropriate scaling strategy ?
Update: I know for sure that the value of feature 1 is an integer that is uniform on the interval [22,42]
But for feature 2 ,3 ,4 I do not have any insight
Thank you for your enthusiast !

You should check the distribution of each feature and scale them accordingly, but in any case you should aim to roughly the same interval of values for every feature.
For example, if f1 has the standard distribution and f2 is close to the uniform one, then you can scale f1 to N(0,1) and f2 to U(-1,1).
In other words, try to have maximum, minimum and mean values for all features as close as possible, while keeping their original distributions.


Optimal mixed strategy in two player zero sum games I am currently studying game theory based on Peter Norvig's 3rd edition introduction to artificial intelligence book. In chapter 17.5, the two player zero sum game can be solved by using the $\textbf{minimax}$ theorem
$$max_x \, min_y \, x^TAy = min_x \, max_y \, x^TAy = v$$
where $x$ is  the probability distribution of actions by the max player (in the left equation) and the min player (in the right equation).
Regarding the minimax theorem, I have 2 questions.

Do both the min and the max players have the same probability distribution of actions ? In the book by Peter Norvig, the book demonstrated that in the game of $\textbf{Morra}$, both the min and max player had $[\frac{7}{12}:one, \frac{5}{12}:two]$ for the probability distributions.

Also, regarding minimax game tree, is the difference between minimax game tree and the zero sum game the fact that for minimax game tree, the opponent can react to the first player's move whereas for zero sum game defined in 17.5, both players are unaware of each other's move ?

I will answer the first question question based on information I have gathered so far. The probability of each action for the $\textbf{two player zero sum game}$ need not be the same for both players. It turns out that in the game of Morra, the probability vectors just turn out to be the same value.
In general to determine $\textbf{optimal mixed strategies}$ for two player two action game: Suppose we have 2 actions $a_1$ and $a_2$. In a mixed strategy, we let the probability of the row player taking action $a_1$ be $p$. The probability of the row player taking $a_2$ is then $1-p$. Now, we will try to find the probability $p$ such that the column player is indifferent to either action $a_1$ or $a_2$. (I.e the expected payoff for column player if row player plays $a_1$ with probability $p$ is the same for either actions).
Solving for $p$, we know that the column player can play any mixed strategy since any strategy played by the column player will yield the same payoff (since expected payoff is the same for either $a_1$ and $a_2$). Since column player can play any mixed strategy, we want to find a probability distribution over $a_1$ and $a_2$ for the column player such that the row player is indifferent to either action $a_1$ or $a_2$. We let the column player take action $a_1$ with probability $q$. It turns out we can repeat the same process for the column player and find the values of $p$. The game value for the row player can be computed from $x^TAy$ and this turns out to be the same value for the column player as well. (Only in zero sum games)
Doing this process yields a $\textbf{mixed strategy Nash Equilibrium}$, using such an approach does not always work however in the scenario when one action choice always dominates another action choice. (I.e in the case of the prisoner's dilemma of Alice and Bob, no matter what mixed strategy Bob tries to use, there will never be a mixed strategy such that Alice is indifferent to refusing or testifying). She will always testify.
I am not too sure about how to solve zero sum games involving more than 2 actions. I think linear programming would have to be used but i am not to familiar with how it can be applied.


Can AI research lead to new findings in general cognitive science? Can AI be used as a tool to investigate our minds?

To be more precise, what am I specifically asking for here are examples of discoveries on artificial intelligence (so algorithms, programs and computers that try to implement intelligent systems) that brought to light facts about intelligence and cognition in general. Have this ever happened? Is it frequent? How influential and important were these discoveries, if any?

A possible example of what I mean could be the PSSH, which states that a formal system is sufficient to simulate general intelligent behaviour. I believe that this is relevant to Cognitive Science in general because it entails our understanding of this phenomena. (Of course, this is just an hypotesis, but I believe that its importance in the AI debate makes it a really compelling result).

This is about hard AI and soft AI: proponents of hard AI work on systems that simulate the way human cognition works, with the eventual (hypothetical) goal of replicating it. This presupposes that you know how cognition works, and presumably you will learn about it as you attempt to replicate it.

Soft AI, on the other hand, tries to emulate the outcomes only. For example, Weizenbaum's ELIZA is clearly on this side, as it uses simple pattern matching, and does not 'understand' anything about the conversations it is having.

Obviously, we don't even know fully what it means to 'understand' something, and building working systems is not really possible with a hard approach. Hence, soft AI is more common, as researchers are usually measured by their outcomes rather than their ideas. As far as I am aware, the hard AI approach has been all but abandoned long ago.

As current AI seems to be dominated by statistical approaches, I doubt that we can find out many useful things about cognition this way.

One interesting side-note: it seems to me that the capabilities of modern AI systems have developed away from human capabilities. A three-year-old can do some things that a sophisticated AI system cannot do, but in some areas (chess, translation, ...) the capabilities of AI systems surpass what humans are capable of. Maybe imitation is indeed not the right way to approach AI.


Are neural networks really used apart from specific hi-tech organisations? This is a generic question. Still posting it to get insights from experts in the field.
I am interested in knowing if Neural Networks are used in general apart from specific hi-tech organizations.
If so, which type of NN is used in which industry and for what purpose?

Before putting up my experience I'd like to provide some facts about AI startups and established Hi-Tech companies.

Most of companies claiming to use AI doesn't actually use ML/DL
And there are products by some companies that are classified to be using AI since they just use linear regression

Coming to my experience I can say that AI is passively as well as actively utilized in product development and services. Though Neural Networks are the driving force behind the current AI tech, the thing that we study in the text books are just basic building blocks that are used to designing complex network architectures. Though most of High-Tech companies as you have mentioned might not be willing to share their secrets, the publications they make in conferences are good enough. I would suggest one to go through the papers that are published by companies like NVIDIA, Microsoft, and Google to get understanding on how complex networks are designed to address specific problems.


What is the difference between linear and non-linear regression? In machine learning, I understand that linear regression assumes that parameters or weights in equation should be linear. For Example:

$$y = w_1x_1 + w_2x_2$$

is a linear equation where $x_1$ and $x_2$ are feature variables and $w_1$ and $w_2$ are parameters.

Also

$$y = w_1(x_1)^2 + w_2(x_2)^2$$

is also linear as parameters $w_1$ and $w_2$ are linear with respect to $y$.

Now, I read some articles stating that in the equation like

$$y = \log(w_1)x_1 + \log(w_2)x_2$$

can also be made linear by considering other variables $v_1$ and $v_2$ as:

\begin{align}
v_1 &= \log(w_1)\\
v_2 &= \log(w_2)
\end{align}

Thus,

$$y = v_1x_1 + v_2x_2$$

So, in this sense, any non-linear equation can be made linear, then what is non-linear regression here? I think I am missing something important here. I am a beginner in the field of Machine Learning. Can somebody help me?

The difference is simply that non-linear regression learns parameters that in some way control the non-linearity - e.g. any weight or bias that is applied before a non-linear function.

For instance:

$$y = (w_1 x_1 + w_2 x_2)^2 + w_3$$

With such a function to learn, you cannot separate out transformed values of $w_1$ and $w_2$ and turn this into a linear function of just $x_1$ and $x_2$.

What you are describing as non-linearities in your examples are instead all applied by the machine learning engineer to create new candidate features for linear regression. This is not usually described as non-linear regression, but feature transformation or feature engineering.

There is also a kind of middle ground where a central linear algorithm e.g. linear regression, is trained on many variations of the original features, by automated generation and filtering of transformed features. The most general variants of this approach are not hugely popular because they suffer from same risks of overfitting as non-linear models whilst not offering much in the way of improved performance. However, if you narrow down the types of feature and transformation combinations based on some knowledge of how you expect the target function to behave, it leads to many useful variants of linear regression - e.g. regression on fourier transforms, radial basis functions etc.


What error should I use for RNN? I'm relatively new to machine learning, and I don't know what error I should use for an RNN.
I want to use a simple Elman RNN to predict the cases of Covid-19 there will be in a hospital for the next 15 days. I modeled this as a regression problem, treating the input like a bunch of dots in a graph to predict the tendency that the data is going to take (only show if there will be more cases or less).
With that bunch of dots I in fact refer to this:

Then I would treat this problem as a regression.
I actually don't have anything programmed yet. Firstly I want to write it all on a paper and then get down to work.  I am also considering focusing the problem to predict the actual plot of the time-series input, but right now I want to try the regression.
I've come to the conclusion that I can use these four different errors:

MSE
RMSE
Entropy
Cross-entropy

What are the different characteristics of these errors? Which to use? Where and when to use them?

To provide a good answer would fill several pages.  To keep it very simple try many different loss functions on your model.  Your goal is to have the highest performance based on some desired prediction metric (e.g., RMSE, MAE, MAPE, etc.).  You almost always have plenty of time to try many loss functions so you don't need to have a full understanding, and few people do, to start your project.
I recommend you read the following to learn more:

5 Regression Loss Functions All Machine Learners Should Know by Prince Grover
Loss functions: Why, what, where or when? by Phuc Truong
Loss Functions in Deep Learning: An Overview


Can non-sequential deep learning models outperform sequential models in time series forecasting? Can a CNN (or other non-sequential deep learning models) outperform LSTM (or other sequential models) in time series data?

I know this question is not very specific, but I experienced this when predicting daily store sales and I am curious as to why it can happen.

You are right CNN based models can outperform RNN. You can take a look at this paper where they compared different RNN models with TCN (temporal convolutional networks) on different sequence modeling tasks. Even though there are no big differences in terms of results there are some nice properties that CNN based models offers such as: parallelism, stable gradients and low training memory footprint. In addition to CNN based models there are also attention based models (you might want to take a look at the transformer)


What is meant by "the number of examples is reduced", and why is this the case? I am currently studying the paper Learning and Evaluating Classifiers under Sample Selection Bias by Bianca Zadrozny. In section 3.2. Logistic Regression, the author says the following:

3.2. Logistic regression
In logistic regression, we use maximum likelihood to find the parameter vector $\beta$ of the following model:
$$P(y = 1 \mid x) = \dfrac{1}{1 + \exp(\beta_0 + \beta_1 x_1 + \dots + \beta_n x_n)}$$
With sample selection bias we will instead fit:
$$P(y = 1 \mid x, s = 1) = \dfrac{1}{1 + \exp(\beta_0 + \beta_1 x_1 + \dots + \beta_n x_n)}$$
However, because we are assuming that $y$ is independent of $s$ given $x$ we have that $P(y = 1 \mid x, s = 1) = P(y = 1 \mid x)$. Thus, logistic regression is not affected by sample selection bias, except for the fact that the number of examples is reduced. Asymptotically, as long as $P(s = 1 \mid x)$ is greater than zero for all $x$, the results on a selected sample approach the results on a random sample. In fact, this is true for any learner that models $P(y \mid x)$ directly. These are all local learners.

This part is unclear to me:

However, because we are assuming that $y$ is independent of $s$ given $x$ we have that $P(y = 1 \mid x, s = 1) = P(y = 1 \mid x)$. Thus, logistic regression is not affected by sample selection bias, except for the fact that the number of examples is reduced.

What is meant by "the number of examples is reduced", and why is this the case?

I think what he means is that, while distribution $P(y  \mid x,s)$ has three variables, $P(y  \mid x)$ has two variables. The number of parameters required to describe a distribution (or samples to approximate it) grows exponentially with the number of variables (for more info see for instance Ian goodfellow, .. deep learning").


How can I design the input layer of a feed-forward neural network to be trained with a medical dataset with three features? I am building a feed-forward neural network with two hidden layers, which I will train with a medical dataset, which consists of both data, such as age and sex, and images of x-ray scans ($1024 \times 1024$). The labels are types of cancer.

I believe that ages and sex will affect the output of the network. But including the images will make the network biased towards the images, because images will occupy most of the input layer.

How can I design the input layer of the network?

I would give some advice here:

You should probably use Convolutional Neural Network for image data or you'll end up with too many input features, and too much computation. What are your constraints on this point?

I would not mix image data with numerical data. Maybe you can implement two neural networks, one handling image data and another managing the numerical data, each one outputting their results. A second step would be to use their results to handcraft a global result or let another NN manage this.


Why would neural networks be a particularly good framework for "embodied AI"? There is this claim around that the brain's cognitive capabilities are tightly linked to the way it processes sensorimotor information and that, in this or a similar sense, our intelligence is "embodied". Lets assume, for the sake of argument, that this claim is correct (you may think the claim is too vague to even qualify for being correct, that it's "not even false". If so, I would love to hear your ways of fleshing out the claim in such a way that it's specific enough to be true or false).
Then, since arguably at least chronologically in our evolution, most of our higher-level cognitive capabilities come after our brain's way of processing sensorimotor information, this brings up the question: what is it about the way that our brains function that make them particularly suitable for the processing of sensorimotor information? What makes our brains'  architecture particularly suitable for being an information processing unit inside a body?
This is my first question. And what I'm hoping for are answers that go beyond the a fortiori reply "Our brain is so powerful and dynamic, it's great for any task, and so also for processing sensorimotor information".
My second question is basically the same, but, instead of the human brain, I want to ask for neural networks. What are the properties of neural networks that make them particularly suitable for processing the kind of information that is produced by a body?
Here are some of the reasons why people think neural networks are powerful:

The universal approximation theorem (of FFNNs)
Their ability to learn and self-organise
Robustness to local degrading of information
Their ability to abstract/coarse-grain/convolute features, etc.

While I see how these are real advantages when it comes to evolution picking its favorite model for an embodied AI, none of them (or their combination) seems to be unique to neural networks. So, they don't provide a satisfactory answer to my question.
What makes a neural network a more suitable structure for embodied AI than, say, having a literal Turing machine sitting inside our head, or any other structure that is capable of universal computation?
For instance, I really don't see how neural networks would be a particularly natural choice for dealing with geometric information. But geometric information is pretty vital when it comes to sensorimotor information, no?

To my mind the essential reason why neural networks and the brain are powerful is that they create a hierarchical model of data or of the world. If you ask why that makes them powerful, well, that's just the structure of the world. If you are stalked by a wolf, it's not like its upper jaw will attack you frontally, while his lower jaw will attack you from behind. If you want to respond to the threat with a feasible computational effort, you'll have to treat the wolf as one entity. Providing these kinds of entities or concepts from the raw bits and bytes of input is what a hierarchical representation does.

Now, this is quite intuitive for sensory information: lashes, iris, eyebrow make up an eye, eyes, nose and mouth make up a face and so on. What is less obvious, is the fact that motor control works exactly the same way! Only in reverse. If you want to lift your arm, you'll just lift it. But for your brain to actually realise this move, the high level command has to be broken down into precise signals for every muscle involved. And this is done by propagating the command down the hierarchy.

In the brain these two functions are strongly intertwined. You use constant sensory feedback to adapt your motor control and in many cases you'd be incapable of integrating your stream of sensory data into a coherent representation if you didn't have the additional information of what your body is doing to change that stream of data. Saccades are a good example for that.

Of course this doesn't mean that our cognitive functions are dependent on the processing of sensorimotor information. I would be surprised if a pure thinking machine wouldn't be possible. There is however a specific version of this "embodied intelligence hypothesis" that sounds plausible to me:

Creating high level cognitive concepts with unsupervised learning is a really difficult problem. Creating high level motor representation might be significantly easier. The reason is that there is more immediate useful feedback. I have been thinking about how to provide a scaffolding for the learning of a hierarchy of cognitive concepts and one thing I could imagine is that high level cognitive concepts basically hitch a ride with the motor concepts. Just think of what a pantomime can express with movement alone.


Why Feature Scaling for skewed contour? Why is it that the skewed contour (unscaled features) will result in slow performance of gradient descent? In other words, how (or why) will the gradients end up taking a long time before finding the global minimum in such cases? This might be an obvious question but I'm finding a hard time Visualizing the 3D shapes of the respective contours and relating it to the convergence.

Left one is the contour for the unscaled feature and the right one is scaled (and will apparently converge quickly).

Feature scaling is making your features have an equal representation or say in the final loss of the function. Intuitively as your picture suggests the contour will be elongated along one axes (feature with higher values).

Another example is lets say you have to separate two data clusters, one is  (x,2) and another is (x,5) where x is variable. Now if you are using sigmoid activation, sigmoid always gives positive values, but to separate the two classes you need to have negative value for (x,2) in the second last layer, thus the NN will for a longer time have to adjust the weights to be negative (assuming positive weight initialisation) of bias and other features until (x,2) gives negative value in the second last layer. But with feature normalisation the initial value would have been already negative something like (x,-1/3).

EDIT: From my answer on DataScience.SE:

Purpose: Normalisation is done so that the Neural Network weights converge faster. In CNN's and Deep Neural Nets this is of particular help especially in CNN this helps to prevent exploding/vanishing gradients.

The most common explanation for normalisation I have come across is that if you have 2 features, one of them has a significant larger scale than the other e.g house price and house area then the feature with a larger scale will dominate the output. This is quite incorrect according to me, since when you back-propagate through the Neural Net the weight updates are directly proportional to the activations, so larger activation means larger feedback and hence weights get reduced faster and become smaller until w1*house price = w2*house area approximately this relation holds true. Yes, it will lead to more oscillations (intuitionally, since learning rate also becomes multiplied with a larger scale) but it will ultimately probably converge.

So the best 3 reasons for using normalisation are:

If scale of a feature is large the weights connected to that feature will have larger oscillations resulting in slower convergence and if deep NN is used probably no convergence, whereas normalisation helps in making the values small -1 to 1 so the gradient updates are also small resulting in faster convergence.
The best intuition for normalisation can be found from this video of Stanford and its subsequent video. Since we know weight updates is directly proportional to inputs it will also take the sign of the inputs (or exact opposite sign) always. Now we know house price and area is always positive (in our Universe at least!). So the weight updates will always have definite sign either both positive or both negative (depending the sign of downstream gradients). But the weight updates may have a optimal value in the 4th quadrant, so the weight updates will have to follow a zig zag pattern to effectively make a weight update in the 4th quadrant.

Finally when you are dealing with Deep Neural Nets like CNN if you do not normalise pixels it will result in exponentially large/vanishing gradients. Since generally softmax/sigmoid is used in the last layer, it squashes the outputs. If you have a large output, generally due to un-normalized data, it'll result in wither exact 0 or exact 1 output, which is fed into a log function and BAM! overflow. The error becomes inf or NaN in python. So inf error means exploding gradients and NaN means gradient cannot be calculated. This can probably be remedied by using higher floating point precision but it will result in higher memory and processor consumption ultimately inefficiency.

TL;DR: Normalisation is used for faster weight convergence. Issues faced by un-normalised data are larger weight oscillations, weight updation in non optimal direction, overflow in precision in Deep Neural Nets.

Here are a few great links for better insight:

How and why do normalization and feature scaling work?

Why feature scaling?


Advantage computed the wrong way? Here is the code written by Maxim Lapan. I am reading his book (Deep Reinforcement Learning Hands-on). I have seen a line in his code which is really weird. In the accumulation of the policy gradient $$\partial \theta_{\pi} \gets \partial \theta_{\pi} + \nabla_{\theta}\log\pi_{\theta} (a_i | s_i) (R - V_{\theta}(s_i))$$ we have to compute the advantage $R - V_{\theta}(s_i)$. In line 138, maxim uses adv_v = vals_ref_v - value_v.detach(). Visually, it looks fine, but look at the shape of each term.

ipdb> adv_v.shape
torch.Size([128, 128])

ipdb> vals_ref_v.shape
torch.Size([128])

ipdb> values_v.detach().shape
torch.Size([128, 1])

In a much simpler code, it is equivalent to

In [1]: import torch

In [2]: t1 = torch.tensor([1, 2, 3])

In [3]: t2 = torch.tensor([[4], [5], [6]])

In [4]: t1 - t2
Out[4]:
tensor([[-3, -2, -1],
        [-4, -3, -2],
        [-5, -4, -3]])

In [5]: t1 - t2.detach()
Out[5]:
tensor([[-3, -2, -1],
        [-4, -3, -2],
        [-5, -4, -3]])

I have trained the agent with his code and it works perfectly fine. I am very confused why it is good practice and what it is doing. Could someone enlighten me on the line adv_v = vals_ref_v - value_v.detach()? For me, the right thing to do was adv_v = vals_ref_v - value_v.squeeze(-1).

Here is the full algorithm used in his book :

UPDATE

As you can see by the image, it is converging even though adv_v = vals_ref_v - value_v.detach() looks wrongly implemented. It is not done yet, but I will update the question later.

Yeah, it seems like it's a wrong implementation.
vals_ref_v is a matrix of 1 row, and 128 columns.
value_v.detach() is a matrix of 128 row


Do we ever need more then 1 hidden layer in a binary classification problem with ANNs? If yes why? I have read about the universal approximation theorem. So, why do we need more than 1 layer? Is it somehow computationally efficient to add layers instead of more neurons in the hidden layer?

This is akin to asking "Why do we need more than one instance of sine to represent any repeating function" or "why can't we represent any polynomial with an equivalent polynomial of just the first degree?"  There are many, many problems... I'd even want to say most... that will require more than one layer to solve because the higher dimensional relationships cannot be well represented by just one layer.  This is not to say that the theorem is wrong, but consider the applied aspects.  We can approximate any continuous function, but that might require a single layer that is infinitely wide, however that same function might be approximated by a deep network having only a few dozen neurons.
However, this is not to say that many networks could not be represented with networks that perform at least as well, or perhaps even better, by simpler networks of fewer layers/neurons.  There is active research into how to generalize this.
Ultimately, non-trivial problems often require an empirical approach in this space currently because there is no general solution to "learning."


Who first coined the term Artificial Intelligence? Who first coined the term Artificial Intelligence? Is there a published research paper that first used that term?

John McCarthy (1927 - 2011) was an American computer scientist. A pioneer in the foundations of artificial intelligence research, he coined the term "artificial intelligence". He was one of the creators of the (original) Lisp programming language, which was quite involved in early AI research in the 1960s and 1970s.

He coined the term in 1955, and organized the first Artificial Intelligence conference in 1956, while working as a math teacher at Dartmouth. He founded the AI labs at MIT and Stanford.

He's responsible for developing several other important concepts in today's mainstream computer science. Namely, he developed garbage collection (used by a Lisp interpreter) and designed the first time-sharing systems.

On a side note, McCarthy predicted that creating a truly intelligent machine would require 1.8 Einsteins and one-tenth the resources of the Manhattan Project


AI techniques and competencies necessary for creating a Virtual Assistant I want to develop a virtual assistant like Google Assistant, Alexa, etc., for mobile phones.

What I should be researching apart from M/C Learning and Python?

First of all, making app for mobile is not normally written in Python, but if you want it, that is possible [1].

Other non ai question is that the counting of speech recognition is so heavy, that at least Alexa etc make it on external server. Wake word recognition is done in machine because of real time needs [3]. Ofcourse your app can trigger from mobile touch event too so speech recognition implementation on mobile can be avoided.

Using server needs knowledge about communication and web apis as well as server maintenance in cloud.

Speech recognition can be made with open source tools, but the dictionaries with enough samples may not yet exist or they're coming [2]. Also Google offers speech api, that needs mainly programming knowledge to attach speech recognition.

For questions you also need to generate answers. To form that you may reverse the flow in speech generation [4]. All the logic to find useful answers needs some more effort depending how intelligent your assistant will be. That maybe is the most challenging part where direct answer do not exist.

Of course, if you could attach to ready assistant's public api, it would be magnificent. Actually, Alexa api at least is open [5]!! Playing with it directly from mobile makes app soon ready but if you want to learn more algorihms, some pointers are on my above other points.

Sources:

[1] https://www.quora.com/Can-I-make-an-Android-app-with-Python

[2] https://research.mozilla.org/machine-learning/

[3] https://iot.stackexchange.com/questions/736/why-do-most-smart-assistants-offer-little-if-any-customisation-of-the-wake-wor

[4] How could you generate sentences from lists of facts

[5] https://github.com/alexa/alexa-avs-sample-app


Are there real applications of fuzzy logic? This question covers in detail, what fuzzy logic is and how it relates to other math fields, such as boolean algebra and sets theory.

This question is also very related, but the answers are focused more on general intuition and potential applicability. The only working system based on fuzzy logic, mentioned there, is MYCIN, which goes back to the early 70s. This quote from wiki summarizes my impression of it:

  MYCIN was never actually used in practice.

From my experience in AI, the best tool to deal with uncertainty is Bayesian probability and inference. It allows to apply not only a wide range of probabilistic tools, such as expectation, MLE, cross-entropy, etc, but also calculus and algebra.

Can you call fuzzy logic a "pure theoretical" concept, which only played its role in the early development of AI? Are there real practical applications of fuzzy logic? What problem would you recommend to solve and to code using fuzzy logic?

You've obviously never heard of fuzzy logic washing machines.

  ● Typically, fuzzy logic controls the washing process, water intake,water temperature, wash time, rinse performance, and spin speed. This optimises the life span of the washing machine. More sophisticated machines weigh the load (so you can’t overload the washing machine), advise on the required amount of detergent, assess cloth material type and water hardness, and check whether the detergent is in powder or liquid form. Some machines even learn from past experience,memorising programs and adjusting them to minimise running costs.

Fuzzy logic is used in a variety of control applications.  If your furnace can only be on or off, for example, you might use a probabilistic function of temperature to determine when to turn it on and off, rather than having fixed high and low temperatures activate your thermostat.  In some applications, that's been found to improve perceived comfort or efficiency.

For more sophisticated AI applications, you could use fuzzy logic for activations in a neural net, but I don't think it's offering much improvement over fixed, weighted activations.


How can I identify bigrams and trigrams that represent concepts? I have many text documents and I want to identify concepts in these documents in an unsupervised manner. One of my problems is that the concepts can be bigrams, trigrams, or even longer.
So, for example, out of all the bigrams, how can I identify the ones that are more likely to represent a concept?
A concept could be "machine learning".
Are you aware of any standard approaches to solve this problem?
Edit: The corpus I am working with consists of papers accessed from web of science. That is, they are all in some given domain niche. I want to extract words, bigrams, trigrams... that represent common concepts/buzzwords from these papers. These could be "Automated machine learning", "natural language processing" et cetera. I need to be able to distinguish these from other common n-grams such as "New York", "Barack Obama",...
I know that I could do this using a NER approach but this would require hand-labelling. Are you aware of any unsupervised ways to approach this problem? Or even a semi-superised method with little labelled data?

One of the renowned methods is using the TF-IDF method. You can define a lower bound for the TF-IDF score of the bigram or trigram to identify it as a keyword. Here, you can find more information about it.