What does AI software look like, and how is it different from other software? What does AI software look like? What is the major difference between AI software and other software?

Code in AI is not in principle different from any other computer code. After all, you encode algorithms in a way that computers can process them. Having said that, there are a few points where your typical "AI Code" might be different:

A lot of (especially early) AI code was more research based and exploratory, so certain programming languages were favoured that were not mainstream for, say, business applications. For example, much work in early AI has been coded in Lisp, and probably not much in Fortran or Cobol, which were more suited to engineering or business. Special languages were developed to make it easy to program with symbols and logic (eg Prolog).

The emphasis was more on algorithms than clever/complex programming. If you look at the source code for ELIZA (there are multiple implementations in many different languages), it's really very simple.

Before the advent of neural networks and (statistical) machine learning, most AI programming was symbolic, so there hasn't been much emphasis on numerical computing. This changed as probabilities and fuzziness were increasingly used, but even if using general purpose languages there would be fewer numerical calculations.

Self-modifying code is inherently complex; while eg Lisp made no difference between code and data (at least not in the same way as eg C or Pascal), this would just complicate development without much gain. Perhaps in the early days this was necessary when computers had precious little memory and power and you had to work around those constraints. But these days I don't think anybody would use such techniques anymore.

As modern programming languages evolved, Lisp and Prolog (which were the dominant AI languages until probably 20 to 30 years ago) have been slowly replaced by eg Python; probably because it is easier to find programmers comfortable in an imperative paradigm rather than a functional one. In general, interpreted languages would be preferred over compiled ones due to speed of development, unless performance is important.

The move to deep learning has of course shifted this a lot. Now the core processing is all numeric, so you would want languages that are better with calculations than symbol handling. Interpreted languages would now mainly make up the 'glue' code to interface between compiled modules, and be used for data pre-processing. So current AI code is probably not really that different from code used in scientific computing these days.
There is of course still a difference between R&D and production code. You might explore a subject using an interpreted language, and then re-code your algorithm for production in a compiled language to gain better performance. This depends on how established the area is; there will for example be ready-made libraries available for neural networks or genetic algorithms which are well-established algorithms (where performance matters).
In conclusion: I don't think AI code is any more complex than any other code. Of course, that's not very exciting to portray in a film, so artistic licence is used to make it more interesting. I guess self-modifying code also enables the machines to develop their own conscience and take over the world, which is even more gripping as a story element. However, given that a lot of behaviour is nowadays in the (training/model/configuration) data rather than the algorithm, this might even be more straight forward to modify.
Note: this is a fairly simplified summary based on my own experience of working in AI; other people's views might vary, without either being 'wrong'.
Update 2021: I now work at a company that extracts business information/events from news data on a large scale using NLP methods. And we're using Lisp... so it's still in active, commercial use in AI.


Must people tell an AI which algorithm it should use? Can an AI learn algorithms by itself? I'm a freshman to machine learning. We all know that there are 2 kinds of problems in our life: problems that humans can solve and problems we can't solve. For problems humans can solve, we always try our best to write some algorithm and tell machine to follow it step by step, and finally the machine acts like people.

What I'm curious about are these problems humans can't solve. If humans ourselves can't sum up and get an algorithm (which means that we ourselves don't know how to solve the problem), can a machine solve the problem? That is, can the machine sum up and get an algorithm by itself based on a large amount of problem data?

There are problems we for which we don't have a known, optimal, deterministic algorithm. By and large we use heuristics to "solve" those problems.  A closely related idea is that of satisficing where we seek out answers that are "good enough" for immediate purposes.

Likewise, machines can also use heuristics, whether they are programmed in explicitly or, presumably, learned.  Within the range of ways that a machine can use heuristics, there are meta heuristics and hyper heuristics.

Going beyond that, there are other ways that machines an learn "algorithms" or "rules" for solving problems.  One are that I'm particularly interested in is known as rule induction.

This is all an area of open and active research BTW... so if you're interested in exploring any of these approaches, you'll probably find a lot of ground to cover.


How to fill missing values in a dataset where some properties can be inputs and outputs? I have a dataset with missing values, I would like to use machine learning methods to fill.
In more detail, there are $n$
individuals, for which up to 10 properties are provided, all numerical. The fact is, there are no individuals for which all properties are given. The first rows (each row contains data for a given individual) do look as the following

\begin{bmatrix}
    1 & NA & 3.6 & 12.1 & NA \\
1.2 & NA & NA & 4 & NA \\
    NA & 4 & 5 & NA & 7
  \end{bmatrix}

What methods could be applicable in general?

I have some basic experience in classifiers and Random Forests. Modulo the obvious difference that this is not a classifying problem, what I struggle most with is that the same variable (described in the e.g
$n$-th column) is both an input and an output. Say I want to predict the value $A_{2,3}$ in the dataset above. In this case, all the values in the third column could be used as input, excluded of course $A_{2,3}$
itself, which would be an output.

This seems to be different than the more conventional set-up of predicting a property, given a set of other properties (e.g, predict income given education, work sector, seniority, etc.). In this case, sometimes the income is to be predicted, sometimes used for predicting another variable.
I am aware of methods which, given a vector $X_i$, could approximate a function $F$ and predict responses $Y_i$ with

$$ Y_i = F(X_i)$$

In the scenario I described though, it looks like some implicit function $\Phi$ is to be found, a function of all the variables $Z_i$ (columns in the dataset above)

$$ \Phi (Z_i) = 0$$

What methods could handle this aspect? I understand the question is probably too general, but I could not find much and could do with a starting point. I would be already content with some hints for my further reading, but anything more would be gratefully welcomed, thanks.

As you mentioned in the comments about a possible problem of using mean, median type of imputations naively could lead to wrong predictions. In such cases, you need to first check whether you have enough data.

If you have enough data

You can try using MICE (Multivariate Imputation By Chained Equations) algorithm on your missing data. The method is based on Fully Conditional Specification, where each incomplete variable is imputed by a separate model. The MICE algorithm can impute mixes of continuous, binary, unordered categorical and ordered categorical data. One note of caution with this method: It is a computationally expensive method, so it use it if you are not short on time.

The important thing to keep in mind is that in order to tackle such problems, you would be needing multiple iterations as a part of your algorithm. The conventional setup you are describing, does not seem to iterative in nature and hence you are running into the problem of features being input and output at the same time.

If by any chance you insist on finding the missing values just for solving a downstream task like classification or regression, you can try the XgBoost algorithm. It can be used as a classifier or as a regressor. This algorithm can handle missing values inherently. Source: this answer

If you don't have enough data

In such a case you would need to introduce bias in your model using your insights or domain knowledge about the problem. For, e.g., in the a possible problem of estimating weights using heights, you had an insight that your data comprises more of short people. So instead of naively using median values of the total dataset for weight, you can try to bin the data according to their heights, say 'S', 'M', 'L', 'XL', and estimate the weights of each bin separately using the median values of their respective bins. The thing to keep in mind is that when data is low, you need to provide knowledge to the model by enforcing some bias using your insights and domain knowledge about the problem.


Understanding the application of Sobel kernel followed by ReLU to a zero-padded image Let's say I have a $2 \times 2$ pixel of grayscale picture, where there is one edge such that the left pixel contains a value, 30, and the right pixels contain a value 0 (in red below). And for edge detection I have zero-padded the input image and then used the Sobel vertical filter to find out the vertical edges and apply ReLU to the output. The output is a $2 \times 2$ matrix with all pixel values $0$. So that should mean there is no edge in the picture whereas in actual case it has one. Where am I going wrong?

I assume that you would like to use convolution with padding with the same size of output matrix as this picture. You messed up calculations with convolution with "full padding". If we imagine these two matrices as windows that slide on each other, you can see that the filter is symmetrically inverted. I used a little bit different filter to show you in a better way how it works (I changed the last row to [-3, 0 3]).

Assuming these matrices:

You should add to your picture matrix two rows and columns of zeros:

Then you can start matrix multiplication, but notice that the filter is symmetrically inverted. The result shown as matrix $4 \times 4$ is the convolution with "full padding". The $2 \times 2$ matrix in the middle is a result of convolution with the "same padding", that you requested.

Next step:

Some iterations later:

And later:

And finally:

For convolution with the same size, the result will be the small $2 \times 2$ matrix in the middle.

After using the ReLU function, the result will be exactly the same.

So with using of your filter, the result would look like [[0,90], [0,90]].


What are the main differences between skip-gram and continuous bag of words? The skip-gram and continuous bag of words (CBOW) are two different types of word2vec model.

What are the main differences between them? What are the pros and cons of both methods?

So as you're probably already aware of, CBOW and Skip-gram are just mirrored versions of each other. CBOW is trained to predict a single word from a fixed window size of context words, whereas Skip-gram does the opposite, and tries to predict several context words from a single input word.

Intuitively, the first task is much simpler, this implies a much faster convergence for CBOW than for Skip-gram, in the original paper (link below) they wrote that CBOW took hours to train, Skip-gram 3 days.

For the same logic regarding the task difficulty, CBOW learn better syntactic relationships between words while Skip-gram is better in capturing better semantic relationships. In practice, this means that for the word 'cat' CBOW would retrive as closest vectors morphologically similar words like plurals, i.e. 'cats' while Skip-gram would consider morphologically different words (but semantically relevant) like 'dog' much closer to 'cat' in comparison.

A final consideration to make deals instead with the sensitivity to rare and frequent words. Because Skip-gram rely on single words input, it is less sensitive to overfit frequent words, because even if frequent words are presented more times that rare words during training, they still appear individually, while CBOW is prone to overfit frequent words because they appear several time along with the same context. This advantage over frequent words overfitting leads Skip-gram to be also more efficient in term of documents required to achieve good performances, much less than CBOW (and it's also the reason of the better performances of Skip-gram in capturing semantical relationships).

Anyway, you can find some comparisons in the original paper (section 4.3).
Mikolov et al. 2013

About the architecture, there's not much to say. They just randomly initialised word embedding for each word, then a projection matrix NxD (number of context words times embeddings dimension) is generated at each iteration, there is no hidden layer, the vectors are just averaged together and then fed into an activation function to predict index probabilities in a vector of dimension V (the size of the vocabulary).

For a more specific explanation (even Mikolov's paper lack some detail) I suggest checking this blog page (Words as Vectors), even though the model described there do apply a hidden layer, unlike the original architecture.


Structured lighting basic principles for depth mapping I've been wondering, how, in the most simple-to-implement basic principle, does the light projection to depth map technique described here https://www.lightform.com/how-it-works actually functions? Is it some kind of an average based on the color of x pixel over all the patterns or what? How difficult would it be to code something that could do this, up?

how, in the most simple-to-implement basic principle, does the light projection to depth map technique described here https://www.lightform.com/how-it-works actually functions?Is it some kind of an average based on the color of x pixel over all the patterns or what?

No it is not that simple , more details on that are described in https://kevinkarsch.com/publications/sa11-lowres.pdf

P.S the author of this paper went on to become CTO of lightform.

Is it some kind of an average based on the color of x pixel over all the patterns or what? How difficult would it be to code something that could do this, up?

As mentioned above it is not simple average based on color , coding would be a tough task , you could implement model if you are capable enough to code a ray-tracer from scratch. But replicating the exact results would require a lot of low-level knowledge and certain domain expertise as not much of the implementation details and hyperparameters are described.


Which meta-learning approach selection methodology should I use for similarity learning of an image? Meta-learning has 3 broad approaches: model, metric and optimization-based approach. Each of them has its own sub-approach, like matching network, meta-agonistic and Siamese-based network, and so on.
How do I decide which approach to select for a task? For my case, I have a noisy image, and they need to be compared with 10 different new images every time. Do I have to start with the trial and error method, or there is some methodology behind this approach selection?

The only principled approach is to see if the problem you are working has been worked on before. If yes, then, use what they used for your problem. It is the alternative to go through all the models one-by-one.
Like, in your case, it bears a semblance with face models, so, Siamese network with ArcFace loss for getting your embeddings is a good bet for you.


Are there any rules of thumb for having some idea of what capacity a neural network needs to have for a given problem? To give an example. Let's just consider the MNIST dataset of handwritten digits. Here are some things which might have an impact on the optimum model capacity:

There are 10 output classes
The inputs are 28x28 grayscale pixels (I think this indirectly affects the model capacity. eg: if the inputs were 5x5 pixels, there wouldn't be much room for varying the way an 8 looks)

So, is there any way of knowing what the model capacity ought to be? Even if it's not exact? Even if it's a qualitative understanding of the type "if X goes up, then Y goes down"?
Just to accentuate what I mean when I say "not exact": I can already tell that a 100 variable model won't solve MNIST, so at least I have a lower bound. I'm also pretty sure that a 1,000,000,000 variable model is way more than needed. Of course, knowing a smaller range than that would be much more useful!

This may sound counter intuitive but one of the biggest rules of thumb for model capacity in deep learning:

IT SHOULD OVERFIT.

Once you get a model to overfit, its easier to experiment with regularizations, module replacements, etc. But in general, it gives you a good starting ground.


How is it possible to get the output size of `n` Consecutive Convolutional layers? Given network architecture, what are the possible ways to define fully connected layer fc1 to have a generalized structure such as nn.Linear($size_of_previous_layer$, 50)?
The main issue arising is due to x = F.relu(self.fc1(x)) in the forward function. After using the flatten, I need to incorporate numerous dense layers. But to my understanding, self.fc1 must be initialized and hence, needs a size (to be calculated from previous layers). How can I declare the self.fc1 layer in a generalized manner?
My Thought:
To get the size, I can calculate the size of the outputs from each of Convolution layer, and since I have just 3, it is feasible. But, in case of n layers, how can you get the output size from the final convolutional layer?
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(3, 10, kernel_size=3, padding = 1)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=3, padding = 1)
        self.conv2_drop = nn.Dropout2d(0.4)
        self.conv3 = nn.Conv2d(20, 40, kernel_size=3, padding = 1)
        self.conv3_drop = nn.Dropout2d(0.4)

        self.fc1 = nn.Linear(360, 50)  # self.fc1 = nn.Linear($size_of_previous_layer$, 50)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = F.relu(F.max_pool2d(self.conv3_drop(self.conv3(x)), 2))

        x = x.flatten(1)

        x = F.relu(self.fc1(x))
        return F.log_softmax(x)

Input to the following architecture can assumed to be [3, 32, 32] (num_of_channels, height, width).
PS:

Answers are expected in PyTorch.
For single convolutional layer, it is quite easy. The question refers to solve for n consecutive convolutional layers.

The question was also asked in Pytorch Forums, where it has been sort of resolved and would love to have further discussions over there.
Link to Pytorch thread: https://discuss.pytorch.org/t/how-is-it-possible-to-get-the-output-size-of-n-consecutive-convolutional-layers/87300


Indoor positioning with variable number of distance measurements in tensorflow Currently I have a setup where I'm determining the position of a transmitter using the RSSI of 4 receivers. Its a simple feed-forward network with some hidden layers, where the input is the RSSI values, and the output is a 2d coordinate.

Now, if I decide to add/remove receivers, I have to train the network again, since the input size changes. This is not ideal, since the receivers can move around, dissapear, etc. I have looked at some alternatives, but being pretty new to machine learning, it's difficult to pick which direction to go.

I have looked at a potential solution (stolen from another question), but I'm lost at how to implement it using tensorflow:

Any help is appreciated.

The solution mentioned seems feasible but you'd probably encounter a lot of problems, such as -

Since you're outputting coordinates, each one of the input networks must be trained differently. Considering $N$ should ideally be a variable, how many input networks do you train?
You'd use an intermediate encoding of sorts as co-ordinate representation which would be averaged and then passed through the output network. - Interpolation properties of the encoding might not be too great, meaning a linear change in the representation might not lead to a linear change in the co-ordinate, causing an averaging function to give skewed results.

Just a suggestion but I think it would be better if instead of learning multiple estimates of co-ordinate representations and then averaging them it might be better if you tried to learn some sort of distance function.

Essentially, each receiver would have a fixed position (Assumption) and the input network (or two input networks preferably) would output the distance  ($r$) of the transmitter from that receiver and the angle ($\theta$) it makes with a common axis. The problem with this approach is you wouldn't be leveraging multiple receivers as they would not be learning together, and the advantage being that whatever $N$ is, the RSSI readings pass through the same network always.

Once an $N$ number of $(r, \theta)$ values are obtained - a robust algorithm
$f(r_1, r_2,..., \theta_1, \theta_2, ...) = (x_t, y_t)$ could be found to output the transmitter co-ordinates.

I'm not too confident if it would work too well but just a suggested direction I could think of! Hope this helped!


How does a software license apply to pretrained models? Google provides a lot of pretrained tensorflow models, but I cannot find a license.

I am interested in the tfjs-models. The code is licensed Apache-2.0, but the models are downloaded by the code, so the license of the repository probably does not apply to the models and I am not able to find anywhere a note about the license of the pretrained models.

How should I handle this, especially when I may want to distribute models derived from the pretrained Google models?

I am not sure if a pretrained machine learning model is actually protected by copyrights or not. Copyright protection exists to protect the creators of creative works from having their work "stolen", and I am not sure if training a ML model is an act of creativity.

That said, assuming that a pretrained ML model is actually protected by copyrights, then it is more likely that the model is a derived work of the data set used for training than that it is a derived work of the software that uses the model.

The software reads the model in as data, assuming that the software can be used with many differently trained models. In that case, the software and the model are considered completely independent works in the same way that MS Word and the documents you write with it are independent works for copyright.

Thus, if you want to publish the trained model with a license, I would recommend to use the BSD license that was also used for the training set.


How to avoid running out of solutions in genetic algorithm due to selection? The genetic algorithm consists of 5 phases of which 4 are repeated:

Initial population (initially)
Fitness function
Selection
Crossover
Mutation

In the selection phase, the number of solutions decreases. How is it avoided to run out of the population before reaching a suitable solution?

There are multiple ways to interpret those steps. The most common standard approaches are

select two parents and produce two offspring; repeat until child population is the same size as parent population, and let the children replace their parents unconditionally (generational GA)

same as the above, but allow a few parents to live on instead of a few children if the parents have higher fitness (elitism)

each iteration, select two parents, produce one child, let the child replace a member of the parent population if it is better (steady state GA)

But there are other ways to go. There's an algorithm called CHC that lets the child population get smaller over time, and when it reaches zero, the algorithm triggers a smart restart. The point is there's no single definition for what makes an evolutionary algorithm. It's up to you to decide how to make something that works well for your problem. When you're a beginner though, it's handy to start from known points, like the three I mentioned above.


How does the Kullback-Leibler divergence give "knowledge gained"? I'm reading about the KL divergence on Wikipedia. I don't understand how the equation gives "information gained" as it says in the "Interpretations" section

Expressed in the language of Bayesian inference, ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ is a measure of the information gained by revising one's beliefs from the prior probability distribution $Q$ to the posterior probability distribution $P$

I was under the impression that KL divergence is a way of measure the difference between distributions (used in autoencoders to determine the difference between the input and the output generated from latents).
How does the equation $$D_{KL}(P \| Q)=\sum P(x) \log \left( \frac{P(X)}{Q(X)} \right)$$ give us a divergence? Also, in encoding and decoding algorithms that use KL divergence, is the goal to minimize $D_{KL}(P \| Q)$?

You can know it better, if you know the concept of entropy:

Information entropy is the average rate at which information is produced by a stochastic source of data. The information content (also called the surprisal) of an event ${\displaystyle E}$  is an increasing function of the reciprocal of the ${\displaystyle p(E)}$ of the event, precisely ${\displaystyle I(E)=-\log _{2}(p(E))=\log _{2}(1/p(E))}$. Shannon defined the entropy Η of a discrete random variable ${\textstyle X}$ with possible values ${\textstyle \left\{x_{1},\ldots ,x_{n}\right\}}$ and probability mass function ${\textstyle \mathrm {P} (X)}$ as:
$${\displaystyle \mathrm {H} (X)=\operatorname {E} [\operatorname {I} (X)]=\operatorname {E} [-\log(\mathrm {P} (X))].}$$
Here ${\displaystyle \operatorname {E} }$ is the expected value operator, and $I$ is the information content of ${\displaystyle I(X)}$ is itself a random variable. The entropy can explicitly be written as
$${\displaystyle \mathrm {H} (X)=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{b}\mathrm {P} (x_{i})}}$$
where $b$ is the base of the logarithm used.

Now, the KL divergence, try to find the cross entropy of two probability distributions.
So, what is the cross entropy:

The cross entropy between two probability distributions ${\displaystyle p}$ and ${\displaystyle q}$ over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution ${\displaystyle q}$, rather than the true distribution  ${\displaystyle p}$.
The cross entropy for the distributions ${\displaystyle p}$ and  ${\displaystyle q}$ over a given set is defined as follows:
$${\displaystyle H(p,q)=\operatorname {E} _{p}[-\log q]}.$$
The definition may be formulated using the Kullback–Leibler divergence${\displaystyle D_{\mathrm {KL} }(p\|q)}$ of  ${\displaystyle q}$ from ${\displaystyle p}$ (also known as the relative entropy of ${\displaystyle p}$ with respect to ${\displaystyle q}$).
$${\displaystyle H(p,q)=H(p)+D_{\mathrm {KL} }(p\|q)},$$
where ${\displaystyle H(p)}$ is the entropy of ${\displaystyle p}$.

Now, as we want to approximate a distribution function $p$ with other distribution function $q$, we want to minimize the cross entropy between these two. The first part $H(p)$ could not be changed as we want to find a distribution $q$ and $p$ is given. Hence, we need to minimize $KL$ divergence of these two, to minimize the cross entropy and better approximation for the $p$ distribution.


Does a solution for Wumpus World with neural networks exist? The Wumpus World proposed in book of Stuart Russel and Peter Norvig, is a game which happens on a 4x4 board and the objective is to grab the gold and avoiding the threats that can kill you. The rules of game are:

You move just one box for round
Start in position (1,1), bottom left
You have a vector of sensors for perceiving the world around you.
When you are next to another position (including the gold), the vector is 'activated'.
There is one wumpus (a monster), 2-3 pits (feel free to put more or less) and just one gold pot
You only have one arrow that flies in a straight line and can kill the wumpus
Entering the room with a pit, the wumpus or the gold finishes the game

Scoring is as follows: +1000 for grabbing the gold, -1000 for dying to the wumpus, -1 for each step, -10 for shooting an arrow. Fore more details about the rules, chapter 7 of the book explains them.

Well now that game has been explained, the question is: in the book, the solution is demonstrated by logic and searching, does there exist another form to solve that problem with neural networks? If yes, how to do that? What topology to use? What paradigm of learning and algorithms to use?

1*: My English is horrible, if you can send grammar corrections, I'm grateful.

2*: I think this is a bit confusing and a bit complex. if you can help me to clarify better, please do commentary or edit!

Yes! If you read ahead to the chapters in reinforcement learning in the same book, you'll see that the wompus world appears again there. Techniques like Q-learning can be used to solve it, and since Q-learning involves learning the shape of a function, a neural network can be employed as a function approximator.

The basic idea is to treat this problem as an input/output mapping (states -> actions), and to learn which actions produce the greatest rewards.

Note however, that these approaches rely on trial and error. The logic based approach reasons about the rules of the game, and can play reasonably well right away. The learning approach will need to try and fail many times before playing well.


How is a neural network where the majority of inputs are 0 trained? Consider AlexNet, which has 1000 output nodes, each of which classifies an image:

The problem I have been having with training a neural network of similar proportions, is that it does what any reasonable network would do: it finds the easiest way to reduce the error which happens to be setting all nodes to 0, as in the vast majority of the time, that's what they'll be. I don't understand how a network where 999 times out of 1000, the node's output is 0, could possibly learn to make that node 1.

But obviously, it's possible, as AlexNet did very well in the 2012 ImageNet challenge. So I wanted to know, how would one train a neural network (specifically a CNN) when for the majority of the inputs the desired value for an output node is 0?

It's the loss function.

I was using squared sum error, which I didn't think would have as a negative effect as it does, and I had to come to the explanation in my own time. Here's why:

From the perspective of the loss function, 999 times out of 1000, the output should be 0, so there will be an inherent massive bias towards 0 for all the output nodes. But this only occurs if the output nodes are actually trained when their desired outputs are 0, which is what happens in the case of the squared sum/mean error. However, in the case of cross-entropy loss, which is explained excellently here, you can see that the only node that receives gradients is the node that should be trained towards 1. This removes the massive bias towards 0, and punishes confident false positives, making it perfect for a classification problem.

As to how you would achieve something like this for regression I do not know, but at least this solves the issue for classification problems.


Is language understanding possible without qualia? I feel that many words if not all of them have a direct mapping to some kind of inner subjective experience, to a physical object, mental feeling, process or some other kind of abstract thing. Given that machines don't have qualia and no mapping of this kind, can they really understand anything even though they are made to answer to questions with lots of statistical training?

I'm going to be controversial here; so please don't vote this answer down if you just disagree with it.

Your question presupposes that machines do not or cannot possess qualia, which are required for true understanding. Given that we don't really know what it means to 'understand' something, and that even the meaning of 'meaning' itself is by no means a resolved issue, it might be overly specific.

In one strand of linguistics, the meaning of a word is defined by its use, and by the context of surrounding words. We could hazard a guess that children acquire the meaning of words through exposure to language, and the correlation of experiences with the corresponding sounds. How that works in detail is AFAIK not fully understood. But there would be nothing 'inherent' in a new-born human that would enable it to 'understand' anything.

If that is the case, then we could train a machine to do the same. Obviously, it would be a long and tedious process, and there is probably a reason why it takes us years to become proficient in our use of language. But if we correlate sensory input with linguistic utterances, a sufficiently sophisticated learning algorithm might be able to acquire some meaning for such utterances from the way they are used.

There are, of course, rather a lot of unknowns here. That is because the topic straddles various fields, from child language acquisition, corpus linguistics, the psychology of learning, and many more. And to my knowledge, none of these fields is sufficiently advanced to shed any light on this issue yet. There is the whole question of abstract words and concepts. How do we segment the continuous stream of sounds into discrete units (phonemes) without knowing what they are? With all that complexity I begin to appreciate why Chomsky opted for his Language Acquisition Device to avoid getting frustrated... :)

So, to answer your question: yes, it should be possible. A properly set up machine, which would be able to simulate human learning, would pick up its own mapping of linguistic structures to its experience from the world outside. And if we call this mapping the 'meaning' of those structures, then a machine can learn this, and presumably 'understand' language. If we ever get to that stage with AI is another question.


Is average pooling equivalent to a strided convolution with a specific constant kernel? It seems to me that average pooling can be replaced by a strided convolution with a constant kernel. For instance, a 3x3 pooling would be equivalent to a strided convolution (of stride $3$) with a $3 \times 3$ matrix of constants, with each entry being $\frac{1}{9}$.
However, I haven't found any mention of this fact online (perhaps it's too trivial of an observation)? Why then are explicit pooling layers needed if they can be realized by convolutions?

Is average pooling equivalent to a strided convolution with a specific constant kernel?

Yes.

Why then are explicit pooling layers needed if they can be realized by convolutions?

It is probably because the convolution is more expensive than the usual/natural implementation (i.e. just summing and then dividing).
To see why, let's consider your example. If you implement pooling with a convolution (or cross-correlation), we would need to perform $9$ multiplications, then $8$ summations, for a total of $17$ operations. If we implement pooling as usual, we would need to perform $8$ summations and $1$ division (or multiplication), for a total of $9$ operations.
Moreover, convolution may also be more prone to numerical instability (multiplications of numbers in the range $[0, 1]$ are not nice), but I am not completely sure about this, given that we multiply always by the same numbers.


Do we have to use CNN for Deep Q Learning? I read top articles on Google Search about Deep Q-Learning:

https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8
https://skymind.ai/wiki/deep-reinforcement-learning
https://neuro.cs.ut.ee/demystifying-deep-reinforcement-learning/
Q-Learning page on wikipedia*

and then I noticed that they all use CNN as approximator. If deep learning has a broader definition than just CNN, can we use the term "Deep Q-Learning" on our model if we don't use CNN? or is there a more appropriate definition for that kind of Q-Learning model? for example, if my model only using deep fully-connected layer.

*it doesn't say explicitly Deep RL means CNN on RL, but it uses the DeepMind (that uses CNN) as an example on Deep Q-Learning

No. DQN and other deep RL methods work well with fully connected layers. Here's an implementation of DQN which doesn't use CNNs: github.com/keon/deep-q-learning/blob/master/dqn.py

DeepMind mostly use CNN because they use image as input state, and that because they tried to evaluate performance of their methods vs humans performance. Humane performance is easy to measure at games with image as input state, and that's why CNN based methods present so promptly in RL now.


What's the difference between estimation and approximation error? I'm unable to find online, or understand from context - the difference between estimation error and approximation error in the context of machine learning (and, specifically, reinforcement learning).
Could someone please explain with the help of examples and/or references?

Section 5.2 Error Decomposition of the book Understanding Machine Learning: From Theory to Algorithms (2014) gives a description of the approximation error and estimation error in the context of empirical risk minimization (ERM), so in the context of learning theory. I will just summarise their definition. If you want to know more about these topics, I suggest that you read that section.  If you are looking for the definitions of these topics in other contexts (e.g. reinforcement learning), you should ask a new question and provide more context (i.e. a paper or book that uses those terms in RL), because I also don't intend to extend this answer, although it is possible that these definitions below apply to other contexts too.
Approximation error
The approximation error (AE), aka inductive bias, denote as $\epsilon_{\mathrm{app}}$, is the error due to the specific choice of hypothesis class (or set) $\mathcal{H}$.
Here are some properties.

The larger $\mathcal{H}$ is, the smaller this error is (this should be intuitive!)

This error does not depend on the training data

Estimation error
The estimation error (EE) is the difference between the approximation error $\epsilon_{\mathrm{app}}$ and the training error $L_{\mathcal{D}}\left(h_{S}\right)$, i.e.
$$\epsilon_{\mathrm{est}}=L_{\mathcal{D}}\left(h_{S}\right)-\epsilon_{\mathrm{app}}$$
Here are some properties.

The EE arises because the empirical risk is just a proxy for the expected risk (i.e. the true risk, which is not computable because we don't know the underlying probability distribution that generated the labeled data).

The EE depends on the

training data (given that it is defined as a function of the training error, which depends on the training data), and

the choice of the hypothesis class (given that it is defined as a function of $\epsilon_{\mathrm{app}}$).

Bias-complexity tradeof
If we increase the size and complexity of the hypothesis class, the approximation error decreases, but the estimation error may increase (i.e. we may have over-fitting). On the other hand, if we decrease the size and complexity of the hypothesis class, the estimation error may decrease, but the bias may increase (i.e. we may have under-fitting). So, we have a bias-complexity trade-off (where the bias refers to the approximation error or inductive bias) and the complexity refers to the complexity of the hypothesis class.


In variational autoencoders, why do people use MSE for the loss? In VAEs, we try to maximize the ELBO $\mathbb(E_q log\ p(x|z) + D_{KL}(q(z|x), p(z))$), but I see that many implement the first term as MSE of the image and it's reconstruction. Is this mathematically sound?

On page 5 of the VAE paper, it's clearly stated

We let $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ be a multivariate Gaussian (in case of real-valued data) or Bernoulli (in case of binary data) whose distribution parameters are computed from $\mathbf{z}$ with a MLP (a fully-connected neural network with a single hidden layer, see appendix $\mathrm{C}$ ).
...
As explained above and in appendix $\mathrm{C}$, the decoding term $\log p_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i, l)}\right)$ is a Bernoulli or Gaussian MLP, depending on the type of data we are modelling.

So, if you are trying to predict e.g. floating-point numbers (in the case of images, these can be the RGB values in the range $[0, 1]$), then you can assume $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ is a Gaussian, then you can equivalently minimise the MSE between the prediction of the decoder and the real image in order to maximise the likelihood. You can easily show this: just replace $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ with the Gaussian pdf, then maximise that wrt the parameters, and you should end up with something that resembles the MSE.
So, to answer your question more directly: yes, minimizing the MSE is theoretically founded, provided that you're trying to predict some floating-point number.


How can neural networks that extract many features be fooled by adversarial images? I have been reading a bit about networks where deep layers able to deal with a bunch of features (be it edges, colours, whatever).

I am wondering: how can possibly a network based on this 'specialised' layers be fooled by adversarial images? Wouldn't the presence of specialised feature detectors be a barrier to this? (as in: this image of a gun does share one feature with 'turtles' but it lacks 9 others so: no, it ain't a turtle).
thanks!

The link in DuttA's comment provides a good answer to the more general question of how adversarial images are generated. As to why it's possible at all, the key is that these specialized feature detectors really are just picking up features, and it's possible to construct an image that has those features, without resembling the target object much at all.

For example, we might imagine that the key features that predict an image of a gun are regions like the trigger guard and perhaps the muzzle. The features that are extracted really will just be "round-ish thing with a curved bit in it" and "perfectly circular opening". These might predict a gun really well, but it's also easy to imagine that you could place shapes satisfying those criteria the correct distance apart without including any of the other components that we'd expect. This is what you actually see in the adversarial images that are generated. Consider the bagel or crossword puzzle images below. While they are not actually pictures of the objects in question, it is fairly easy to see what the important features were for the network: a bagel is a round shape that's shadowed in the center and lighter between the center and rim. It's a bit orange. A crossword puzzle is a bunch of black and white squares distributed in a rough grid. Possibly the copyright symbol or the letter C was also important (it is found on many crosswords!).


Lego minifigure facial recognition: where to start? I'm interested in starting a project that will identify the face of a Lego mini figure from a digital photo. My goal is to eventually map the expression of a person's face to the Lego mini figure.

I don't have any experience working with image recognition technology (my technical experience is mainly in web technology), and I am looking for recommended platforms or resources that I could get started with.

Most helpful would be recommendations for image recognition technologies (Python would be great!) that I could start to experiment with.

NOTE: I'm aware of SparkAR as a library designed to for Instagram camera effects specifically, and even though I'm not interested in Instagram, I wonder if there are comparable libraries/studios/products for working with image recognition development.

Since you are mainly interested in starter libraries and packages to read up on here are some pointers:

Image recognition as well as general spatial classification mainly consists of two major tasks.

Translating images to data that can be fed into a ML model of any kind.
Building a model that is able to use this data to complete tasks like classification (which lego figure is this?), spatial clustering (which pixels form the "face"), etc.

For the first task you will find a lot of python libraries that suit the task but `` PIL/Pillow``` should be the main library for this. So start by reading some documentation and transferring the images you do have (this includes standardizing and cleaning the images to improve the results of step 2).

For the second task the actual model you will use depends on your task but generally speaking some form of neural net is a good place to start and if you prefer python then you should look into tensorflow and keras (easy to use interface to tensorflow).

Where to start

Start with the classical MNIST numbers recognition case and if you have grokked that, you will now where to go from there or at least be able to ask a more specific question here.

References

https://pillow.readthedocs.io/en/stable/
https://keras.io/
https://towardsdatascience.com/image-classification-in-10-minutes-with-mnist-dataset-54c35b77a38d


What are the differences in scope between statistical AI and classical AI? What are the differences in scope between statistical AI and classical AI?

Real-world examples would be appreciated.

Statistical AI, arising from machine learning, tends to be more concerned with "inductive" thought: given a set of patterns, induce the trend.
Classical AI is the branch of artificial intelligence research that concerns itself with attempting to explicitly represent human knowledge in a declarative form i.e given a set of constraints, deduce a conclusion.
 Another difference is that C++ tends to be a favorite language for statistical AI while LISP dominates in classical AI.

A system can't be more intelligent without displaying properties of both inductive and deductive thought. This lends many to believe that in the end, there will be some kind of synthesis of statistical and classical AI.


What is it called in AI when a program is designed to make "x in the style of y"? Simplified: What is it called in AI when a program is designed to make "x in the style of y;" when it trains off of two types of sources in order to make a thing from source one, informed by features from source two? For example, if a network made up of two smaller networks were to take sheet music of a specific compositional style in network A and audio samples from a certain genre in B and through an interface creates music from a certain genre in a certain compositional style; the is comes from One, the seems comes from Two.

For more coarse and obvious examples:

"Compose synthpop in the style of Beethoven"
"Draw impressionism in the style of Mondrian"
"Generate casserole recipes using only ingredients most likely to fluctuate in price given current market data"
"Sketch baseballs that look like they're made of espresso foam"

This problem is known as "style transfer".
The field was started by Gatsys et al. in 2016, and has seen a lot of work over the last few years, including conditional translation from Isola et al. in 2017 and unpaired translation from Zhu et al. 2017.

This picture (Figure 2 from Gatsys et al.) shows the idea behind style transfer, and illustrates the dramatic results that are possible with these models.


What kind of neural network architecture is suitable for variable length block-like time series data? I'm not sure what this type of data is called, so I will give an example of the type of data I am working with:

A city records its inflow and outflow of different types of vehicles every hour. More specifically, it records the engine size. The output would be the pollution level X hours after the recorded hourly interval.

It's worth noting that the data consists of individual vehicle engine size, so they cant be aggregated. This means the 2 input vectors (inflow and outflow) will be of variable length (different number of vehicles would be entering and lraving every hour) and I'm not sure how to handle this. I could aggregate and simply sum the number of vehicles, but I want to preserve any patterns in the data. E.g. perhaps there is a quick succession of several heavy motorbike engines, denoting a biker gang have just entered the city and are known to ride recklessly, contributing more to pollution than the sum of its parts.

Any insight is appreciated.

(this response should be a comment but I don't have yet the reputation to comment).

If I'm understanding your problem correctly you have a variable number of input which have an order and only one output ? It look like the kind of task where you could use recurrent neural network (the most common ones are the LSTM and GRU).

If you use a recurrent neural network you could (if you have the timestamps of your data) cut the hourly interval into smaller time step to help detect pattern.


When exactly is a model considered over-parameterized? When exactly is a model considered over-parameterized?
There are some recent researches in Deep Learning about the role of over-parameterization toward generalization, so it would be nice if I can know what exactly can be considered as such.
A hand-wavy definition is: over-parameterized model is often used to described when you have a model bigger than necessary to fit your data.
In some papers (for example, in A Convergence Theory for Deep Learning
via Over-Parameterization), over-parameterization is described as:

they have much more parameters than the number of training samples
meaning that the number of neurons is polynomially large comparing to the input size
the network width is sufficiently large: polynomial in $L$, the number of layers, and in $n$, the number of samples

Shouldn't this definition depend on the type of input data as well?
For example, I fit:

1M-parameters model on 10M samples of 2 binary features, then it should not be over-parameterized, or

1M-parameters model on 0.1M samples of 512x512 images, then is over-parameterized, or

the model in the paper Exploring the Limits of
Weakly Supervised Pretraining "IG-940M-1.5k ResNeXt-101 32×48d" with 829M parameters, trained on 1B Instagram images, is not over-parameterized

Ok so after a little more reading, I am currently satisfy with what I found for this question.

Yes, the "under-parameterized" and "over-parameterized" terms do not currently have a widely accepted definitions.
Any definition for those term should consider the input data domain as well as the architecture and training procedure.

In a recent paper Deep Double Descent from OpenAI Nakkiran et. al 2019, the authors tried to formalized and generalized the concept of "interpolation threshold" in the "double descent" phenomenon, both terms popularized by Belkin et. al 2019 Reconciling modern Machine Learning practices with the Bias-Variance Tradeoff.

In the Deep Double Descent paper, they define a concept called "Effective Model Complexity (EMC)", which include model architecture, training procedure and data to describe the "interpolation threshold" (the moment at which model can fit near perfect the training data).

EMC below the interpolation threshold is considered "under-parameterized" and above interpolation threshold is considered "over-parameterized".

So according to this definition of EMC, I suppose:

  1M-parameters model on 10M samples of 2 binary features, then it should not be over-parameterized

is over-parameterized because of the simplicity of input data.

  1M-parameters model on 0.1M samples of 512x512 images, then is over-parameterized, or

is probably not parameterized, if it cannot fit the training data with near 0 loss.

  the model in the paper Exploring the Limits of Weakly Supervised Pretraining "IG-940M-1.5k ResNeXt-101 32×48d" with 829M parameters, trained on 1B Instagram images, is not over-parameterized

is not over parameterized because it cannot fit the entire training data perfectly.

I am curious to see if EMC will catch on and be a popular measure for model complexity in the future.


How does the Jetson AI write its screenplays? I've watched the Sunspring video which didn't make any sense to me (a lot of nonsense monologues), mainly because it was created by Jetson AI.

What was the mechanism of creating such screenplay?

On what criteria was it trained? What was the goal or motivation in terms of training criteria of defining when text does make sense? And what was missed (that it's so bad) and how possibly this could be improved?

It appears to use Recurrent NNs (RNNs) that have a 'Long Short-Term Memory' (LTSM) architecture.

Here's a summary of the development process that the author, Ross Goodwin, went through to create it.

It seems to me (and is also observed in the above link) that the output is rather poor - simply comparable to what one might expect from Markov chains, a technique that is over 100 years old.

I haven't dug deeply into the technique, so I could be misktaken, but perhaps one of the reasons that it's so bad is that (as far as I can see), the model-building process is essentially lexical - i.e. it is linking together tokens (words) without any more informed language model to guide it. In particular, the generated output doesn't seem to know anything about the functional roles played by objects (chairs are supporting objects, used by humans for sitting on etc), which is something that might be fairly readily incorporated.


What are the standard problems for CNNs and LSTMs? What are the standard (or baseline) problems (or at least common ones) for CNNs and LSTMs? As an example, for a feed-forward neural net, a common problem is the XOR problem.

Is there a standard problem like this for CNNs and LSTMs? I think for a CNN the standard test is to try it on MNIST, but I'm not sure of an LSTM.

It's more domain- or task-specific. There is no obvious baseline anymore because these models and this field has evolved into too large of an ecosystem. Nonetheless, I'll list a couple of notorious examples below.

Image classification:

MNIST
CIFAR
ImageNet

Detection/segmentation:

PascalVOC
COCO
CityScapes

Pose estimation:

MPII
LEEDS

Text classification:

IMDB
yelp

Question answering:

SQuAD

Translation:

WMT
IWSLT

This is just a taste. There are tons more both in each category and the number of categories, a good source is the Papers with Code website.

Therefore, there is no single standard problem, given that there are too many that all in one shape or form use CNNs or RNNs (or others).


Are there some guidelines for designing the architecture of neural networks? I started to study neural networks recently. I understand how I should define the input and output layers. But I can't find any guidelines on how to build hidden layers. More concretely, for each specific task, are there some guidelines (or rules of thumb) for answering the following questions?

How many layers do I need?
How many neurons per layer?
What activation functions should I use?
How do I connect the neurons?

In general no. The research area that you seem to be looking for is called generalization, which is very much so still an active area of research. Actual architecture design strongly depends on the dataset itself and available resources, and thus there isn't a general rule of thumb that works for every case.


Are there tools to help labelling images? I need to manually classify thousands of pictures into discrete categories, say, where each picture is to be tagged either A, B, or C.

Edit: I want to do this work myself, not outsource / crowdsource / whatever online collaborative distributed shenanigans. Also, I'm currently not interested in active learning. Finally, I don't need to label features inside the images (eg. Sloth) just file each image as either A, B, or C.

Ideally I need a tool that will show me a picture, wait for me to press a single key (0 to 9 or A to Z), save the classification (filename + chosen character) in a simple CSV file in the same directory as the pictures, and show the next picture. Maybe also showing a progress bar for the entire work and ETA estimation.

Before I go ahead and code it myself, is there anything like this already available?

There are a few tools that you can use to annotate (or label) data. For example, labelme or Labelbox. Have a look at this question for more alternatives.


Is it possible to design a neural network from the plot of data points? Suppose I have been given the following diagram

I need to design a neural network to separate the points or classes. How can I do that?

From the diagram you have given, it's quite clear that you have to design a supervised network. Also, it's clear that you are dealing with a problem that's not linearly separable i.e. you can't separate two classes using a single line. Particularly, it will require three lines to separate these two classes.
From the above observations, what you can do is:

Make training data sets of form [(x,y), o] where (x,y) are the co-ordinates on the plot and o is the class that point belongs to.
Note: You can actually divide your data set into training data and test data. Just, extract the data, randomly shuffle it and take first 80% of data to train and rest 20% of data to test.

Design a 2-2-2-1 neural network with bias units and some random initial weight values. This is because we will require minimum three lines to separate two classes, so 2 hidden layers (each consisting of 2 nodes and bias nodes) and 1 output layer (with bias node). The first layer is input layer with 2 nodes (no bias) as we have x-coordinate and y-coordinate.

Train your training data using this network using the Error Back Propagation algorithm i.e. update the weights between the layers and the bias weights too. Run the EBP algorithm for 100-2000 epochs.

Then, use the test data to see, if you get the desired output.

Note: I am NOT quite sure how to obtain the coordinates from the diagram. You can either manually note all points or research for some methods to get those points automatically from the image (some image processing algorithms).
Edit: I found a good video which explains neural networks. In the video, there is non-linear data like the one in your image which is separable using multiple lines and the network is designed to try to find curves such that data can be separated and the output resembles to that when used straight lines.


Confusion about temporal difference learning I have a couple of small questions about the David Silver lecture about reinforcement learning, lecture slides  (slides 23, 24). More specifically it is about the temporal difference algorithm:

$$V(s_{t}) \leftarrow  V(s_t)+ \alpha \left[ G_{t+1}+\gamma V(s_{t+1})- V(s_t) \right]$$

where $\gamma$ is our discount rate and $\alpha $ the learning rate.
In the example given in the lecture slides we observe the following paths:

$(A,1,B,0), (B,1), (B,1), (B,1), (B,1), (B,1), (B,1), (B,0)$

Meaning for the first trajectory we are in state $A$, get reward $1$, get to state $B$ and get reward $0$ and the game finishes. For the second trajectory we start in state $B$, get reward $1$ and the game finishes ...

Let`s say we initialize all states with value $0$ and choose $\alpha=0.1, \gamma=1$

My first question is whether the following "implementation" of the $TD(0)$ algorithm for the first two of the above observed trajectories correct?

$V(a)\leftarrow0 + 0.1(1+0-0)= 0.1; \quad V(b)\leftarrow0+0.1(1+0-0)=0.1$
$V(b)\leftarrow0.1+(0.1)(1+0-0.1)= 0.19$

? If so, why dont we use the updated value function for $V(b)$ to also update our value for $V(a)$?

My third question is about the statement that

  $TD(0)$ converges to solution of max likelihood Markov model

this means that if we keep sampling and apply the $TD(0)$ algorithm that the thereby obtained solution converges towards the ML-estimate of that sample using the Markov Model? Why dont we just use the ML-estimate immediately?

My first question is whether the following "implementation" of the (0) algorithm for the first two of the above observed trajectories correct?

$V(a)\leftarrow0 + 0.1(1+0-0)= 0.1; \quad V(b)\leftarrow0+0.1(1+0-0)=0.1$
$V(b)\leftarrow0.1+(0.1)(1+0-0.1)= 0.19$

Your calculations for the first trajectory $(A,1,B,0)$ is incorrect for either TD or Monte Carlo. For some reason you have assigned either an immediate reward or return of $1$ to the second step, whilst in the example, it is $0$ for both the sampled return and the single-step TD target.
In addition, you quote this update rule for single-step TD:
$$V(s_{t}) \leftarrow  V(s_t)+ \alpha \left[ G_{t+1}+\gamma V(s_{t+1})- V(s_t) \right]$$
. . . actually that is not the usual notation. The symbol $G_t$ is normally used to show a "return" value - a sum of rewards (often weighted by some factor, such as $\gamma$). The usual way of showing the TD update rule would be:
$$V(s_{t}) \leftarrow  V(s_t)+ \alpha \left[ r_{t+1}+\gamma V(s_{t+1})- V(s_t) \right]$$
i.e. using the immediate reward. This might be a simple typo, however I am explaining this because it may behind your incorrect calculation.
The correct calculation is not very much different from yours though:

$V(a)\leftarrow0 + 0.1(1+0-0)= 0.1; \quad V(b)\leftarrow0+0.1(0+0-0)=0.0$
$V(b)\leftarrow0.0+(0.1)(1+0-0.0)= 0.1$

If so, why dont we use the updated value function for $V(b)$ to also update our value for $V(a)$?

You can, and would do this in either of the following situations:

In online learning, you experience a trajectory with states in order (A,B) again

In offline learning, you repeat the previous experience in batch learning or using experience replay

It is worth noting that if you take a small batch of data and repeat it again and again to update the value functions, that they will converge to values depending on your data set. That is what the slide is explaining in the lecture - highlighting the difference that TD and MonteCarlo will make when you do this. However, if that data set is a very small subsample of possible random behaviour in the environment, then you may not create an accurate value function, but instead it will be the best one that you can given the limited data. If it is easy to collect more experience, then that is often preferable.

Why dont we just use the [maximum-likelihood]-estimate immediately?

Because it is not directly useful for a value prediction task, and you would need some mechanism to use that maximum likelihood MDP model to generate value predictions. With TD, you are already in the process of making this estimate*.
You could take the existing samples, and use them to generate the parameters of an MRP (Markov Reward Process, as there are no example actions in the trajectory) based on the observations. That "best guess" MRP is your maximum likelihood MDP model, and would evaluate the same as your converged repeated TD batch over the samples.
The main difference explained by the slide is that Monte Carlo will converge to $V(a) = 1$ because the only sample with A in it has a return of 1 following state A. Whilst TD will converge to $V(a) = 1.75$, because it treats the same sample as the only instance of state progression from A - e.g state A "always" has an immediate reward of 1 then goes to state B. Both algorithms will converge to $V(b) = 0.75$.

* There are algorithms, such as Dyna-Q, which partially do this, using experience gathered so far to create a model of the environment dynamics. Sometimes this is useful and effective. However, it is not always possible or the best approach.


How do I detect similar objects in an image? I want to tackle the problem of detecting similar objects in an image. To illustrate the problem consider this photo of some Lego bricks as my "input":

The detection routine should identify similar objects. So for the given input, it should e.g. identify the following output:

So an object might appear none to multiple times in the input image. For example, there are only two bricks marked with a blue cross, but three bricks marked with a red cross.

It can be assumed that all objects are of similar kind, so e.g. only Lego bricks or a heap of sweets.

My initial idea is to apply a two-phased approach:

Extract all objects in input image.
Compare those extracted objects and find similar ones.

Is this a valid approach or is there already some standard way of solving this kind of problem? Can you give me some pointers how to solve this problem?

Disclaimer: I have never used Siamese networks

I would approach this problem in two steps:

First: train object detector and train it for eligible classes of object, for example using Yolo architecture.

You could use pretrained object detector and finetune it for your classes of objects.

Second: extract a lot of bounding boxes of eligible objects of the same classes from your dataset and train Siamese network on their subimages.

Siamese network output similarity measure between two objects.

Your pipleline would look like:

Run object detetctor
For each two pair of objects of the same class rescale bounding boxes and run Siamese network.
Check if the similarity distance between two objects is less then threshold (tunable hyperparameter)


What are the state-of-the-art results in OpenAI's gym environments? What are the state-of-the-art results in OpenAI's gym environments? Is there a link to a paper/article that describes them and how these SOTA results were calculated?

There is the leaderboard page at the gym GitHub repository that contains links to specific implementations that "solve" the different gym environments, where "to solve" means "to reach a certain level of performance", which, given a fixed reward function, is typically measured as the average (episodic) return/reward. For example, in the case of the CartPole environment, you solve it when you get an average reward of $195.0$ over $100$ consecutive trials.


How should I incorporate numerical and categorical data as part of the inputs to the U-net for semantic segmentation? I am using a U-Net to segment cancer cells in images of patients' arms. I would like to add patient data to it in order to see if it is possible to enhance the segmentation (patient data comes in the form of a table containing features such as gender, age, etc.). So far, my researches have led me nowhere. What can I do in order to achieve this?

What you want to do is called multi-task learning. Here's what you do:

Create a second Input.
Attach it to 1D CNN (2-3 layers), so it aggregates this tabular information.
Concatenate this feature with the intermediate feature generated by the U-Net using Concatenate layer.
Put a dense layer of 2 after this.
Put softmax with units = number of classes.
Add CE loss calculated with this one and the ground labels to the loss of U-Net.

This is in regards to TensorFlow. The same can be done in PyTorch easily.


Which pathfinding algorithms can be applied on coloured graphs? Are there any (well validated) approaches for applying pathfinding algorithms on a graph following specific rules?

To be more specific: I want to introduce a graph with coloured edges. The idea is to apply a well known pathfinding algorithm (such as Dijkstra) on the graph given the rule: "only black and red edges".

The idea is to apply a well known pathfinding algorithm (such as Dijkstra) on the graph given the rule: "only black and red edges"

Remove all the non-black-and-red edges from the graph first, then run it through any off-the-shelf pathfinding implementation.  Or implement your own, and have it completely ignore the non-black-and-red edges.


Neural Network Optimizers in Reinforcement Learning non-well behaved environments https://stackoverflow.com/questions/36162180/gradient-descent-vs-adagrad-vs-momentum-in-tensorflow

Here, the nice gifs explain how different algorithms approach towards the root. Unfortunately, the environment in the gif is way too simple and real cases have much more complex environments. Also, in reinforcement learning, the solutions should change each moment in a difficult enough environment since things are dynamic.

My question is which optimizer is best for reinforcement learning in such dynamically changing environment? Adadelta should not move beyond local minima so do we have to use SGD or Adadelta with an exploration heuristic? Please let me know in detail your thoughts.

The most commonly-used optimizer in Deep Reinforcement Learning research the past few years is probably ADAM (or its AMSGrad variant, which in most frameworks like keras/tensorflow/pytorch etc. can be used by setting an amsgrad flag to True in the construction of an ADAM optimizer, and I believe is often also already set to True by default). This is a somewhat newer optimizer which isn't included in that visualization you linked to.

RMSProper is also still quite popular, and there may be some arguments that it might be better suited for non-stationary learning problems (which we typically have in RL, our learning targets tend to be non-stationary due to the agent adapting its behaviour over time).

All of these more "fancy" optimizers, typically including some form of momentum term, can in fact be especially useful for optimization in loss "landscapes" that are not smooth. Imagine that your loss landscape looks like a tall mountain, where we'd like our optimizer to "glide" all the way down. Now suppose that the overall trend of a side of the mountain is downwards, but that there are lots of little bumps along the way down, that it's not a smooth mountain. The more fancy optimizers with momentum terms are more likely to "jump" over those bumps than a regular SGD optimizer without momentum terms. This is because, when a little bump is encountered, the local gradient may point the wrong way, but a momentum term "remembers" the overall trend of gliding downwards along the mountain and still points in that same direction for a while.


How to define machine learning to cover clustering, classification, and regression? How to define machine learning to cover clustering, classification, and regression? What unites these problems?

I report three definitions of machine learning (ML) and I also explain that ML can be divided into multiple sub-tasks or sub-categories in this answer. However, it may not always be clear why classification, regression, or clustering can be considered machine learning tasks or can be solved with ML algorithms/programs, so let me explain why these tasks can be solved with ML, based on Tom Mitchell's definition of an ML algorithm that I report below for completeness.

A computer program is said to learn from experience $E$ with respect to some class of tasks $T$ and performance measure $P$, if its performance at tasks in $T$, as measured by $P$, improves with experience $E$.

So, according to this definition, for a computer program to be a "machine learner", we need to identify $E$, $T$, and $P$, and show that the computer program improves with $E$ at performing the task(s) $T$, according to $P$.
For the case of classification or regression (these are the tasks, which is just a synonym for problems), let's suppose that we are given the training labelled dataset $D = \{(x_1, y_i), \dots, (x_N, y_N) \}$ of $N$ tuples $(x_i, y_i)$, where $x_i$ are the inputs to the program (or model) and $y_i$ (the label, aka class, hence the name classification) is the output that the program should produce. In the case of classification, $y_i$ is an element of a discrete set, e.g. $\{0, 1\}$ in the case of binary classification, while, in the case of regression, $y_i \in \mathbb{R}$ is a real number.
So, in this case, the experience $E$ is $D$ (the training labeled dataset). The task $T$ is classification or regression, depending on whether $y_i$ belongs to a discrete or continuous space. The performance measure $P$ can be e.g. the cross-entropy (which is typically used to solve the classification task) or the mean squared error (which is typically used to solve the regression task). (I will not recall the definitions of these performance measures here: you can take a book on ML for the details).
So, we have identified $E$, $T$ and $P$. Now, we also need to argue why a program's performance measured by $P$ would improve with $E$. Let's suppose that you are using a neural network trained with gradient descent to solve these tasks. After one epoch/iteration of gradient descent, the loss will be smaller, so the performance of the program will be higher. So, this would indeed be a machine learner, according to Mitchell's definition of ML.
In the case of clustering (the task $T$), the only difference is that the dataset $D$ (i.e. the experience $E$) is unlabelled, i.e. we do not have labels, but we are given only the inputs, and the goal, as you know, is to group these inputs based on their similarity, according to some notion of similarity, which is your performance measure $P$.
The other definitions of ML that I report in the other answer are also consistent with Mitchell's definition. More precisely, most of these definitions are based on the idea that an ML algorithm is an algorithm that "finds patterns in data". To solve the classification, regression and clustering tasks/problems, an ML algorithm/program needs to find patterns in the data (either explicitly, like in the case of clustering, or indirectly, like in the case of classification), in order for the program's performance to improve.
Moreover, given that you brought this up in the now-deleted comments, I don't think that the definition of machine learning necessarily implies prediction (i.e. the use of a model to forecast something about future data), but, yes, ML is often used for prediction.
It's also important to note that, when someone says "machine learning" without further details or information (like you did in your post/question), one may refer to

the field/area of study that studies and applies ML algorithms, or
an ML algorithm (and sometimes model).

Mitchell's definition is the definition of what it means for an algorithm/program to be called an ML algorithm/program. However, my definition of ML as a field immediately follows from the definition of an ML algorithm.
Finally, we could say that a problem that can be formulated in such a way that an ML algorithm can be applied is an ML problem. Note that Mitchell's definition does not mention "ML problem" anywhere, but just "problem", so this is not a circular definition. So, we could say that classification, regression, and clustering are "ML problems" because they can be solved with ML algorithms. More generally, all these three problems can be thought of as "function approximation" problems, i.e., in all these cases, the solutions are functions.


What is the difference between reinforcement learning and AutoML? My vague understanding of reinforcement learning (RL) is that it's very similar to supervised learning except that it updates on a continuous feed of data/activity, this to me sounds very similar to AutoML (which I've started to notice being used).

Do they use different algorithms? What is the fundamental difference between RL and AutoML?

I'm after an explanation for somebody who understands technology but does not work with machine learning tools regularly.

Automated machine learning (AutoML) is an umbrella term that encompasses a collection of techniques (such as hyper-parameter optimization or automated feature engineering) to automate the design and application of machine learning algorithms and models.

Reinforcement learning (RL) is a sub-field of machine learning concerned with the task of making decisions and taking actions in an environment so as to maximize (long-term) reward (which is the goal of the so-called RL agent). RL is (at least partially) based on the way animals (including humans) learn. For example, the usual way of training a dog to perform a certain task is to reward it with food whenever it takes the correct action (for example, jumping, if you want the dog to jump whenever you make a certain gesture with your hand). In this case, the RL agent is the dog, the task the dog needs to perform (e.g. jumping) is the environment, food is the reward and the goal is to get food.

Given that reinforcement learning (RL) is a sub-field of machine learning, then, in principle, AutoML can also be used to automate the design of RL algorithms, models or agents. For example, if you use a neural network to represent the policy (the function the determines which action to take in the environment), then you can potentially use AutoML to find the most appropriate architecture (for example, the most appropriate number of layers) for this neural network.


As an AI researcher, what subjects do you find yourself referring to most often? This is a bit of a weird question.
I am hoping to create an online reference since I have some downtime.  I know some about statistics but very little about computer science.  As a result, the reference guide I am hoping to create will be very statistics oriented - even though I wish that it could be a reference for someone who wants to start from scratch and work their way to AI.
While I would love to be involved with AI, from what I have read about ML and AI, seems like AI does not involve much statistics.  (A lot of statistical theory is based off normal assumption and math, and ML seems to bypass that by not requiring strong assumptions nor analytical results). CS seems to be more relevant.
And so my question is, since my guide will mostly cover statistics, how relevant would it be for someone who wants to get into AI?  If it's not relevant, then I guess I'll just make my guide for someone who wants to get into stats/data science, as opposed to someone who wants to be an AI researcher.
I guess another way to phrase my question is, as an AI researcher, when you "google" stuff, wikipedia things, or go to your notes, what subjects are you looking at and what exactly are you googling?  Are you getting a refresher on how to code back propagation?  Or are you getting a refresher on the pros and cons of L1 vs. L2?  Do you ever look at how to implement a boosting tree or NN using a pre-existing package?
Basically, I know that what I can provide will be relevant to HS/college stats and data science students.  But what really want to do is create something useful for aspiring/current AI researchers.  The former is realistic, the latter is a dream.  I want to see if my dream is realistic.
Thanks!

I think AI researcher mostly google some new papers because now there's just a crazy amount of them published. Sometimes people just forget the new concept which was introduced in a paper they read several months ago and they google that concept. Sometimes I forget some loss functions (and intuition behind them) used in a specific area like computer vision, natural language processing or audio processing. Like dice loss or contrastive losses. So these are more advanced things than pros and cons of L1 and L2 losses and how to code backprop. Usually I find the answers on https://distill.pub/ or https://towardsdatascience.com/. Have a look at those, I think they represent correctly current interests and topics for refreshment in AI research community. From my experience I can say not much statistics is used in contemporary AI research (unless you're doing research in statistical learning theory). Sometimes I google some statistical tests to prove results of my experiments are statistically significant and I think that's it.


How to decrease accuracy from 99% to 80%~85% using keras for training a model How do I decrease the accuracy value when training a model using Keras; which parameters can I change to decrease the value?

My objective is not to actually decrease it, but just to know which parameters influence the accuracy

sgd = optimizers.SGD(lr=1e-2)

There are many things affecting accuracy. I'm gonna assume a lot here because you don't say anything about the model, what you're trying to achieve or how many classes you have. You're not even saying whether you're classifying or not. Also, you're not saying which accuracy you're using (classification, AUC, F1 etc.).

I'm gonna assume here that you have some classification problem.

Accuracy is the measure of how many classifications you got correct. In my experience 99% is a warning sign because it's too good to be true, and a result like that is often due to overfitting. Since this, in my experience, is the main reason you'd actually want the accuracy down, this is what I'm going to assume is your problem.

Overfitting occurs when you train "too much" and the model only learns things that are within your training set, and fails on everything else. That is: it generalized bad.

To prevent this there are a number of things you could do;

1) Data segmentation

The most common is to split your data into three bulks: training (~70% of the data), validation(~20%) test (~10%). These percentages are indications and would vary depending on how much data you have, and the balancing.

The idea is that you train on the training data, you run the validation through the network, and calculate the accuracy. When this accuracy, call it validation accuracy, is satisfactory, then you stop the training and run the test data through it. The latter accuracy (test accuracy) is the one that most papers publish (combined with AUC and F1 score.

Important: When you have split the data into these bulks you should put away the test, and not use it during training at all. You only use this at the very end to do an extra check that you haven't overfitted.

2) Regularization
There are many types of regularization. Two very popular regularization methods for preventing overfitting are the L2-regularization (see previous link) and the dropout methods.

Without going into detail, these methods prevent the model weights from becoming too large. This is a good thing since the model won't rely too much on one feature, which in turn attenuates overfitting.

I hope you learned something, and the most important lesson is that you should know what you're doing. If not, you could end up with a model that is not behaving like you thought. In the case of overfitting, you'd end up with a model that only works on your training data, which doesn't really do much good.

I really recommend the book by Goodfellow: deeplearningbook.org.


Can games be solved without an evaluation function? Fundamentally, a game-playing AI must solve the problem of choosing the best action from a set of possible actions.
Most existing game AI's, such as AlphaGo, do this by using an evaluation function, which maps game states to real numbers. The real number typically can be interpreted as a monotonic function of a winning probability estimate. The best action is the one whose resultant state yields the highest evaluation.
Clearly, this approach can work well. But it violates one of Vladimir Vapnik's imperatives (in his book "Estimation of Dependences Based on Empirical Data"): "When solving a problem of interest, do not solve a more general problem as an intermediate step." In fact, he specifically states as an illustration of this imperative,

Do not estimate predictive values if your goal is to act well. (A good strategy of action does not necessarily rely on good predictive ability.)

Indeed, human chess and go experts appear to heed his advice, as they are able to act well without using evaluation functions.
My question is this: has there has been any recent research aiming to solve games by learning to compare decisions directly, without an intermediate evaluation function?
To use AlphaGo as an example, this might mean training a neural network to take two (similar) board states as input and output a choice of which one is better (a classification problem), as opposed to a neural network that takes one board state as input and outputs a winning probability (a regression problem).

Human chess and go experts clearly use evaluation functions. They do come up with moves that look sensible without evaluating the board position, but to validate these candidate moves they evaluate board positions that occur at the end of the variations they calculate. Pretty similar to AlphaGo.

Inputting two board states and outputting a preference is a (much) more complex task than mapping one board state into the real numbers. And it gives you less information. So its a lose-lose choice. (I did try something very similar and it didn't work at all. The reason is that you didn't just double the input size, rather you made the input space quadratically bigger.)

If you compare two board states that differ just by one move, then your input space doesn't quite explode as much, but you have to do a ton of comparisons to make a decision. The logical choice would be to output a preference distribution over all possible moves - but that's exactly what AlphaGo does in its policy network. There is also an earlier paper which trained a network to predict expert moves, which comes down to the same thing. And yes, both these networks play quite strongly without any search or board evaluation. But nowhere near AlphaGo-level.


Which API can I use for tracking the position of animal in one or more images? I'd like to build an application for tracking the position of a given animal (e.g. a cat) in a series of images.

Is there any off-the-shelf API I could use?

Azure has some Vision APIs, but it seems to me they can't be used to get the position of something in an image.

FastAI is the most “out of the box” API for this type of task.

For video examples (and a little theory) check out the MOOC section of their site.

Practical Deep Learning and Cutting Edge Deep Learning are the two sections most relevant to you.

But if you want a working implementation check out this GitHub repo that implements SSD for your purposes. I can’t say how simple the API is but it does what you are seeking (in pure PyTorch).

note: FastAI was originally built on top of PyTorch (although they are expanding out now), so you’ll be using PyTorch and need a rough idea of how to work with tensors. Most of the challenges are already implemented in their api. Installing their library will automatically install PyTorch.


How can one use Genetic Algorithms to solve the "15 Puzzle" (Mystic Square)? How would one go about solving the 15 squares puzzle using a Genetic Algorithms approach?  In particular, I'd like to understand how you would
represent the "chromosome" in the evolving system.  That is, what's the relationship between the (artificial) "genes" and some sort of phenotypical expression w/r/t the problem.  It seems like genes would somehow represent moves or sequences of moves but I'm not entirely clear how this would work.

There are a few ways of handling this within GA's, but most of them actually amount to using some kind of Genetic Programming instead.

The simplest way, and most similar to what you've proposed is called linear genetic programming. In this representation, you break the genome into a set of equal-width pieces. Each piece is interpreted as a machine-language instruction for a virtual machine. In your case, plausible instructions might be "move left" or "move right". Most versions use variable-length genomes, so your program's length depends only on how many instructions there are.

Another approach is to use the standard LISP-like genetic programming system, which Koza documents in his book for other simple problems, like the Santa Fe Trail.

More complex encodings are also possible. Grammatical evolution is another one that is similar to a GA in spirit, but interprets a genome according to a CFG instead of as instructions to a virtual machine.


What is the advantage of using MCTS with value based methods over value based methods only? I have been trying to understand why MCTS is very important to the performance of RL agents, and the best description I found was from the paper Bootstrapping from Game Tree Search stating:

Deterministic, two-player games such as chess provide an ideal
test-bed for search bootstrapping. The intricate tactics require a
significant level of search to provide an accurate position
evaluation; learning without search has produced little success in
these domains.

I however don't understand why this is the case, and why value based methods are unable to achieve similar performance.
So my question would be:

What are the main advantages of incorporating search based algorithms with value based methods?

Assuming a continuous/uncountable state space, we can only estimate our value function using function approximation, so our estimates will never be true for all states simultaneously (because, loosely speaking, we have far more states than weights). If we can look at the (approximated) value of states we take in, say, 5 actions time, it is better to make a decision based on these estimations, taking into account the true rewards observed after the 5 actions.
Further, MCTS also allows more implicit exploration as when choosing the actions to expand the tree we are potentially choosing lots of non-greedy actions that lead to better future returns.


Could it be possible to detect text, symbols, and components directly in a scanned PDF file with a program like Tensorflow or another program? I have this problem where I need to get information out of PDF document sent from a scanner. The program needs to be learnable in some way to recognize what different figures mean. Most of this should happen without human interference so it could just give a result after scanning the file.
Do anyone know if it's possible to do with a machine learning program or any alternative way?

Yes, that's possible.
I am working on a project in which I have to detect text in images. I did a quick search and found these two algorithms:

1. EAST: (Efficient and Accurate Scene Text Detector)
I am not sure if it is based on Machine Learning. Here are some links link1  link2 explaining how to use it with an example and using tesseract to extract the detected text.

2. CTPN: (Connectionist Text Proposal Network)
This algorithm is based on Machine Learning. Here is its link in github. In the description, you will find a link to a pre-trained model that you can use. Or simply, you can prepare your own data and train your own model.

For me, I tried both of them, and the CTPN model gave better results especially when the image contains large text.


Strategy for playing a board game with Minimax algorithm I want to build a player for the following game:
You have a board where position 1 is your player, position 2 is the rival player, -1 is a blocked cell and some positive value is a bonus. You can move up, down, left, or right. Also, each bonus has a timer until it disappears (number of steps). Furthermore, each move has a timeout limit.  At the end game, when at least one of the players is stuck, we check the scores and announce the winner.
Board example:
 -1 -1  0  0  0 -1 -1  -1
 -1  0 -1 -1 -1  0  0  340
 -1 -1  0  0  0 -1  0   0
 -1  0  0 -1  1 -1  0  -1
 -1  0  0 -1 -1  0  0   0
  0  0 -1 -1 -1  0  2  -1
  0 -1  0  0 -1  0  0  600
 -1 -1  0  0 -1 -1 -1  -1
  0 -1  0  0  0  0 -1  -1

I'm using the MiniMax algorithm with a time limit to play the game. If we got to children, we return $\infty$ for a player win, $-\infty$ for the rival win, and $0$ for a tie. If we got to a specific depth, we calculate the heuristic value. If we got timeout in some place in MiniMax, then we return the last calculated direction.  I'm trying to figure out a good strategy to win this game or get to a tie if no solution is possible.
What heuristic function would you define?
What I thought - four factors:

$f_A$ - The number of steps possible from each direction from the current position.
$f_B$ - The analytical distance from the center.
$f_C=\max_{b\in Bonus}\frac{X * I}{Y}$ - where $X$ is value of the bonus,  $I$ is $1$ if we can get to the bonus, before it disappears (otherwise $0$) and $Y$ is the distance between the bonus and the player.
$F_D$ - The distance between the players.
The final formula:
$$
f(s)=0.5\cdot(9-f_A(s))+0.2\cdot f_C(s)-0.2\cdot f_D(s)-0.1\cdot f_B(s)
$$

I'm not sure if it will be a good strategy for that game or not. How would you define the heuristic function? It should also be quick to calculate it because the game has a timeout for each move.
In order words, what will give us the best indication that our player is going to win/lose/tie?

I'm not familiar with your game so I can't tell you what a good heuristic woul be in your specific case, but I can give you some advice on how to look for a good heuristic function.
As a rule of thumb, the heuristic function for a MiniMax algorithm is best kept simple and efficient, so you can get deeper into the tree. But it depends on how costly it is to compute the heuristic function compared to simulating moves in the game.
If the heuristic takes longer than simulating a game move, it might be worth simplifying it so it runs faster and you can look ahead further. This often leads to more emergent and advanced strategies that are hard to express mathematically. An extreme example of a simple heuristic would be the current player score minus the opponent's score. Since scores only change when someone lands on a bonus tile, many paths you take down the tree would have equal value, so you need to be able to look ahead many moves to find a non-zero heuristic and be able to prune parts of the tree. But because the heuristic is so fast to compute, you can do this and discover more non-obvious strategies, simply by brute force simulation. This leads to more emergent behavior, and would tell you more about different ways to play the game (if that is your goal).
If simulating a game move takes longer than your current heuristic, it's probably not worth making the heuristic faster since the game simulation is the dominating factor in how deep you can go down the tree. This case is much more difficult because it means you have to find optimal strategies (and how to express them as a heuristic function) yourself. This is more of an art than a science - ask any chess grandmaster. I'd look up existing literature on the game and see if any existing strategies can be translated into a heuristic function. If there is no literature (e.g. because the game is new or unpopular), you could spend some time playing it yourself to discover what works. Alternatively (and perhaps more interesting), you could use a simple heuristic function with more emergent behavior, increase the time the MiniMax algorithm has to make moves, and play against your NPC opponent a few times to see what strategies it discovers. Or even have two NPCs with different heuristic functions play against each other. Then try to incorporate those into your final heuristic function. This is also a way to determine which heuristic function is better, if you have multiple candidates.
There are ways to optimize the heuristic function automatically using machine learning (specifically reinforcement learning), but it's probably not worth opening that can of worms in your case.


How to produce documents like factset blackline? Factset blackline reports essentially can compare two 10-Q SEC filings and show you the difference between the two documents. It highlights added items in green and removed items in red + strikethrough (essentially, it's a document difference, but longer-term I would like to run algorithms on the differences).
I don't care to change colors, but what I would like to do is to produce similar extracts that summarize addition and deletions.
Which AI/ML algorithm could do the same?

Financial information companies spend a very large amount of effort to do this kind of thing properly, and their models are generally proprietary, so the real answer is not something you'll be able to get on a public site.

Of course, you can achieve a very simple approximation with a literal diff on the texts, and a more sophisticated one by using out-of-the-box NLP or computer vision techniques, but you're probably not going to get close to the performance of factset or other competitors using those tools.