Comparing and studying Loss Functions I have a Deep Feedforward Neural Network $F: W \times \mathbb{R}^d \rightarrow \mathbb{R}^k$ (where $W$ is the space of the weights) with $L$ hidden layers, $m$ neurones per layer and ReLu activation. The output layer has a softmax activation function.

I can consider two different loss functions:

$L_1 = \frac{1}{2} \sum_i || F(W,x_i) - y||^2$  $
 \ \ \ $ and   $\ \ \ L_2  = -\sum_i log(F(w,x_i)_{y_i})$

where the first one is the classic quadratic loss and the second one is cross entropy loss.

I'd like to study the norm of the derivative of the loss function and see how the two are related, which means:

1) Let's assume I know that $|| \frac{\partial L_2(W, x_i)}{\partial W}|| > r$, where $r$ is a small constant. What can I assume about $|| \frac{\partial L_1(W, x_i)}{\partial W}||$ ?

2) Are there any result which tell you that, under some hypothesis (even strict ones) such as a specific random initialisation,  $|| \frac{\partial L_1(W, x_i)}{\partial W}||$ doesn't go to zero   during training?

Thank you

Let's first express a network of arbitrary topology and heterogeneous or homogeneous cell type arrangements as

$$ N(T, H, s) := \, \big[\, \mathcal{Y} = F(P_s, \, \mathcal{X}) \,\big] \\
   s \in \mathbb{C} \; \land \; s \le S \; \text{,} $$

where $S$ is the number of learning states or rounds, $N$ is the network of $T$ topology and $H$ hyper-parameter structure and values that at stage $s$ produces a $P$ parameterized function $f$ of $\mathcal{X}$ resulting in $\mathcal{Y}$. In supervised learning, the goal is that $F(P_s)$ approaches a conceptually ideal function $F_i$ as $s \rightarrow S$.

The popular loss aggregation norms are not quite as the question defines them. The below more canonically expresses the level 1 and 2 norms, which systematically aggregate multidimensional disparity between an intermediate result at some stage (epoch and example index) of training and the conceptual ideal toward which the network in training is intended to converge.

$$ {||F-\mathcal{Y}||}_1 = \sum{|F_i - y_i|} \\
   {||F-\mathcal{Y}||}_2 = \sqrt{\sum{(F_i - y_i)}^2} $$

These equations have been mutated by various authors to make various points, but those mutations have obscured the obviousness of their original relationship. The first is where distance can be aggregated through only orthogonal vector displacements. The second is where aggregation uses the minimum Cartesian distance by extending the Pythagorean theorem.

Note that quadratic loss is a term with some ambiguity. These are all broadly describable as quadratic expressions of loss.

Distance as described in the second expression above
RMS where the sum is divided by the number of dimensions
$\text{count}(i)$
Just the sum of squared difference by itself

Cross entropy is an extension of Claude Shannon's information theory concepts based on the work of Bohr, Boltzmann, Gibbs, Maxwell, von Neumann, Frisch, Fermi, and others who were interested in quanta and the thermodynamic concept of entropy as a universal principle running through mater, energy, and knowledge.

$$ S = k_B \log{\Omega} \\
   H(X) = - \sum_i p(x_i) \, \log_2{\, p(x_i)} \\
   H(p, \, q) = -\sum_{x \in \mathcal{X}} \, p(x) \, \log_2{\, q(x)} $$

In this progression of theory, we begin with a fundamental postulate in quantum physics, where $k_B$ is Boltzmann's constant and $\Omega$ are the number of microstates for the quanta. The next relation is Shannon's adaptation for information, where $H$ is the entropy in bits, thus the $\log_2$ instead of a natural logarithm. The third relation above expresses cross-entropy in bits for features $\mathcal{X}$ is based on the Kullback-Leibler divergence.  the p-attenuated sum of bits of q-information in .

Notice that $p$ and $q$ are probabilities, not $F$ or $\mathcal{Y}$ values, so one cannot substitute labels and outputs of a network into them and retain the meaning of cross entropy. Therefore level 1 and 2 norms are closely related, but cross-entropy is not a norm; it is the dispersion of one thing Cartesian distance aggregation scheme like them. Cross-entropy is remotely related but is statistically more sophisticated. To produce a cross-entropy loss function of form

$$ {||F-\mathcal{Y}||}_H = \mathcal{P}(F, y) \; \text{,} $$

one must derive the probabilistic function $\mathcal{P}$ that represents the cross entropy for two distributions in some way that is theoretically sound on the basis of both information theory and convergence resource conservation. It is not clear that the interpretation of cross entropy in the context of gradient descent and back propagation has caught up with the concepts of entropy in quantum theory. That's an area needing further research and deeper theoretical consideration.

In the question, the cross-entropy expression is not properly characterized, most evident in the fact that the expression is independent of the labels $\mathcal{Y}$, which would be fine if for unsupervised learning except that no other basis for evaluation is represented in the expression. For the term cross-entropy to be valid, the basis for evaluation must include two distributions, a target one and one that represents the current state of learning.

The derivatives of the three norms (assuming the cross entropy is properly characterized) can be studied for the case of $\ell$ ReLU layers by generalizing the chain rule (from differential calculus) as applied to ReLU and the loss function developed by applying each of the three norms to aggregate measures of disparity from optimal.

Regarding the inference in sub-question (1) nothing of particular value can be assumed about the Jacobians of level 2 norms from level 1 norms, both with respect to parameters $P$ or vice versa, except the retention of sign. This is because we cannot determine much about the correlation between the output channels of the network.

There is no doubt, regarding sub-question (2), that some constraint, set of constraints, stochastic distribution applied to initialization, hyper-parameter settings, or data set features, labels, or number of examples have implications for the reliability and accuracy of convergence. The PAC (probably approximately correct) learning framework is one system of theory that approaches this question with mathematical rigor. One of its practical uses, among others, is to derive inequalities that predict feasibility in some cases and produce more lucid approaches to learning system projects.


Is there a reason to use TensorFlow over PyTorch for research purposes? I've been using PyTorch to do research  for a while and it seems to be quite easy to implement new things with. Also, it is easy to learn and I didn't have any problem with following other researchers code so far.

However, I wonder whether TensorFlow has any advantage over PyTorch. The only advantage I know is, it's slightly faster than PyTorch.

In general, does TensorFlow have any concrete advantages over PyTorch apart from performance, in particular for research purposes?

In the past, I have used TensorFlow (1 and 2), Keras and PyTorch, so I will give an answer based on my experience. Currently, I use TF 2 and Keras (the version shipped with TF 2).

In my (but not only) opinion, TF 1 is really ugly and painful, given that it involves sessions, placeholders and, in general, you need to define the computational graph before executing anything (even the simplest programs). With TF 2, you don't need sessions and placeholders anymore, and this is a very big improvement because you don't need to think that you first need to define a computational graph and then feed it with the input. Essentially, in TF 2, you can write TF code and you can almost be sure that your TF 2 code will look like any other Python code.

TF and Keras have some bugs, but I think that also PyTorch must have some bugs (even though I don't remember having encountered them).

TF comes with TensorBoard. You can also easily use TensorBoard from Keras. About one year ago, I had written a blog post about visualization tools for PyTorch. At the time, the tools didn't seem as developed as TensorBoard, but there were already a few options.

Debugging in TF 2 isn't still perfect, but I would say that it has improved with respect to TF 1 (mainly because there is less boilerplate code). To debug in TF 2.0, I mainly use tf.print (which behaves similarly to Python's print function), but there may be more advanced tools (e.g. I think there's a TF debugging plugin for PyCharm), even though I've never used them. TensorBoard could also be used as a debugging tool (for example, to understand how your models are learning, etc).

The simplicity of PyTorch is comparable to that of Keras. TF now comes with an integrated version of Keras, which is optimized for TF, so TF code can look as simple as Keras code.

In general, my impression is that, if you need to, TF provides more flexibility, but it's been a while since I used PyTorch. Furthermore, with TF 2.0, the code looks simpler, more consistent and the framework is better organized. On the other hand, PyTorch may still be easier to learn (because it doesn't have the past of TF, i.e. there are still many TF code that uses sessions, etc.).

Both TF and PyTorch are supported by big companies (respectively, Google and Facebook), but TF is more mature and there are functionalities/libraries that exist for TF that still do not exist for PyTorch (AFAIK). I doubt that there is a functionality for PyTorch that doesn't also exist for TF or Keras.


How can I calculate the "mean best fitness" measure in genetic algorithms? I've just started to learn genetic algorithms and I have found these measurements of runs that I don't understand:

MBF: The mean best fitness measure (MBF) is the average of the best
fitness values over all runs.

AES: The average number of evaluation to solution.

I have an initial random population. To evolve a population I do:

Tournament selection
One point crossover.
Random resetting.
Age based replacement with elitism (I replace the population with all offsprings generated).
If I have generated G generations (in other words, I have repeated these four points G times) or I have found the solution, the algorithm ends, otherwise, it comes back to point 1.

Is the mean of the best fitness the mean fitness of all of each generations (G best fitness)?
MBF = (BestFitness_0 + ... + BestFitness_G) / G

I'm not English and I don't understand the meaning of "run" here.

The typical way you'll see a GA measured is that an algorithm with a population size of $N$ is ran $K$ times from new random seeds each time. That gives you $K$ total runs of the algorithm, each of which, at the end, had a final population of $N$ individuals. If you take the best of those $N$ from each run, you get $K$ "best" solutions found. The average fitness value of those $K$ solutions is your MBF.
AES here refers to the number of evaluations of the objective function the algorithm required. The reason for this is to provide a standardized amount of computation budget each algorithm used. Imagine if you compared algorithms on the basis of how many generations it took to find a good solution. In that case, I could just increase the population size of my algorithm by 100-fold or 1000-fold, and I'll probably look better. Same number of generations, but I gave my algorithm far more chances to find something good. Suppose I use wall-clock time. Now my algorithm might look better than yours just because I ran it on a much more powerful computer than you did.
The insight to make here is that GAs work by evaluating new search points. Let's just count how many times that happens. You can run a larger population for a shorter number of generations, or you can allocate the same amount of computation to running a longer time with fewer points each generation. What matters is just how many fitness function evaluations you exhausted in finding the answer.
For sudoku, you might have a fitness function that counted the number of rows, columns, or blocks that don't contain the correct digits of 1-9 and you minimize that function. You run your algorithm $K$ times from random seeds, and for each run, you record how many times you had to evaluate that fitness function before you found a $0$ (i.e., the puzzle was successfully solved). Average all $K$ of those counts, and that's your AES.
Generalizing a bit, you might calculate average evaluations to find some "good enough" solution, but the general concept is the same.


Why can't we use Google Translate for every translation task? Once a book is published in a language, why can't the publishers use Google Translate AI or some similar software to immediately render the book in other languages? Likewise for Wikipedia: I'm not sure I understand why we need editors for each language. Can't the English Wikipedia be automatically translated into other languages?

Google has achieved significant progress in AI translation, but it's still no-where near a qualified human translator. Natural language translation is already very challenging, adding domain knowledge to the equation is too much even for Google.

I don't think we have the technology to translate an arbitrary book from one language to another reliably.


Small size datasets for object detection, segmentation and localization I am looking for a small size dataset on which I can implement object detection, object segmentation and object localization.

Can anyone suggest me a dataset less than 5GB? Or do I need to know anything before implementing these algorithms?

There are various dataset available such as

Pascal VOC dataset: You can perform all your task with these.

size of the dataset is as  follows

ADE20K Semantic Segmentation Dataset: you can perform only segmentation here

COCO dataset: This is rich dataset but a size larger then 5 GB so you can try downloading using google colab in your drive and then make a zip file of data as less than 5 GB

You can download all these datasets using Gluoncv easily.here link.


How do self-driving cars construct paths? I wonder how self-driving cars determine the path to follow. Yes, there's GPS, but GPS can have hiccups and a precision larger than expected. Suppose the car is supposed to turn right at an intersection on the inner lane, how is the exact path determined? How does it determine the trajectory of the inner lane?

As you say, GPS is not precise enough for the purpose (until recently it was only accurate within 5m or so, since 2018 there are receivers that have an accuracy of about 30cm). Instead, autonomous vehicles have a multitude of sensors, mostly cameras and radar, which record the surrounding area and monitor the road ahead. Due to them being flat, mostly one colour, and often with lines or other markers on them, roads are usually fairly easy to spot, which is why most success has been made driving on roads as opposed to off-road. Once you know exactly where you are and where you want to go, computing the correct trajectory is then just a matter of maths and physics.

For an academic paper on the subject of trajectory planning see Local Trajectory Planning and Tracking of Autonomous Vehicles, Using Clothoid Tentacles Method.

It quickly becomes more complex when other road users and obstacles are taken into account; here machine learning is used to identify stationary and movable objects at high speed from the sensor input. Reacting to the input is a further problem, and one reason why there aren't any self-driving cars on the roads today.

This is all on driving automation level 2 and above; on the lower levels things are somewhat easier. For example, the latest model Nissan LEAF has an automatic parking mode, where the car self-steers, guided by camera images and sonar, but still requires the driver to indicate the final position of the vehicle. Apart from that, it is fully automatic.


Why are the landmark retrieval and facial recognition literature so divergent? Context and detail
I've been working on a particular image retrieval problem and I've found two popular threads in the literature:
Image retrieval (usually benchmarked with landmark retrieval datasets)

[x] Neural codes for Image Retrieval
[x] Particular object retrieval with integral max-pooling of CNN activations
[ ] Deep Image Retrieval: Learning global representations for image search
[ ] End-to-end Learning of Deep Visual Representations for Image Retrieval
[ ] Large-Scale Image Retrieval with Attentive Deep Local Features
[ ] Fine-tuning CNN Image Retrieval with No Human Annotation

Face recognition/verification:

[x] Facenet
[x] Deep Face Recognition
[x] SphereFace
[ ] Arcface

I'm still making my way through these lists and more (I've checked the ones I've looked at already) but I'm starting to get a sense that there's not much overlap in the techniques used, or the collective trains of thought in the research community. Here are the main points of divergence where I think both communities should be borrowing from each other.

Facial recognition seems to focus on getting embeddings to be as discriminative as possible by playing around with loss functions and training methods, whereas image retrieval seems to care more about ways of extracting feature descriptors from CNN pretrained backbones (types of pooling operations, which feature maps to look at, etc..).
Image retrieval has a considerable amount of work on what needs to happen after an embedding is obtained. Eg: dimensionality reduction, whitening + l2 norm, databise-side augmentation, query expansion, reranking etc
Facial recognition cares about keeping a minimum margin between non-matching faces in order to avoid mismatches, but I would think that should be imposed in image retrieval tasks as well (this is kind of a sub-point to my first point)

So to sum up: Why is it that facial recognition focuses on generating discriminative embeddings, while landmark retrieval focusses on generating rich "descriptors"? Why does landmark retrieval use this cool bag of tricks for database search while facial recognition just mentions kNN? Shouldn't all these considerations boost performance in either domain?

Landmark retrieval has photographs of landmarks that you need to find out. Consider the degrees of freedom for this, the landmarks can many different colours (more than humans' faces) and also the colour range is all over the place (a landmark may be blue or white or red). The shapes of the various landmarks will also vary.
Now, consider face recognition problem. All humans faces look alike morphologically. If you look at the colour, it is not as varied as landmark recognition.
Because of the inherent data in both the problems, the research focuses on the diverging lines of thought. Rich descriptors are good for landmarks because the data itself is very rich and mired with variation. On the other hand, discriminative features are more desirable for face recognition because faces are more similar and less rich in variation, so differentiating between is hard.
It is the requirement of the problem that steers research in diverging directions.


How should I predict which characters are going to die in a certain movie? How would one go about predicting which characters (actors) are going to die e.g. in the next Avengers movie?

To elaborate a bit, given all leaked scripts (fake or not), interviews of different actors and directors, contracts with different actors and directors. How would/should one go about predicting if an actor is going to die in the upcoming movie or not.

NOTE: I am not sure, but I think a similar effort has already been made for Game of Thrones.

This is more of a probability question. Having said that, based on each leaked script, you generate the possibilities of each of the characters dying (for example yes 0.2%, no 0.7%, unknown 0.1%) - you also give a weight for each leaked script based on on its perceived authenticity (from 0.1 to 1). You multiply the weight of each script by each of the possibilities and then you add the result for each possibility and each character across all scripts. So, for example, say you have 2 scripts. The first script has perceived authenticity of 20% (0.2) and the second script has perceived authenticity of 40% (0.4). In the first script, Character A has 50% (0.5) chance of dying, 20% (0.2) of not dying, and 30% (0.3) chance of unknown outcome. In the second script, character A has 20% (0.2) chance of dying, 50% (0.5) chance of not dying, and 30% (0.3) chance of unknown outcome. So the following applies for Character A:

Dying: (0.2 x 0.5) + (0.4 x 0.2) = 0.18 (18% chance of dying)
Not dying: (0.2 x 0.2) + (0.4 x 0.5) = 0.24 (24% chance of not dying)
Unknown outcome: (0.2 x 0.3) + (0.4 x 0.3) = 0.20 (20% unknown outcome)

The probability of not dying is 24%, which is the highest.

Note: I am not saying that probability has nothing to do with AI (in fact, it is very important in AI).


When did Artificial Intelligence research first start? When did research into Artificial Intelligence first begin?  Was it called Artificial Intelligence then or was there another name?

The earliest research into thinking machines was inspired by a confluence of ideas that became prevalent in the late 30s, 40s and early 50s (e.g. formal logic, automata, robots). Although the Turing test was proposed in the 1950s by Alan Turing, the work culminated back in the 1940s in the invention of the programmable digital computers, an abstract essence of mathematical reasoning. These ideas were inspired by a handful of scientists from a variety of fields who began seriously considering the possibility of building an electronic brain. The field of artificial intelligence research was officially founded as an academic discipline in 1956 during the Dartmouth workshop.
However, the concept of artificial beings is not new and it's as old as the Greek myths of Hephaestus and Pygmalion, which incorporated the idea of intelligent robots (such as Talos) and artificial beings (such as Galatea and Pandora).
See the following articles at Wikipedia for further details:

Artificial intelligence (AI)
History of artificial intelligence
Timeline of artificial intelligence


Can this tic tac toe program be considered AI? I coded a tic tac toe program, but I don't know if I can call it artificial intelligence.

Here's what I did.

There is a random player, which always makes random valid moves.

And then there is the AI player, which will receive input before every move, that input is the state of the board, and all the posibilities.
The AI, will try any move that it hasn't tried before. But if it knows every possibility, it will select the one that has the higher value.
This value is assigned by the outcome of the match, if the match was won ,+1, 0 for draw, -1 for lose.
Every move, will be stored in a database, or updated if it's known.

Eventually it will know every possible move.

I also added a threshold to compare the best moves, so it really select the best move. Example, two moves with a value of 100, the AI will keep trying them both, randomly until one has surpased the other by the threshold, say 50.

It takes about 20.000 games to make the AI perfect, it never loses a game, just draws and wins.

I'm new to AI, and I'm wondering, could this be really considered Artificial Intelligence?
And how is this different from a Neural Network approach? (I'm been reading about it, but I still don't quite get it.)

This is basically reinforcement learning. The state space contains your moves, and the value function are the value you store at the end. And your rewards are the end results. And you have episodic game. It is an AI method. Consider looking at value iteration, policy iteration, SARSA, Q-learning. The difference between neural network method and yours is you are not doing function approximation with neural network for the value function, you are doing tabular method.


Deep Reinforcement Learning: Rewards suddenly dip down I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using Prioritized Experience Replay. In the initial stage the behavior seems to be normal, i.e the agent is learning gradually. However, after a while the rewards suddenly go down by a lot. The TD erros also seem to be going up at the same time. I'm not sure how to interpret this problem.

My hypotheses are:

The policy network is overfitting
Some filters fail to activate thereby misrepresenting the state information

I would really appreciate if you guys could give me some tips to narrow down this problem debug it. Cheers.

Without digging into your diagnostics more deeply, on it's face this seem like a local optima issue. Assuming you are optimizing via GD, there are many local optima that a network or agent in this case can converge to and stay at which will cause symptoms like seeing above.

With that being said, assuming this is our issue, here are some things you can try:

Regularization, try adding dropout or L2 and see how that affects convergence and learning.
Adjust network architecture, number of layers, nodes, etc.
Try a different type of RL(Q learning for example), this will be dependent on your problem of course.
Adjust starting seed. Assuming you have a static seed used for weight initialization, you will always converge to the same solution. It could be as simple as adjusting the seed value.

If all these steps fail, you likely have a deeper issue at work, and I would suggest coming back after with some additional detail if this does not succeed.


Do we have anything like accuracy and loss in RNN models? I have a paper about trading which has been implemented with RNN on Tensorflow. We have about 2 years of data from trading. Here are some samples :

Date, Open, High, Low, Last, Close, Total Trade Quantity, Turnover (Lacs)

2004-08-25 , 1198.7, 1198.7, 979.0, 985.0, 987.95, 17116372.0, 172587.61

2004-08-26 , 992.0, 997.0, 975.3, 976.85, 979.0, 5055400.0, 49828.65

I need to predict the the future of trading (for example, the latest 10 days ). So, how can I make sure that my model is working correctly. Do we have any "accuracy" or "loss" like what we have in Deep Learning?

RNN's stand for Recurrent Neural Networks which is, in fact, Deep Learning.

There has to be a loss since you're dealing with supervised learning and the typical loss metrics used are the same as you would see in feedforward networks (usually binary cross-entropy), the main difference being loss would be calculated between the true label at a particular time stamp $(t)$ and the prediction made from the subset of the network until time-stamp $(t-1)$. This leads the loss to act on all timestamps.

Accuracy metrics also would be used in the same way such as Mean Square Error or L1. For more details you can go through this link.

Hope this was helpful!


Is it necessary to clear the replay memory regularly in a DQN when an agent plays against itself? I studied the article "Demystifying Deep Reinforcement Learning" extensively during the last days, while trying to implement the proposed algorithms myself.

My goal is to have an agent learn by playing a simple board game against itself using the methods of deep reinforcement learning. The algorithm described in pseudo code in the chapter called "Deep Q-learning Algorithm" is straight forward, but I cannot wrap my head around the fact that the replay memory only gets initialized once and never gets cleared.

Besides the obvious issue of growing too large for the available memory, there seems to be a more fundamental flaw. In the beginning the games will appear almost random, because the Q-function just has been initialized randomly. This means that bad moves will hardly be punished by the opponent. It makes sense to learn from those random encounters in the beginning, as we do not have any better data to learn from and this actually describes the observed behavior of the environment very well.

But when the agent improves, the old memorized plays will no longer be valuable, as certain moves will no longer be played by the opponent and therefore the old games will no longer reflect the current behavior of the environment.

That's at least my interpretation. Now I am wondering if this is simply missing from the pseudo code in the article or if my way of thinking is wrong in this regard. My question is, if we need to flush the replay memory regularly in the given setup and how often it should be done?

The purpose of the replay memory in DQN and similar architectures is to ensure that the gradients of the deep net are stable and doesn't diverge. Limiting what memory to keep and how far into the past it reaches is a problem that arises in in practical AI deployments. Throwing it away usually isn't the best decision because the agent's gradients may become unstable. Rather, keeping a window of the last T timesteps/gameplays can keep enough information while ensuring that the data is relevant as the agent improves. There are some additional methods which can improve the results as well:

Only keeping relevant timesteps (relevant depends highly on the application)
Removing timesteps from memory with probability relative to their age (the oldest has the highest probability to be removed)

It should be noted, that a larger replay memory is usually better than a smaller one to ensure stability but although memory is cheap it isn't infinite.


Is there a Hebb neural network? Is there a Hebb neural network? What kind of functions can it implement? Or, are there multiple "Hebb networks", that is, neural networks that learn in a Hebbian fashion?

In machine learning, the idea behind Hebbian learning is to strengthen (or weaken) the connection (the weight) between the neurons that have similar (or, respectively, dissimilar) outputs, where "similar" can be defined in different ways (e.g. it could be based on the sign of the output of the neurons).

Hebbian learning is a more biologically plausible way of learning than back-propagation, because it is a "local learning strategy" (you locally update the connections and not all the connections of the model at the same time), as opposed to back-propagation, which is a "global learning strategy" (where all connections are usually updated at once, given a "global error" of the network).

There are several neural networks (or models) that can learn in a Hebbian fashion: for example, the Hopfield network or Numenta's temporal memory.


Are all breakthroughs in AI and machine learning are due to increase in computational resources? I have read that all the math responsible for modern day machine learning and AI was already in place in 1900s but we did not have computational resources to implement those algorithms. So, is that true? And if it is, in what areas of machine learning the researchers work? And are all the future breakthroughs will be dependent only on increment of computational resources?

High-level answer: Increase in resources has been important in AI, and definitely was a factor with Deep Blue, but Machine Learning is a newer method that seems to produces more optimal results with less resources on problems of greater complexity.

Here is an article on AlphaGo's hardware: "Google reveals the mysterious custom hardware that powers AlphaGo"

Also an interesting analysis on Quora: "What hardware does AlphaGo run on? Is it customized hardware for best performance?"

Still pretty powerful systems, but I think the algorithms are as important as the computing resources, because all the hardware in the world won't help in the algorithms are poor.

Matthew Lai, creator of Giraffe Chess, said:

  "In the ensuing two decades [since Deep Blue], both computer hardware and AI research advanced the state-of-art chess-playing computers to the point where even the best humans today have no realistic chance of defeating a modern chess engine running on a smartphone."Source: TechExplore

which suggests that hardware and software are both important parts of the overall equation.


Why do popular object detecting models output heatmaps instead of coordinators of object directly? I think heatmap outputs of architectures like CenterNet, OpenPose, etc. can be changed to coordinator outputs, and loss functions like focal loss can be modified so they can deal with coordinators instead of heatmaps. Is there any particular reasons that researchers use heatmaps instead of coordinators?

To have a more powerful representational output. You can derive a bounding box from heatmap but not vice-versa. Also, in case of dense object detection it is hard to create bounding boxes for each object (people standing in front of each other).
That being the case, it is better to run a segmentation loss for these networks. It also leads to less confusion for the network. Also, it is a single shot creation of output. On the other hand, the bounding box approach will have the creation of a lot of bounding boxes then finding their objectness score and finally applying NMS (non-max suppression).
In regards to focal loss and use of bounding boxes, it is something that is a consequence of the heatmaps/activations that neural networks learns. They are built upon that and then regressed.


Do I need to rotate the masks, if I also rotate the images and the masks are generated from the input? I am training a neural network that takes an input (H, W, 3) and has the output of size (H', W', C). Now, to augment my dataset, since I only have 45k images, I am using the following in my custom data generator
def Generator():
img=cv2.imread(trainDir+'\'+imgpath)
img=tf.keras.preprocessing.image.random_rotation(img,20m,row_axis=0,col_axis=1,channel_axis=2)

output_mask=np.load(trainDir+'\'+maskpath)

yield(img/255-.5,output_mask)

Since I am rotating my input images and the output_masks are generated from information about the input (specifically, heat maps around the joint locations) do I also need to rotate the masks as well?

Yes! This is crucial.
If you rotate your input images for segmentation, you need to rotate the output masks as well. Otherwise the loss of your network will not be correctly calculated and your network will not learn how to generalize to rotated input images.
If you use keras, you can use two ImageDataGenerator classes, one for the images and one for the masks, with the same random seed and augmentation parameters. It looks something like this:
data_gen_args = dict(rotation_range=45)

image_datagen = ImageDataGenerator(**data_gen_args)
mask_datagen = ImageDataGenerator(**data_gen_args)

image_generator = image_datagen.flow(train,seed=SEED)
mask_generator = mask_datagen.flow(y_train,seed=SEED)
train_generator = zip(image_generator, mask_generator)

model.compile(...)
model.fit_generator(train_generator, ...)


Looking for comprehensive material on learning in mammals I am looking for a source that really discusses the classic rules of learning in depth. So classical conditioning, operant conditioning, imitation learning... I have found an infinite number of books that supposedly discuss these topics, but have not yet found anything that summarizes all the important findings on this topic. I am familiar with the basics, but I am explicitly interested in a detailed presentation of the topic.

Can anyone tell me a good source about the different forms of classical learning in mammals? I consider "Reinforcement Learning: An Introduction" comprehensive and detailed regarding RL. A comparable book on biological learning systems would be great. Sources in German and English would fit.

I recommend Animal Learning and Cognition: An Introduction by John Pearce.

As an introduction to the field, it's comprehensive. It covers all the topics you mentioned and more. Here is a free preview of the book.
It's well-regarded, having over 500 citations on Google Scholar.
It's relatively modern. Published in 2008, it discusses experimental findings that you won't be able to find in some of the other classics published in the 70s or 80s.

Although it does not specifically focus on mammals, all of the ideas extend to mammals.


Which approach should I use to classify points above and below a sine function $y(x) = A + B \sin(Cx)$? In a linear regression problem, a line can divide a data set into two categories. So, basically, points above the line belong to category 1, and points below the line belong to category -1.
However, my professor has asked me to write a C++ program in which the program will classify whether the data points lie above or below a sine function.
Let me explain a bit more. So, first, we will generate a data set $$D = \{(x_i, y_i) \} \label{0}\tag{0} $$ with random $x$ and $y$ coordinates, for example, according to this equation
$$y(x) = A + B \sin(Cx)\label{1}\tag{1},$$
where $A$, $B$, and $C$ are known.
The data points above the sine function will have a label 1 on them, and the points below the function will have -1.
Now, this data set $D$ in \ref{0} has to be fed to a C++ program. This C++ program has to somehow learn the curve separating the two data point categories. After training, the program will then classify some new query data points.
The key difficulty is that the program does not know in advance that the points were scattered around a sine curve. It does not know the values of $A$, $B$, or $C$ in equation \ref{1}. It also does not know that the curve is a sine curve.
Now, this is where I am stuck. I do not know if I need to use a neural network to solve this problem. If a neural network is to be used, then I presume that backpropagation will have to be used in some way. I can generate the data set and I can feed the data into the program.
Which approach (algorithm and model) should I use to solve this problem?
I have studied linear classification with the perceptron learning algorithm, but this sine-classifier stuff is a huge step-up for me. Another important thing is that I am not allowed to use any ready-made C++ libraries for Machine Learning. If a neural network solution is needed, then I will have to design the neural network from scratch. Note that I don't need any C++ code, but I am just looking for some guidance on how to approach this problem.

You can try using Fourier basis functions to transform your observable variables and then apply a general linear regression model. To clarify, if you have pairs of observables $(y_i, x_i)$ where $y_i$ is $i$-th output, and $x_i$ is $i$-th input then you can transform your input variable into vector
\begin{equation}
\phi = [1, \sin(x), \cos(x), \sin(2x), \cos(2x), \ldots, \sin(nx), \cos(nx)]
\end{equation}
where $n$ is some arbitrary number.
Then you will have a standard linear regression model
\begin{equation}
Y = A\Phi
\end{equation}
where $Y = [y_1, \ldots, y_n]$, $\Phi = [ \phi_1, \ldots, \phi_n]$ and $A$ is matrix of unknown parameters that you need to learn.


How to compute number of weights of CNN? How can we compute number of weights considering a convolutional neural network that is used to classify images into two classes :

INPUT: 100x100 gray-scale images.
LAYER 1: Convolutional layer with 60 7x7 convolutional filters (stride=1, valid
padding).
LAYER 2: Convolutional layer with 100 5x5 convolutional filters (stride=1, valid
padding).
LAYER 3: A max pooling layer that down-samples Layer 2 by a factor of 4 (e.g., from 500x500 to 250x250)
LAYER 4: Dense layer with 250 units
LAYER 5: Dense layer with 200 units
LAYER 6: Single output unit

Assume the existence of biases in each layer. Moreover, pooling layer has a weight (similar to AlexNet)

How many weights does this network have?

Keras Code

import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Conv2D, MaxPooling2D

model = Sequential()

# Layer 1
model.add(Conv2D(60, (7, 7), input_shape = (100, 100, 1), padding="same", activation="relu"))

# Layer 2
model.add(Conv2D(100, (5, 5), padding="same", activation="relu"))

# Layer 3
model.add(MaxPooling2D(pool_size=(2, 2)))

# Layer 4
model.add(Dense(250))

# Layer 5
model.add(Dense(200))

model.summary()

Calculating parameter number in a CNN is very straightforward.
CNN is composed of different filters which is essentially a 3d tensor. CNN weights are shared meaning they are used multiple times and reused in different locations. Each layer have n tensors each with dimension w * h * c where w = width, h = height, c = channels (the input channel size), therefore getting param_no = w * h * c * n. There is also a bias for each output channel, so param_no_bias = n. At the end the parameter number is calculated with: n * w * h * c + n. See more about this in hear: Article
The pooling layer does not have a weight, it only has hyperparameters. You may have confused the two. There are hyperparameters for the stride, the factor and etc. These are predefined and not trainable.
For a keras API, @pasabaporaqui has mentioned https://stackoverflow.com/a/35827171/4886927 which definitely works.
Hope I can help you.


Why would someone want to simulate emotional desicion-making within an AI? The Mario Lives! video (and its follow-up video, Mario Becomes Social!) showcases an AI unit that is able to simulate emotional decision-making within a virtual world, and can enter into "emotional states" such as curiosity, hunger, happiness, and fear. While this seems cool and exciting (especially for video game AI), I am confused about how this would be useful in real-world scenarios.

What would be the point of building autonomous actors that would behave based on these emotional states, instead of simply knowing what they should do (either by hardcoding in the rules, or learning the rules through machine learning)?

Humans have poor understanding of emotional rules. Probably every poster on here has experienced greatly misreading another individual emotionally. Further, people often don't act emotionally how they would expect themselves to act, for example we have all experienced frustration at someone else's irrational concerns and yet we are all guilty of holding irrational concerns of our own. This is the crux of why hard-coded emotional rules do not work - we do not have an understanding of what emotional 'rules' make someone feel real.

By moving towards autonomous state-based actors we move away from this issue. The actor's state transitions will of course be defined rules (hard-coded or learnt) but by abstracting the actor's emotional state from the specific context (e.g. specific actions trigger emotional state transitions which trigger responses, instead of a direct response to a specific action), the programmer prevents them-self from projecting their own beliefs/emotions/logic onto the actor.

Further, autonomous actors are more extendable. Consider an autonomous actor that is hard-coded to move towards 'upset' and 'angry' emotional states when experiencing 'pain'. Simply by associating a new world action with 'pain', one can trigger an emotional response from an autonomous actor that has not experienced that action before. When working in hard-coded emotional rules, this would not be possible.


How to estimate the cost and time to complete an AI Project If you are a freelancer, when a client asks to create a website we can easily measure how much the total cost is needed based on the requirements of the client. (the backend, UI/UX design, features, etc.). We can even measure the estimated time of completion.

What if a client asks you to make an AI project (image recognition, speech recognition, or NLP), how do you tell the client the estimated cost and time needed to complete the project in the beginning? because the results obtained can be very different for each data used

If you are a freelancer, when a client asks to create a website we can easily measure how much the total cost is needed based on the requirements of the client. (the backend, UI/UX design, features, etc.). We can even measure the estimated time of completion.

This is only the case when the full scope and design of the site is a well understood and relatively standard thing. With more experience, and perhaps a dedicated team, more complex sites and features can be addressed like this. It is just as common in my experience to view ongoing development of a site as a series of related projects, each of which can be estimated and costed as they become more feasible and closer to starting.

I have never seen a stand-alone site fully costed and estimated in my own work, because I work in-house on more complex features than this is possible for, but am aware this is a common approach, especially for agencies and freelancers. The difference is that the people involved are experts on producing well-defined work, and that in order to make a pre-estimated or even fixed price sale, they have put a lot of effort into risk management. Some of the risk management is contractual (e.g. "maximum of 3 major revisions" for web site design), some of it is in requirements gathering - often an initial pre-sales consultancy meeting is arranged, and/or a project proposal document generated where a lot of the possible misunderstandings between customer and developer are dealt with, and it is made clear what expectations are on both sides.

For a freelancer or agency, this de-risking is also spread over multiple projects as they will develop solutions that can be re-used including standardised contracts and templates for project planning etc.

There are also a few different project management approaches that deal with initial uncertainty in different ways. A "waterfall" approach of pre-planning as much in advance as possible requires very good knowledge of the work that will occur in later stages of a project, which of course depends heavily on earlier stages meeting all expectations of both the customer and developer. An "agile" approach will acknowledge that there are unknowns, and attempt to address them with some form of just in time planning. The problem with an agile approach is that "just in time" may be far too late if you need to budget up front (although it is still possible for instance by reducing scope to fit, provided that is agreed).

  What if the client asks you to make an AI project (image recognition, speech recognition, or NLP), how do you tell the client the estimated cost and time needed to complete the project in the beginning?

There is no difference in this respect just because it is an AI project. This would apply to any software project, including web development, where one or more of the following is true:

Scope of work is not clear
There are technical hurdles with unknown resolutions. I think it is likely you are focusing on this for your imagined AI project.
Customer expectations are not clear
Project success criteria is not measurable, or not known if achievable

As any software developer, if you are faced with project work with these features, you do some pre-project tasks focused on de-risking the project before attempting it. There are a few different approaches, than can be used in combination:

Gather more requirements, spend more time up front consulting on the project
Test assumptions before starting project. E.g. if you are not sure about quality of the data, take a look at it as early as possible. If you don't know if technology X can do Y, investigate it in advance.
Work on a timesheet basis, not fixed cost. Alternatively fix cost (or have a maximum), but leave scope of work open.
Have a flexible project plan, and re-estimate remaining work routinely.
Put any assumptions about customer support for the project, such as availablilty and quality of data, into project plan and in general form into your standard contract (e.g. something legal along the lines of "Project delivery depends on customer providing resources as outlined in the project plan document. If the resources are not provided in reasonable time, then the project may over-run or have additional costs, which will be covered by the customer, and not by the freelancer")

For additional up-front work on project feasibilty and scope, if you are concerned about risk to yourself as a freelancer (because customer expects these as a pre-sale effort), you could propose to bill for that as consultancy work, prior to starting project proper. You would likely need to produce something like a project proposal document in order that the paying customer received something for spending this money. Producing such a document might even show both you and the customer that the project is not feasible without additional work - in whch case you could note the steps needed to make it feasible (e.g. customer needs to source 10,000 more training images without the flaws noticed in existing images).


How many parameter would there be in a logistic regression model used to classify reviews into "good" or "bad"? Suppose we want to classify a review as good ($1$) or bad ($0$). We have a training data set of $10,000$ reviews. Also, suppose we have a vocabulary of $100,000$ words $w_1, \dots, w_{100,000}$. So the data is a matrix of dimension $100,000 \times 10,000$. Let's represent each of the words in the reviews using a bag-of-words approach over tf-idf values. Also, we normalize the rows such that they sum to $1$.
In a logistic regression approach, would we have $10,000$ different logistic regression models as follows:
$$ \log \left(\frac{p}{1-p} \right)_{1} = \beta_{0_{1}} + \beta_{1_{1}}w_{11} + \dots + \beta_{100,000_{1}}w_{100,000} \\ \vdots \\ \log \left(\frac{p}{1-p} \right)_{10,000} = \beta_{0_{10,000}} + \beta_{1_{10,000}}w_{11} + \dots + \beta_{100,000_{10,000}}w_{100,000}$$
So are we estimating $100,000 \times 10,000$ coefficients?

Nope! Our number of coefficients will be driven by the vocabulary, and we'll use each of those 10K samples to estimate values for those coefficients - so, 'just'  100K samples.  However, word frequency in human languages follows a Zipf distribution => most of those words will be rare, seen in only a few samples (=> won't even be able to determine whether they are useful or not, let alone get good values for coefficients).  For an application like this one, you would probably find most of the value from a few hundred words.


Effective algorithms for OCR I am using Google's OCR to extract text from images, like receipts and invoices.
Whare examples of techniques used to make sense of the text? For example, I would like to extract the date, name of the business, address, total amount, etc.
Before marking this question "too broad", if someone can please direct me to the right set of algorithms the industry uses for machine learning will be great.

Last semester I, along with my team, made a project on OCR.
Note: I am assuming the data set for your pictures has white
background with black (or some other dark text) on it.
These are the overview steps we followed:

Pre-processing includes grayscale conversion, noise reduction,
binarization and skew detection.
Next step was segmentation. This process extracts the individual
characters from the image. Histogram taken along the y-axis divided
the image into lines. This is followed by histogram along the x-axis
which divided them into words and further into characters.
At the end of the step, we used Savgol filter to smooth the curves of the histogram.
Next step was feature extraction. This is the most important step.
The accuracy of your code depends on how well your features are.
We used the following features:

Crossing: Counting number of transitions between foreground and background. We used
two diagonal lines, two horizontal and one vertical line. You can used any number you want.
Zoning: Whole character region is divided into 16 zones, and density of each zone is measured.
Projection Histogram: Each character has unique (almost) vertical and horizontal histogram signature.
Other features include number of endpoints in the character, number of loops and horizontal/vertical line count.

We used three different classification algorithms for our project. They were KNN (K-Nearest Neighbours), Artificial Neural Network (ANN) and Extra Tree classification. Their F1 score was 0.84, 0.82 and 0.77 respectively.
For training, you will need to find datasets. Many data sets for OCR was available online. Make sure you are using good ones.


How many parameters are being optimised over in a simple CNN? Okay so here's my CNN (simple example from a tutorial) along with some arithmetic to get the total number of free parameters.

We've got a dataset of 28*28 grayscale image (MNIST).

First layer is a 2D convolution using 32 3x3 kernels. Dimensionality of the output is 26x26x32 (kernel stride length was 1 and we have 32 feature maps of 26x26). Running parameter count: 288
Second layer is 2x2 MaxPool with a 2x2. Dimensionality of the output is 13x13x32 but then we flatten so we got a vector of length 5408. No extra parameters here.
Third layer is Dense. A 5408x100 matrix. Dimensionality of the output is 100. Running Parameter count: 540988
Fourth layer is Dense also. A 100x10 matrix. Dimensionality of the output is 10. Running Parameter count: 541988

Then we're supposed to do stochastic gradient descent on a 541988 parameter space!

That feels like a ridiculously big number to me. And this is meant to be the hello world problem of CNNs. Am I missing something fundamental in my understanding of how this is meant to work? Or maybe the number is correct but it's not actually a big deal for a computer to crunch?

In case it helps. Here is how the model was built in Keras:

def define_model():
    model = Sequential()
    model.add(Conv2D(32, (3,3), activation = 'relu', kernel_initializer = 'he_uniform', input_shape=(28,28,1)))
    model.add(MaxPooling2D((2,2)))
    model.add(Flatten())
    model.add(Dense(100, activation='relu', kernel_initializer='he_uniform'))
    model.add(Dense(10, activation='softmax'))
    opt = SGD(lr=0.01, momentum=0.9)
    model.compile(optimizer=opt, loss='categorical_crossentropy', metric=['accuracy'])
    return model

Neural networks can have a lot of different structures. CNNs can have a number of parameters that ranges from a few thousands to several millions.

In general you aim to increase the number of filters and reduce the first 2 dimensions, as you go deeper in the network.

So if you had Conv -> pool -> Conv -> pool -> ... , you could do for example first conv with kernel size = 5 and 8 filters and the second conv with kernel size = 5 and 16 filters. And both pools being (2,2).. But this is just an example.

In your network you start with a 28*28 image, and you use 32 3*3 filters. so number of parameters is (3*3 + 1) * 32 = 320.

In the dense layer you have as input a 13*13*32 and use a 100 FC layer. so n_parameters is (13*13*32 + 1)*100 which is 540900.

Then you get (100+1) * 10 FC which is 1010 more.

total = 320 + 540900 + 1000 which is 542230, as expected.

The +1 that shows up in every layer is the Bias neuron. Basically you add a bias neuron per output in a connection between 2 layers. In the FC1000 to the FC10 it is easy to understand, you have a bias per output neuron. In the Convolutional layers you have a bias term per each of the filters applied, so for each filter you have the filters weights plus 1 for the bias.

Apart from that you also had a small math mistake when adding the 288 at the beginning, you added 188. So your missing 242 parameters were: 32 from the conv layer, 100 from the first dense, 10 from the second dense and 100 from the sum. 100+100+32+10 = 242.


YOLO Architecture - kmeans clustering In YOLO, why use k-means clustering to determine bounding-box priors ?

Why if we use standard k-means with Euclidean distance, larger boxes generate more error than smaller boxes?

Why using IOU (Jaccard Index) can avoid/eliminate such error ?

How to derive d(box, centroid) = 1 - IOU(box, centroid) ?

Let's say you have a big box (0.999, 0.999) and a small box (0.001, 0.001).

Euclidean distance = sqrt((0.999 - 0.001)^2 + (0.999 - 0.001)^2) = 1.41.

Now, we look at a small box (0.001, 0.001) and an even smaller box (0.00000001, 0.000000001).

Euclidean distance = sqrt((0.001 - 0.00000001)^2 + (0.001 - 0.000000001)^2) = 0.00141

If you look at the number of the 2nd example, the Euclidean distance
thinks that the second box is pretty similar to the first box. But
this is not true.

IOU solves the problem because it measures the boxes relative to each
other. The result is a percentage, so it would give both boxes a
percentage near 0 which is correct.

  As for the Jaccard Index, why when the denominator equals the numerator, J(b1,b2)=1   ?

  I suppose we still have terms (w1*h1+w2*h2) in the denominator, so how can denominator equals numerator ?

J(b1,b2) = 1 iff intersection/(w1h1 + w2h2 - intersection) = 1 where
intersection = min_of_width * min_of_height

Assume w1h1 = w2h2, then intersection/(2 * w1h1 - intersection). But
intersection is just w1h1, then w1h1/(2 * w1h1 - w1h1) = w1h1/wh1h1 = 1


Is it possible to perform neuroevolution without a fitness function? My question is about neuroevolution (genetic algorithm + neural network): I want to create artificial life by evolving agents. But instead of relying on a fitness function, I would like to have the agents reproduce with some mutation applied to the genes of their offspring and have some agents die through natural selection. Achieve evolution in this manner is my goal.
Is this feasible? And has there been some prior work on this? Also, is it somehow possible to incorporate NEAT into this scheme?
So far, I've implemented most of the basics in amethyst (a parallel game engine written in Rust), but I'm worried that the learning will happen very slowly. Should I approach this problem differently?

You do not always need an explictly coded fitness function to perform genetic algorithm searches. The more general need is for a selection process that favours individuals that perform better at the core tasks in an environment (i.e. that are "more fit"). One way of assessing performance is to award a numerical score, but other approaches are possible, including:

Tournament selection where two or more individuals compete in a game, and the winner is selected.

Opportunity-based selection, where agents in a shared environment - typically with limited resources and chances to compete - may reproduce as one of the available actions, provided they meet some criteria such as having collected enough of some resource. I was not able to find a canonical name for this form of selection, but it is commonly implemented in artificial life projects.

A key distinction between A-life projects and GA optimisation projects is that in A-life projects there is no goal behaviour or target performance. Typically A-life projects are simulations with an open ended result and the developer runs a genetic algorithm to "see what happens" as opposed to "make the best game-player". If your project is like this then you are most likely looking for the second option here.
To discover more details about this kind of approach, you could try searching "artifical life genetic algorithms" as there are quite a few projects of this type published online, some of which use NEAT.
Technically, you could view either of the methods listed above as ways of sampling comparisons between individuals against an unknown fitness function. Whether or not a true fitness function could apply is then partly a matter of philosophy. More importantly for you as the developer, is that you do not have to write one. Instead you can approximately measure fitness using various methods of individual selection.

So far I've implemented most of the basics in amethyst (a parallel game engine written in rust), but I'm worried that the learning will happen very slowly. Should I approach this problem differently?

It is difficult to say whether you should approach the problem differently. However, the biggest bottlenecks against successful GA approaches are:

Time/CPU resources needed to assess agents.

Size of search space for genomes.

Both of these can become real blockers for ambitious a-life projects. It is common to heavily simplify agents and environments in attempts address these issues.


What is the purpose of argmax in the PPO algorithm? I'm kinda new to machine learning and still not too solid on math and particularly calculus. I'm currently trying to implement PPO algorithm as described in the spiningUp website :

This line is giving me a hard time :

What does the $\operatorname{argmax}$ mean, in this context? They are also talking about updating the policy with a gradient ascent. So, is taking argmax with respect to $\theta$ the same as doing:

where $J$ is the min() function?

In this case yes, $J$ is the big $\min$ expression and you apply Adam on that. But be careful because they say they do ascent, but automatic differentiation software usually minimizes given function so your $J$ would be $−\min(⋅)$.


Deepmind Spriteworld run_demo.py not found I'm trying to run the Deepmind Spriteworld demo described on the project's GitHub page, but I'm not finding run_demo.py in the distribution and the closest sounding file, demo_ui.py doesn't launch a UI  when run (tried both on Linux and Windows).

How should the Deepmind Spriteworld demo UI be launched?

I posted the question to the project's issue page, where the recommendation was to manually download and run run_demo.py. That worked for me.


Are FSA and FSTs used in NLP nowadays? Finite state automata and transducers are computational models that were widely used decades before in natural language processing for morphological parsing and other nlp tasks. I wonder if these computational models are still used in NLP nowadays for significant purposes. If these models are in use, can you give me some examples ?

Both are used, for example,  in the GATE framework, which is still widely used. I suspect that this also applies to many other applications.
I would think that many recent academic publications are now on other approaches, as FSAs and FSTs are fairly established and mature technologies, but I've been out of academia for a while now.


Is it possible to use a feed-forward neural network to predict the actions in reinforcement learning? I have done a lot of research on the internet about Reinforcement Learning and I found encountered methods of Reinforcement Learning: Q-Learning and Deep Q-Learning. And I have developed a vague idea of how these two work.

Before I knew anything about Reinforcement Learning this is how I thought it would work:

Suppose I have 2 virtual players in a game who can shoot each other, one of them is a decent playing hard-coded/pre-coded AI, and the other one is the player I want to train (to shoot the other player and dodge his bullets), the aim of the game would be to get the greatest net score (shots you hit minus shots you took) within 1 minute (a session), and you only have 20 bullets.

You have 3 actions, move left-right (0 = maxleftspeed, 1 = maxrightspeed ), jump (0=don't jump, 1=jump), shoot (0 = don't shoot, 1 = shoot).

What I thought was, you could create a basic Feed-Forward Neural Network use the enemy's position and his bullet(s)'s position(s) and bullet direction(s) for the input layer and the action being taken will be given by the (3 nodes in the) output layer.

The untrained player starts off with a randomized algorithm, then (for back-propagation) at the end of each session it modifies one of the parameters by a bit in the neural network, and a new session is started with the slightly modified NN. If this session ends with more points than the previous session, it keeps the changes and makes more changes towards that direction, otherwise, the changes are redone, or possibly reversed. I would visualise this as gradient descent similar to that of Supervised learning.

So my questions are:

Is something like this already out there? What is it called?
If nothing like this is out there, could you give me any tips to optimize this method or point out any key points I should keep in minds while carrying this out?
Since I have written this game, I have control over the speed of the actions, but if I did not, I know this AI would take ages to learn, so is there any way to make the learning faster while still keeping the basic idea in mind?
How exactly is this different from deep Q-learning (if it is)?

Thanks in advance!

Is something like this already out there? What is it called?

There are three algorithms that have similarities to yours, but are not identical:

NEAT, which is an evolutionary method that mutates neural networks and compares their fitness across populations of variations.
The Cross Entropy Method, which trains a single, initially randomly behaving, neural network based on the outcomes of its best performances.
REINFORCE is a policy-gradient method that trains a neural network after completing games based on the results of every game. It is very similar to your idea, except that it includes the maths/logic to sample a full gradient of all NN parameters in a single go. It does not require making a test change to a single parameter, but can calculate the full error gradient at once. Usually, however, to reduce noise and effects of correlation within a game, multiple gradients are collected from different games and averaged.

The main similarity in each case is that you are performing a policy search, and training based on selecting outcomes that are closer to your goal after trials. A policy-based approach means that your NN represents the choice of action directly. A policy is a function that converts knowledge of the state of an environment into an action.

  If nothing like this is out there, could you give me any tips to optimize this method or point out any key points I should keep in minds while carrying this out?

Your suggested method is not exactly the same as any of the suggested algorithms, and may have a problem with performance due to this:

  The untrained player starts off with a randomized algorithm, then (for back-propagation) at the end of each session it modifies one of the parameters by a bit in the neural network, and a new session is started with the slightly modified NN. If this session ends with more points than the previous session, it keeps the changes and makes more changes towards that direction, otherwise, the changes are redone, or possibly reversed. I would visualise this as gradient descent similar to that of Supervised learning.

This single parameter change search is likely to be slow, probably unusably so for any reasonably-sized neural network. Effectively you would be performing a single parameter step for a very noisy gradient descent algorithm, per full play of the game. All of NEAT, CEM and REINFORCE are likely to perform much better than this (and none of them are state-of-the-art).

  is there any way to make the learning faster while still keeping the basic idea in mind?

Instead of sampling the total reward at the end of each game and generating a single parameter's gradient, use a deep reinforcement learning method that works with full gradients.

You might find REINFORCE interesting, as it logically the idea closest to yours, and it leads to some state-of the art methods such as A3C, DDPG, PPO . . . the main problem with all of these being that the maths is hard to understand, even though the implementations can be simple you may be left scratching your head as to why they even work. However, if you are happy with just seeing the policy gradient theorem as some maths which completes your idea, then it is there for the taking.

Otherwise, if your goal is to implement something you understand the internals of fully, I would suggest that you search for a tutorial on deep Q learning, maybe something that implements DQN, as the maths and intuition behind it are more accessible.

  How exactly is this different from deep Q-learning (if it is)?

Both your suggested method and Q learning can probably be considered "deep reinforcement learning".

The main differences are:

Q learning is a value-based method, and yours is a policy-based method
Q learning is off-policy (it learns best overall actions even whilst exploring other possibilities) whilst your approach is on-policy (the actions it learns to take are the same as the actions it takes during exploration)
Deep Q learning updates all the parameters of the NN on each time step, whilst you will be updating a single parameter per episode. That makes Q learning far more sample efficient than your suggested approach.


How to improve neural network training against a large data set of points with varying magnitude I am currently using TensorFlow and have simply been trying to train a neural network directly against a large continuous data set, e.g. $y = [0.014, 1.545, 10.232, 0.948, ...]$ corresponding to different points in time. The loss function in the fully connected neural network (input layer: 3 nodes, 8 inner layers: 20 nodes each, output layer: 1 node) is just the squared error between my prediction and the actual continuous data. It appears the neural network is able to learn the high magnitude data points relatively well (e.g. Figure 1 at time = 0.4422). But the smaller magnitude data points (e.g. Figure 2 at time = 1.1256) are quite poorly learned without any sharpness and I want to improve this. I've tried experimenting with different optimizers (e.g. mini-batch with Adam, full batch with L-BFGS), compared reduce_mean and reduce_sum, normalized the data in different ways (e.g. median, subtract the sample mean and divide by the standard deviation, divide the squared loss term by the actual data), and attempted to simply make the neural network deeper and train for a very long period of time (e.g. 7+ days). But after approximately 24 hours of training and the aforementioned tricks, I am not seeing any significant improvements in predicted outputs especially for the small magnitude data points.

Figure 1

Figure 2

Therefore, do you have any recommendations on how to improve training particularly when there are different data points of varying magnitude I am trying to learn? I believe this is a related question, but any explicit examples of implementations or techniques to handle varying orders of magnitude within a single large data set would be greatly appreciated.

I think there is no special method for training the neural network in large datasets. But I can add some suggestion for you:

1) Use the convolutional neural network for this dataset.

2) You can use huber loss instead of squared loss and see what happens.

3) See if you have enough small magnitude training data.

Also, please define your problem in more detail (like what those images represent and what you want to predict, what are the features etc).


Is machine learning required for deep learning? The answers to this Quora question say it's OK to ignore machine learning and start right away with deep learning.

Is machine learning required or is useful for understanding (theoretically and practically) deep learning? Can I start right away with deep learning or should I cover machine learning first? In what way machine learning useful for deep learning? (leave the mathematics part - I'm ok with it).

Deep learning is part of machine learning.

You will miss out useful information if you ignore machine learning.
You are ok to start your work in machine learning with deep learning and neural networks. You have to start somewhere and starting with a strong and successful method is resaonable, especially if you need to be able to produce good results quickly.
You will learn essential machine learning stuff while reading about deep learning.
The deep learning tutorials and other learning materials you will be reading may not be telling you that what you are learning also applies to other machine learning methods but you will be learning lot's of stuff that applies more generally. You will be studying some machine learning whether you want to or not.
If you have plenty of time a more broad view will help understanding. Still, there is no need to wait with deep learning to after mastering some other methods.
Broader knowledge helps you to relate and memorise concepts and be more aware of potential issues, especially issues that are rarely discussed in the deep learning community. Such knowledge and experience will be most useful when trying to apply deep learning to new problems or if trying to make substantial changes.


What are the state-of-the-art meta-reinforcement learning methods? This question can seem a little bit too broad, but I am wondering what are the current state-of-the-art works on meta reinforcement learning. Can you provide me with the current state-of-the-art in this field?

One of the most recent papers on meta-RL is meta-Q-learning. This paper introduces Meta-Q-Learning (MQL), a new off-policy algorithm for meta-reinforcement learning (meta-RL). MQL builds upon three simple ideas.

Q-learning is competitive with state-of-the-art meta-RL algorithms if given access to a context variable that is a representation of the past trajectory.

Using a multi-task objective to maximize the average reward across the training tasks is an effective method to meta-train RL policies.

past data from the meta-training replay buffer can be recycled to adapt the policy on a new task using off-policy updates

Experiments on standard continuous-control benchmarks suggest that MQL compares favorably with state-of-the-art meta-RL algorithms.
I think that other references to other work on meta-RL are present in the experiments part of the MQL paper.


Is the mean-squared error always convex in the context of neural networks? Multiple resources I referred to mention that MSE is great because it's convex. But I don't get how, especially in the context of neural networks.

Let's say we have the following:

$X$: training dataset
$Y$: targets
$\Theta$: the set of parameters of the model $f_\Theta$ (a neural network model with non-linearities)

Then:

$$\operatorname{MSE}(\Theta) = (f_\Theta(X) - Y)^2$$

Why would this loss function always be convex? Does this depend on $f_\Theta(X)$?

Answer in short:  MSE is convex on its input and parameters by itself. But on an arbitrary neural network it is not always convex due to the presence of non-linearities in the form of activation functions. Source for my answer is here.


What is the difference between latent and embedding spaces? In general, the word "latent" means "hidden" and "to embed" means "to incorporate". In machine learning, the expressions "hidden (or latent) space" and "embedding space" occur in several contexts. More specifically, an embedding can refer to a vector representation of a word. An embedding space can refer to a subspace of a bigger space, so we say that the subspace is embedded in the bigger space. The word "latent" comes up in contexts like hidden Markov models (HMMs) or auto-encoders.
What is the difference between these spaces? In some contexts, do these two expressions refer to the same concept?

Embedding vs Latent Space
Due to Machine Learning's recent and rapid renaissance, and the fact that it draws from many distinct areas of mathematics, statistics, and computer science, it often has a number of different terms for the same or similar concepts.
"Latent space" and "embedding" both refer to an (often lower-dimensional) representation of high-dimensional data:

Latent space refers specifically to the space from which the low-dimensional representation is drawn.
Embedding refers to the way the low-dimensional data is mapped to ("embedded in") the original higher dimensional space.

For example, in this "Swiss roll" data, the 3d data on the left is sensibly modelled as a 2d manifold 'embedded' in 3d space. The function mapping the 'latent' 2d data to its 3d representation is the embedding, and the underlying 2d space itself is the latent space (or embedded space):

Synonyms
Depending on the specific impression you wish to give, "embedding" often goes by different terms:

Term
Context

dimensionality reduction
combating the "curse of dimensionality"

feature extraction feature projection feature embedding feature learning representation learning
extracting 'meaningful' features from raw data

embedding manifold learning latent feature representation
understanding the underlying topology of the data

However this is not a hard-and-fast rule, and they are often completely interchangeable.


What are the main differences between a language model and a machine translation model? What are the main differences between a language model and a machine translation model?

The simple language model will give you the probability of a sequence of tokens(sentence) for that language. So lets say if you have trained a model for English language your model can give you the probability for any random english sentence.
Consider some sentence
$X$ $=$ "the quick brown fox jumps over the lazy dog" $=$  $x_1 \ x_2 \ x_3 \ ... \ x_n$
model will give you $P(X)$
Moreover if the model has been trained properly, in the following scenario, where
$X$ $=$ "the quick brown fox jumps over the lazy dog" $=$  $x_1 \ x_2 \ x_3 \ ... \ x_n$
$Y$ $=$ "dog brown quick fox over the jumps lazy the" $=$  $y_1 \ y_2 \ y_3 \ ... \ y_m$
model will always give $P(X) > P(Y)$, as it has learnt the structure of language.
On the other hand Machine translation model gives you the conditional probability of the next token given your source sentence and partial-target sentence.
So if
$X = $ "I am a student" and $Y = $ "je suis" $=y_1, \ y_2$
model will give you $P(y_3 | X,y1_,y_2)$ .
where $X$ is a source sentence and $Y= y_1, y_2$ is a partial target sentence. The probability of word/token "étudiant" would be maximum among all words of vocabulary.


What is a support vector machine? What is a support vector machine (SVM)? Is an SVM a kind of a neural network, meaning it has nodes and weights, etc.? What is it best used for?

Where I can find information about these?

I find the chapter on machine learning from Russell & Norvig is a pretty good place to start with SVMs. I think this is Chapter 18?

One way to understand an SVM is as a kind of neural network, but this is not usually an intuitive approach for a beginner (unless your NN knowledge is already quite good).

A better way to understand SVMs is as consisting of three simple ideas rolled into one algorithm. Here's an attempt at a "For Dummies" answer though:

Maximum Margin Classification. SVMs are usually used to find a pattern in a set of data. Often, the data allow an infinite set of possible patterns that are all equally descriptive. For example, maybe The relationship is "Lives within 5 miles of a Coast -> Income High". It's easy to imagine that this pattern is just as good as "Lives within 5.0001 miles of a Coast -> Income High" or "Lives within 4.999 miles of a Coast -> Income High". There might actually be a lot more play than that in the data (e.g. 3 miles might work out too). If all these are equally good, then the maximum margin idea says you should pick the one that's "in the middle" of the data. So maybe all values between 5.5 and 4.8 are equally good. In that case, we might pick 5.15 (in the middle). This example is super simplified. Real world data would have a lot more variables, and the idea of "in the middle" ends up being a little more complex, but this is the intuition. It turns out that finding the maximum margin pattern is easy when the patterns are linear. That is, when they can be represented by drawing straight lines through a plot of the dataset.
Projection into higher dimensions. This one needs a bit of math to visualize. Consider a dataset consisting of a circular pattern (for instance, maybe the pattern is that higher incomes are found in the middle of the city). There is no linear relationship that captures this pattern. That is, you can't draw a straight line through the data, and say something meaningful about all the values on one side or the other. However, if you add a new feature to your data that is the square of the original coordinates, it's easy to find such a pattern. Basically, if you pre-compute "circular" functions of the original data, you can add them to the dataset, and then find a pattern that is a linear function of this new feature. This idea generalizes: if you compute a complex enough function of your original data, and then apply the maximum margin idea, you can learn any pattern you like. The problem is that it's slow: adding more features makes it take longer to find the patterns you want.
The Kernel Trick. The thing that made SVMs useful was the kernel trick: finding the maximum margin didn't depend on anything except the product of the coordinates of the various points. It turned out that this product could be computed first, and then run through certain functions to produce a problem that was identical to the one you'd get by first adding extra features and then doing the multiplication. However, computing the problem this way didn't require adding any new features! This made SVMs one of the first reliable, well understood, and fast methods for finding non-linear patterns in data.

Hope that provides a starting point. Consider reading Russell & Norvig as a next starting point, or Bishop if you want to go deeper.


What does the notation $\mathcal{N}(z; \mu, \sigma)$ stand for in statistics? I know that the notation $\mathcal{N}(\mu, \sigma)$ stands for a normal distribution.
But I'm reading the book "An Introduction to Variational Autoencoders" and in it, there is this notation:
$$\mathcal{N}(z; 0, I)$$
What does it mean?
picture of the book:

It means that $z$ has a (multivariate) normal distribution with 0 mean and identity covariance matrix. This essentially means each individual element of the vector $z$ has a standard normal distribution.


What is the most computationally efficient genetic algorithm? In researching genetic algorithms, it seems that there are various methods of selection and other operator methods that can significantly change the performance. For example, this picture contains some of the methods that could be used:

Presumably, you can mix & match these operators to optimize whatever problem you're trying to solve.
What most people care about is how many iterations it takes to get to the target. This is understandable, but I've seen things that would be inefficient in real systems such as:

using sort on the current population $\mathcal{O}(n \log n)$ and picking the first n members for the mating pool

appending to a constantly resizing slice to create a mating pool instead of rewriting on the current array

What I am looking for is, how can I arrive at the target using the least amount of computation and memory possible. The number of iterations and the time taken to get there is still a secondary priority.
It's possible that it may be the process of picking the right operators, but what I am also considering is how parallelizable the implementation could be as well.

First of all, for a lot of realistic problems, the fitness function evaluation is usually orders of magnitude greater in complexity than the rest of the genetic algorithm. This is not always true, but often is true (e.g. imagine trying to optimise a simulation where you need to execute the simulation completely to obtain the fitness). So optimising the GA itself might only be helpful when the fitness function is lightweight (e.g. it's a mathematical function as used in some competitions).
While your diagram shows a lot of operators, it doesn't show variations of the GA itself. Here are two:

generational: selects a new population at each iteration after performing mutation and crossover on the existing population to generate offspring (which are usually more in number than the population size)
steady state: maintains a single population and replaces only individuals during each iteration with their offspring according to selection/replacement rules, so uses less memory than a traditional GA

If you really wanted to maximise efficiency in the GA, a steady state approach would probably work best combined with tournament selection, as tournament selection does not require any sorting. The mutation/crossover operators you list are likely to all be quite efficient, but simpler methods are likely to to be the most efficient (e.g. bit flip mutation+1 point crossover). For a realistic problem, however, problem-specific operators usually work best.


How to deal with a small amount of labeled samples? I'm trying to develop skills to deal with very small amounts of labeled samples (250 labeled/20000 total, 200 features) by practicing on Kaggle "Don't Overfit" dataset (Traget_Practice have provided all 20,000 Targets). I've read a ton of papers and articles on this topic, but everything I've tried did not improve simple regularized SVM results (best accuracy was 75 and AUC was 85) or any other algorithm result (LR, K-NN, NaiveBayes, RF, MLP). I believe the result can be better (on Leaderboard they go even over AUC 95)
What I've tried without success:

Remove outliers  I've tried to remove 5%-10% outliers with EllipticEnvelope and with IsolationForest.

Feature Selection I've tried RFE (with or without CV) + L1/L2 regularised LogisticRegression, and SelectKBest (with chi2).

Semi-Supervised techniques I've tried co-training with different combinations of two complementary algorithms and :100-100: split features. I've also tried LabelSpreading, but I don't know how to provide the most uncertain samples (I tried predictions from other algorithms, but there were many mislabeled samples, hence was unsuccessful).

Ensembling Classifiers StackingClassifier with all possible combinations of algorithms and this also didn't improve the result (the best is the same as SVM accuracy 75 and AUC 85).

Can anyone give me advice on what I'm doing wrong or what else to try?

In that particular competition, you can try using GAN to generate new data or adding noise to existing data. You can also use K-means algorithm. You can try using a smaller network and remove bias. May be you can use logistic regression to compare the result. You can also use a PCA method.


Why does Batch Normalization work? Adding BatchNorm layers improves training time and makes the whole deep
model more stable.  That's an experimental fact that is widely used in machine
learning practice.
My question is - why does it work?
The original (2015) paper motivated introduction of the layers by stating
that these layers help fixing "internal covariate shift". The rough idea is
that large shifts in the distributions of inputs of inner layers makes training
less stable, leading to decrease in the learning rate and slowing down of the
training.  Batch normalization mitigates this problem by standardizing the
inputs of inner layers.
This explanation was harshly criticized by the next (2018) paper -- quoting
the abstact:

... distributional stability of layer inputs has little to do with the success of BatchNorm

They demonstrate that BatchNorm only slightly affects the inner layer inputs
distributions.  More than that -- they tried to inject some non-zero
mean/variance noise into the distributions. And they still got almost the
same performance.
Their conclusion was that the real reason BatchNorm works was that...

Instead BatchNorm  makes the optimization landscape significantly smoother.

Which, to my taste, is slightly tautological to saying that it improves
stability.
I've found two more papers trying to tackle the question: In this paper
the "key benefit" is claimed to be the fact that Batch Normalization biases
residual blocks towards the identity function. And in this paper that it
"avoids rank collapse".
So. Is there any bottom line? Why does BatchNorm work?

To some extend, it get rid of low intensity numerical noise. Condition properties of the optimization problem is always an issue, i suspect BatchNorm alleviate this instability.


Using 3D Points as Inputs to a Neural Net I am currently looking to use a neural network to classify gestures. I have a series of Dx,Dy,Dz readings that represent the differences across the three axes made during the gesture. About 10 movements for each example of the gesture. Basically a 10x3 matrix and then classify the training data into about 15 classes. I plan to use a CNN classifier to do this because, while the time domain is relevant this problem the difference in the movements can be differentiated when presented with as a discrete matrix.

I'm used to using images with a neural net so I instinctively want to just convert the matrices into a 2D tensor and feed them into a CNN, but I was wondering if there was a better way to do this? For example, I have seen 1D tensors passed to a fully connected neural network for classification which seems like it could be more appropriate for this data input type?

Any tips on general architecture would be really appreciated as well!

Thanks!

A few thoughts :

10x3 matrix for each example is really a small amount of data. FCNN could do a good job on that.
As a result, I'm not sure CNN is appropriated. The smallest dimension is 3, so it'll force you to have really small kernels.
Have you thought about LSTM? since that your data is sequential, LSTM may be useful. However, I'm not sure they could be really effective on this little amount of data. but it could be nice to try.


Can ML/AI understand incomplete constructs like humans? We have AI's predicting images, predicting objects in an image. Understanding audio, meaning of the audio if it is a spoken sentence.

In humans when we start seeing a movie halfway through, we still understand the entire movie (although this might be attributed to the fact that future events in movies have a link to past events). But even if we see a movie by skipping lots of bits in-between we still understand the movie.

So can a Machine Learning AI do this? Or do humans have some inherent experiences in life which makes AI incapable of performing such a feat?

I believe that an AI can understand much more of a movie than we do. As you said, sometimes a detail at the beginning of the movie is the key to understanding the final outcome. The problem is that the storyline distracts us. In the case of an AI, he would just be waiting for the movie to complete because he has all the details at all times.

Imagine that an AI is by your side watching the movie. He will not feel emotion like you. He is only analyzing the images, colors, soundtrack, main characters and is extracting from the lines, whatever he can, to understand the movie. What would it be to understand, in your eyes? To be able to generate a synopsis or to compare the film with some reality or criticizes the life of the human?

An artificial superintelligence would have the task of overcoming us. We could pass a movie and let her explain why we cried watching the movie, because we were surprised, and everything. Since an artificial superintelligence would have the role of not being distracted by the film but analyzing all possible content, try to analyze human reactions and spit out an analysis that could leave you reflecting on life for a few days.

But this is an artificial superintelligence that I imagine may one day exist, but I believe it is far away.

Something real and what we can build is an AI that can extract the genre from the movie and maybe even generate a synopsis with a spoiler like "X character dies."

Another factor that also makes it easier for us to understand movies is our huge film base. We watch so many American cliches movies, that most are already very predictable: "The main character does not usually die," "When the villain is close to killing the main character, something surprising happens and he is prevented," ...


Learning similarities between customers and offers representation I am interested in a framework for learning the similarity of different input representations based on some common context. I have looked into word2vec, SVD and siamese networks, all of which are similar to what I want.

For example, suppose we have some customers we are sending different advertisements to, and I would like to create a system to map offers to customers. I am thinking in the lines of creating a customer representation, and a representation of the offers, and feeding them in parallel to a neural network that has a label of whether they acted on the advertisement or not. The idea is that I should be able to locate the best offer for any customer given these representations.

I have looked into siamese networks and word2vec, both are close to what I want. The problem differs slightly in that for the siamese networks, it tends to be identical parallel networks, which I don't want because my inputs are not equivalent. And for word2vec the vectors tend to be in the same domain, while I want to apply this in a more general setting.

If anyone has any resources on a similar problem statement, I would be very interested in it.

The idea is that I should be able to locate the best offer for any customer given these representations.

I think you need a Recommender System. As you want to map the offers to customers based one their representation you can check Content-Based Recommender System.

The method is by taking pattern from a customer history and try to find similarities with the new offers. You can use many techniques for a recommender system, from a simple TF-IDF or Deep Learning for more complex problems.


If the point of the ResNet skip connection is to let the main path learn the residual relative to identity, why are there convolutional skips? In the original ResNet paper they talk about using plain identity skip connections when the input and output of a block have the same dimensions.

When the input and output have different dimensions they propose two options:

(A) Use an identity mapping padded with zeros to make up for the extra dimensions

(B) Use a "projection".

which (after some digging around in other people's code) I see as meaning: do a convolution with a 1x1 kernel with trainable weights.

(B) is confusing to me because it seems to ruin the point of ResNet by making the skip connection trainable. Then the main path is not really learning a "residual" relative to an identity transformation. So at this point, I'm no longer sure how to interpret the intent or expected effect of this type of block. And I would think that one should justify doing it in the first places instead of just not putting a skip connection there at all (which in my mind is the status-quo before this paper).

So can anyone help explain away my confusion here?

Well, I found an answer that satisfies me.

The zero-padded identity is not ideal. Suppose we're mapping from 64 channels to 128 channels. Then the zero-padded identity will map to an output where half of the channels are the same as the inputs, and the other half are all zeros. So that means the main path is learning a residual for half of the output channels, and learning a mapping from the ground up for the other half of the channels.

Now, the alternative is to use option (B) which is the single 1x1 convolution. After reading their ResNetV2 paper I realised that they really double down on this concept of maintaining a "clean" alternative path all the way up and down the network. So then the question is, what's "cleaner"?: a block of two convolutions of the form 3x3x64 then 3x3x128, or a shortcut with a 1x1x128 convolution? The shortcut is a tempting choice here. And in fact, in the original paper they show empirically that the convolutional shortcut is better than the identity shortcut. Maybe it might be worth running a test to see if the convolutional shortcut is better than no shortcut at all, but until I decide to run such a test, I'll presume the authors would have mentioned it. The other thing worth noting is that there are only a few places in the whole network where the dimensions need to be increased, so maybe the impact is small.


Does the bag-of-visual-words method improve the classification accuracy? I'm a beginner in computer vision. I want to know which option among the following two can get better accuracy of image classification.

SIFT features + SVM
Bag-of-visual-words features + SVM

Here's a reference: https://www.mathworks.com/help/vision/ug/image-classification-with-bag-of-visual-words.html.

Bag-of-visual words (BOVW) was classicly used in computer vision before the introduction of neural networks or some more advanced classical techniques, such us VLAD or Fisher Vectors. In any case, it is a good technique to use, but it is not the state-of-the-art today, and I won't recommend you use it for a real-life project.


Can a CNN be trained incrementally? Like our human brain, we can first learn (train) the handwriting 0 and 1. After the traing (and test) accuray is good enough, we only need to study (traing) the hardwriting 2,  Instead of cleaning all of learned memory, and relearn handwriting data 0, 1, and 2 at the same time.

Can CNN do the same thing? Can CNN learn something new, but keep the previous memory? If yes, the efficiency could be high.  Right now, I have to give all of data at the same time, the efficiency is very very low.

You are looking for incremental (or online) learning.

A CNN can be trained incrementally. For example, in the paper Incremental Learning of Convolutional Neural Networks, the authors propose an incremental learning algorithm (inspired by AdaBoost and Learn++, which is another incremental learning algorithm for supervised learning of neural networks) for CNNs.

However, note that incremental learning is a challenging task, given the  stability-plasticity dilemma: a completely stable model, in order to keep being stable, will attempt to preserve the existing knowledge, so it will not learn new knowledge; similarly, a completely plastic model, in order to keep being plastic, it will keep forgetting previously acquired knowledge so that to learn new information.