Should neural nets be deeper the more complex the learning problem is? I know it's not an exact science. But would you say that generally for more complicated tasks, deeper nets are required?

Deeper models can have advantages (in certain cases)

Most people will answer "yes" to your question, see e.g. Why are neural networks becoming deeper, but not wider? and Why do deep neural networks work well?.

In fact, there are cases where deep neural networks have certain advantages compared to shallow ones. For example, see the following papers

The Power of Depth for Feedforward Neural Networks (2016) by Ronen Eldan and Ohad Shamir
Benefits of depth in neural networks (2016) by Matus Telgarsky.
Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks (2017) by Safran and Shamir
Optimal approximation of piecewise smooth functions using deep ReLU neural networks (2018) by Petersen and Voigtlaender

What about the width?

The following papers may be relevant

Wide Residual Networks (2017) by Sergey Zagoruyko and Nikos Komodakis
The Expressive Power of Neural Networks: A View from the Width (2017) by Zhou Lu et al.

Bigger models have bigger capacity but also have disadvantages

Vladimir Vapnik (co-inventor of VC theory and SVMs, and one of the most influential contributors to learning theory), who is not a fan of neural networks, will probably tell you that you should look for the smallest model (set of functions) that is consistent with your data (i.e. an admissible set of functions).

For example, watch this podcast Vladimir Vapnik: Statistical Learning | Artificial Intelligence (AI) Podcast (2018), where he says this. His new learning theory framework based on statistical invariants and predicates can be found in the paper Rethinking statistical learning theory: learning using statistical invariants (2019). You should also read "Learning Has Just Started" – an interview with Prof. Vladimir Vapnik (2014).

Bigger models have a bigger capacity (i.e. a bigger VC dimension), which means that you will more likely overfit the training data, i.e., the model may not really be able to generalize to unseen data. So, in order not to overfit, models with more parameters (and thus capacity) will also require more data. You should also ask yourself why people use regularisation techniques.

In practice, models that achieve state-of-the-art performance can be very deep, but they are also computationally inefficient to train and they require huge amounts of training data (either manually labeled or automatically generated).

Moreover, there are many other technical complications with deeper neural networks, for example, problems such as the vanishing (and exploding) gradient problem.

Complex tasks may not require bigger models

Some people will tell you that you require deep models because, empirically, some deep models have achieved state-of-the-art results, but that's probably because we haven't found cleverer and more efficient ways of solving these problems.

Therefore, I would not say that "complex tasks" (whatever the definition is) necessarily require deeper or, in general, bigger models. While designing our models, it may be a good idea to always keep in mind principles like Occam's razor!

A side note

As a side note, I think that more people should focus more on the mathematical aspects of machine learning, i.e. computational and statistical learning theory. There are too many practitioners, who don't really understand the underlying learning theory, and too few theorists, and the progress could soon stagnate because of a lack of understanding of the underlying mathematical concepts.

To give you a more concrete idea of the current mentality of the deep learning community, in this lesson, a person like Ilya Sutskever, who is considered an "important and leading" researcher in deep learning, talks about NP-complete problems as if he doesn't really know what he's talking about. NP-complete problems aren't just "hard problems". NP-completeness has a very specific definition in computational complexity theory!


Is it possible to have a dynamic $Q$-function? I am trying to use Q-learning for energy optimization. I only wish to have states that will be visited by the learning agent, and, for each state, I have a function that generates possible actions, so that I would have a Q-table in form of a nested dictionary, with states (added as they occur) as keys whose values are also dictionaries of possible actions as keys and Q-values as values. Is this possible? How would it affect learning? What other methods can I use?

If it is possible and okay, and I want to update the Q-value, but the next state is one that was never there before and has to be added to my nested dictionary with all possible actions having initial Q-values of zero, how do I update the Q-value, now that all of the actions in this next state have Q-values of zero?

I have a function that generates possible actions, so that I would have a Q-table in form of a nested dictionary, with states (added as they occur) as keys whose values are also dictionaries of possible actions as keys and q-values as values. Is this possible? How would it affect learning? What other method can I use?

(Disclaimer: I provided this suggestion to OP here as an answer to a question on Data Science Stack Exchange)

Yes this is possible. Assuming you have settled on all your other decisions for Q learning (such as rewards, discount factor or time horizon etc), then it will have no logical impact on learning compared to any other approach to table building. The structure of your table has no relevance to how Q learning converges, it is an implementation detail.

This choice of structure may have a performance impact in terms of how fast the code runs - the design works best when it significantly reduces memory overhead compared to using a tensor that over-specifies all possible states and actions. If all parts of the state vector could take all values in any combination, and all states had the same number of allowed actions, then a tensor model for the Q table would likely be more efficient than a hash.

  I want to update Q-value, but the next state is one that was never there before and has to be added to my nested dictionary with all possible actions having initial Q-values of zero; how do I update the Q-value, now that all of the actions in this next state have Q-values of zero.

I assume you are referring to the update rule from single step Q learning:

$$Q(s,a) \leftarrow Q(s,a) + \alpha(r + \gamma \text{max}_{a'} Q(s',a') - Q(s,a))$$

What do you do when you first visit $(s,a)$, and want to calculate $\text{max}_{a'} Q(s',a')$ for the above update, yet all of $Q(s',a') = 0$, because you literally just created them?

What you do is use the zero value in your update. There is no difference whether you create entries on demand or start with a large table of zeroes. The value of zero is your best estimate of the action value, because you have no data. Over time, as the state and action pair are visited multiple times, perhaps across multiple episodes, the values from experience will back up over time steps due to the way that the update formula makes a link between states $s$ and $s'$.

Actually you can use any arbitrary value other than zero. If you have some method or information from outside of your reinforcement learning routine, then you could use that. Also, sometimes it helps exploration if you use optimistic starting values - i.e. some value which is likely higher than the true optimal value. There are limits to that approach, but it's a quick and easy trick to try and sometimes it helps explore and discover the best policy more reliably.


Transfer learning to train only for a new class while not affecting the predictions of the other class I am basically interested in vehicle on the road.

YoloV3 pytorch is giving a decent result.

So my interested Vehicles Car Motorbike Bicycle Truck and bus, I have a small vehicles being detected as truck.

Since the small vehicle is nicely being detected as truck. I have annotated this small vehicle as a different class.

Though, I could add an extra class say 81th class, since the current YoloV3 being used is trained on 80 classes.

81th class would contain the weight of the truck, I would freeze the weights such that the rest of the 80 classes remain unaltered and only the 81th class of this new data gets trained.

The problem is the final layer gets tuned according to the prediction of all the classes it learns.

I was not able to find any post that could actually mention this way of preserving the predictions of the other classes and introducing a new class using transfer learning.

The closest, I was able to get is this post of
Weight Sampling Tutorial SSD using keras

Its mentioned in

Option 1: Just ignore the fact that we need only 8 classes

This would work, and it wouldn't even be a terrible option. Since only 8 out of the 80 classes would get trained, the model might get gradually worse at predicting the other 72 clases in the second paragraph.

Is it possible to preserve the predictions, of the previous pre trained model while introducing the new class and use transfer learning to train only for that class?

I Feel that this is not possible, would like to know your opinion. Hope someone can prove me wrong.

Even if you want to re-train your model for just one new class you will have to prepare your training data such that it includes all or most of the classes which you want to predict. Most of the times last two layers of a network have the data of number of labels which are to be predicted and that should always be sum of the number of classes you already trained on and the number of classes you want to add for training.


How do I solve the knapsack problem using the hill climbing algorithm? I need to solve the knapsack problem using hill climbing algorithm (I need to write a program). But I'm clueless about how to do it.

My code should contain a method called knapsack, the method takes two parameters, the first is a 2xN array of integers that represents the items and their weight and value, and the second is an integer that represents the maximum weight of the knapsack. I can assume that the initial state is an empty knapsack, and the actions are either putting objects in the Knapsack or swapping objects from in and out of the knapsack.

I think there are at least three points that you need to think before implement Hill-Climbing (HC) algorithm:

First, the initial state. In HC, people usually use a "temporary solution" for the initial state. You can use an empty knapscak, but I prefer to randomly pick items and put it in the knapsack as the initial state. You can use a binary array to save the "pickup status" of each item such that if $a_i=1$ then you pick the $i$-th item and if $a_i=0$ then you don't pick the $i$-th item.

The next is the heuristic function. A function that evaluate your current state. Actually, you can define it freely (like the initial state) as long as the function can evaluate which state is better than the others. You can use a simple one:

$$
h =
    \begin{cases}
      -1, & \text{if}\ \sum (w_i \times a_i) > \text{MaxWeight} \\
      \sum (v_i \times a_i), & \text{otherwise}
    \end{cases}
$$

with $w_i$ and $v_i$ is the weight and the value of the $i$-th item respectively. One of the disadvantage of this function is if you have more than one invalid state (a state with the total weight is more than the limit), they have the same value, which is $-1$.

The last is the method to generate the neighbors of the current state. Once again, you are also free to define the method that you will use. I think your proposed actions are good enough:

reverse the status of an item (1 to 0 or 0 to 1). That means I add or reduce the item on the current state.
swap the status two items that have different value. Formally swap $a_i$ and $a_j$ with $ i \neq j$ and $ a_i \neq a_j$. That means I swap the item that I picked before with the item that I didn't pick before

Then, after you define everything you can run the algorithm as explained in Wikipedia:

currentNode = startNode;
loop do
    L = NEIGHBORS(currentNode);
    nextEval = -INF;
    nextNode = NULL;
    for all x in L
        if (EVAL(x) > nextEval)
            nextNode = x;
            nextEval = EVAL(x);
    if nextEval <= EVAL(currentNode)
        //Return current node since no better neighbors exist
        return currentNode;
    currentNode = nextNode;


In Alpha(Go)Zero, why is the policy extracted from MCTS better than the network one? I've read through the Alpha(Go)Zero paper and there is only one thing I don't understand.
The paper on page 1 states:

The MCTS search outputs probabilities π of playing each move. These search probabilities usually select much stronger moves than the raw move probabilities p of the neural network fθ(s);

My question: Why is this the case? Why is $\pi$ usually better than $p$? I think I can imagine why it's the case but I'm looking for more insight.
what $\pi$ and $p$ are:
Say we are in state $s_1$. We have a network that takes the state and produces $p_1$ (probabilities for actions) and $v_1$ (a value for the state).
We then run MCTS from this state and extract a policy $\pi(a|s_1) = \frac{N(s_1,a)^{1/\tau}}{\sum_b N(s_1,b)^{1/\tau}}$.
The paper is saying that $\pi(-|s_1)$ is usually better than $p_1$.

The most important word for answering your question from that quote from the paper is probably the word "usually": These search probabilities usually select much stronger moves than the raw move probabilities $p$ of the neural network. It's not always going to be true, but more often than not / most of the time / "on average", we intuitively expect it to be true. In theory there could be pathological cases, especially with low MCTS iteration counts and/or a poorly-trained network, where it may not be true. But even if we just find it to be true most of the time, that can be good enough for the algorithm to work well in practice.
Recall that in the Selection phase of MCTS, Alpha(Go) Zero always selects the action $a$ that maximises the following expression for a state $s$ when traversing the tree (see supplementary material of Alpha Zero paper):
$$Q(s, a) + C(s) \frac{P(s, a) \sqrt{N(s)}}{1 + N(s, a)}$$
The $Q(s, a)$ values here are the value estimates resulting from the MCTS search; these are the results produced by our tree search, and these generally become more and more accurate the longer our tree search runs. If we run our tree search for an infinite amount of time, we expect these to converge to the true minimax values.
These same $Q(s, a)$ values are the things that can cause the final distribution of visit counts $\pi$ to shift away from the prior neural network distribution $P$. Suppose that we have a hypothetical case where for all the different actions $a$, the $Q(s, a)$ values end up staying identical. In this (unlikely) hypothetical case, the visit counts would continue getting distributed proportionally to $P(s, a)$, causing the $\pi$ distribution to stay equal to the $P$ one (barring minor potential differences due to $\pi$ being derived from discrete, integer visit counts which may not be able to exactly replicate the real-valued probabilities in $P$). Only if the $Q(s, a)$ values resulting from a (hopefully smart!) tree search algorithm give us reason to shift away from the prior distribution $P$ do we actually really start shifting away from it.


Is possible to train a robot or AI to prune fruit trees? I live in a rural area where there is a growing necessity for people with knowledge to prune Pear trees, this process is crucial for the industry, but as people go to the big cities, this skill is being lost, and in a few years there will be no one to do it.

I wanna know if it is possible to train a robot using AI to do this, and want would it take to make this work!

Keep in mind that this would be viable only in an "industrial" way. The trees all have approximately the same size and are disposed in a certain preset way (distance between each other, height, etc).

I am not an expert in robotics (definitely not in pear trees pruning either) but I will try to give some hints to partially answer and also to help reframe the problem a bit. On overall I'll give already an answer: it is most likely possible, but also most likely not convenient.
Problem statement
First things first: in general the rule is that machine learning should be applied when a task can't be automatised otherwise. Before asking if an AI can be trained to solve a task one should always think how to automatise a task using a rule based system. This is important especially because during the process of thinking how to automatise the task you will realise that some steps can be perform without an expert, while others can't be performed without any supervision. Let's brake down your task in subtasks: a system that prune trees should be (at least) capable of:

Moving between trees
Selecting branches to cut
Cut the selected branches

Selecting the branch to cut is probably the step that requires most of the know how and expert supervision and for which a machine learning component might be suitable. Moving instead is a perfect example of a subtasks that could be tackled at different levels. Creating a machine able to anticipate other objects movements and avoid them in real time definitely require to train an AI component, but when you say that the environment is highly structured (trees disposed in a grid) this make me think that maybe some hand coded rules would do the trick without bothering machine learning.
Theoretical tools
Once you have understood which subtasks your machine should be capable of solving, you can start dig on the theoretical feasibility of them. Following the same order as in the previous paragraph:

Self-driving is a widely studied topics, the algorithms used to train robotic vacuum cleaners to move automatically in a house could be applied straight away to the problem of training an agent to move between trees.

Selecting the right branches involves mostly computer vision. This sub-task should actually be also dived again into sub tasks: detecting branches from other objects and select which ones should be cut. Nevertheless, the field is quite huge and training two models able to perform both actions is, in my opinion, doable.

Cutting a branch is probably harder than driving in this situation, because of the small movements that might be required to reach branches positioned in difficult spots. Anyway, it is possible to train robots to perform fine-grained movements (for a funny example see: Robot learn to flip pancakes).

Again it depends also on how high your expectation are about the final machine/system. Should the system have a surgical precision or could it risks to brake few extra branches in hard situations? Obviously the higher the expectations the harder it would be to make everything work harmoniously.
Resources required
Last but not least you also need to understand if what you're trying to do is feasible in reality and not just in theory. A big problem when it comes to use machine leaning is that these models can be trained only with huge amount of data.

Train an artificial agent to move in an environment can be done by reproducing the environment in an artificial simulator, which is good news. A single guy with a laptop could potentially do the job.

Collecting data to train a model able to detect branches on photo and then select which one should be cut will be highly tedious, and also expensive because data need also to be labelled by experts. Which means that some experienced people will have to take photos, in the order of tens of thousands at least, and write for each photo (using a software) which are the branches they will cut if they were working for real on that tree. I strongly doubt that a dataset like this already exists.

Training a robotic arm to cut branches will also be challenging. Despite the fact that also in this case simulators could be leveraged, the task is inherently harder and this comes with bigger difficulties (for example in designing a proper reward function if using reinforcement leaning). Concretely this mean more time to spend in research and testing.

Consider also that the success of the final model trained for each subtask would be not guaranteed at all, reason why I said at the beginning that training a system with AI modules would be probably not convenient, and that the best thing is always to try to create a rule based system first.


Does feature scaling have any benefits if all features are on the same scale? By scaling features, we can prevent one feature from dominating the decisions of a model. For example, say heights (cm), and age (years) are two features in my data. Since range of heights is larger than of years, a trained model could weight importance of heights much more than years. This could result in a poor model in return.
However, say that all of my features are binary, they take a value of either 0 or 1. In such a case, does feature scaling still have any benefits?

If all you features are binary, then, you don't need to apply normalization on them. Since their values are on the same scale already.


Reinforcement Learning with asynchronous feedback I want suggestions on literature on Reinforcement Learning algorithms that perform well with asynchronous feedback from the environment. What I mean by asynchronous feedback is, when an agent performs an action it gets feedback(reward or regret) from the environment after sometime not immediately. I have only seen algorithms with immediate feedback and asynchronous updates. I don't know if literature on this problem exists. This is why I'm asking here.
My application is fraud detection in banking, my understanding is when a fraud is detected it takes 15-45 days for the system to flag it as a fraud sometimes until the customer complains the system doesn't know its fraud.
How would I go about designing a real-time system using reinforcement learning to flag transactions that are fraudulent or normal?
Maybe my understanding is wrong, I'm learning on my own if someone could help me I would be grateful.
The reason I'm looking at reinforcement learning instead of supervised learning is, it's hard to get ground truth data in the banking scenario. Fraudsters are always up-to-date or exceeding the state of the art in fraud detection. So I've decided that reinforcement learning would be an optimal direction to look for solutions to this problem.

I have been looking for a while into pretty much precisely the problem you describe (including the same application domain), but haven't been able to find much.

The most obvious, mathematically "correct" solution would be to simply delay your standard Reinforcement Learning update rule (of whatever algorithm you choose to implement) by 45 days; if it still wasn't reported as a fraud then, assume it was genuine. This leads to some problems though;

Need lots of memory to store experiences that were not yet used for updates
Learning only starts after a significant delay, in which you don't learn anything at all yet and likely therefore run a suboptimal policy for a long time
Very slow to adapt to new strategies of the fraudsters
What to do with people who already report fraud cases earlier, like after 10 days? Delay them for the full 45 days anyway, or trigger updates immediately (and potentially mess up the ordering in which experiences actually occurred)?

A quick and dirty "solution" is the following;

When a transaction occurs, immediately trigger a learning update under the assumption that it was a genuine transaction (for example, with a reward of R = +1).
If that transaction is later reported as a fraud, trigger an additional update (with same (state, action) pair), but with the negation of the reward that was previously assigned erroneously on top of the normal negative reward for a fraudulent case. For example, if you would normally give R = +1 for genuines, and R = -100 for frauds, give a reward of R = -101 now. This reward will not correct for the previously assigned wrong reward in completely the right way (potentially wrong position in sequence of updates, discounting due to gamma and maybe lambda depending on algorithm used, etc.), but it should be somewhat close (especially if gamma and lambda are close to 1.0).

This is certainly not ideal, has very little theoretical basis and probably breaks quite a bit of Reinforcement Learning theory, but at least it is efficient in terms of computation and memory and in my experience it works alright in practice.

If you're using off-policy RL algorithms, you can use Experience Replay buffers (very popular in DQN-style things in Deep RL these days, but can also be used in tabular RL / RL with linear function approximation etc.). If you already have historical data generated through some non-RL policy in the past (which is typically the case in fraud detection / banking applications, they do have lots of data even if they don't always share it), you can use this to fill your experience replay buffer. In the case of the first solution (at the top of this answer), this can be used for training during the initial delay of 45 days.

Since you expect there to be concept drift though (fraudsters adapting their behaviour over time), you'll want to be careful with experience replay. Old data will become less useful.

A very different style of solution is to assume that you have a team of human experts available who can investigate a very small portion of incoming transactions relatively quickly. This tends to be true for large companies in practice ("investigating" often means a phone-call to a card holder). This enables you to generate accurate feedback for a small portion of your data more quickly, so that you can also do Reinforcement Learning with much less of a delay (albeit only on a small percentage of your experience).

You can read more about this idea in the following paper (disclaimer: I'm an author on it):

Dennis J.N.J. Soemers, Tim Brys, Kurt Driessens, Mark H.M. Winands, and Ann Nowé (2018). “Adapting to Concept Drift in Credit Card Transaction Data Streams Using Contextual Bandits and Decision Trees”. In Thirtieth Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-18), pp. 7831-7836. AAAI Press.

Apart from that idea you might furthermore find it interesting for references to other related work, links to data you could use, etc.

I feel like it should be possible to extend the existing Reinforcement Learning theory with proper algorithms that can properly;

Take immediate learning steps with an assumed, default, potentially incorrect reward, and
Retroactively correct for previous incorrect updates if the reward turns out to be something else than previously assumed in hindsight.

I'm not aware of existing literature in which this is done though, and it certainly doesn't seem trivial; it will require starting pretty much from "first principles" (e.g., Bellman operator).

Intuitively, I also expect doing this completely correctly will always require a significant amount of memory (memory of all previous transactions of a card holder, such that state-action pairs can be re-generated if necessary). Banks likely already store that kind of data anyway for every customer, so it may not be a problem in practice.

If anyone's planning to work on this, feel free to contact me, I'll likely be happy to collaborate :D


Is there a methodology for splitting up annotated orthophotos into smaller photos that retain the original bounding boxes? I'm trying to train an object detection algorithm (i.e. YOLOv4 Scaled, Faster R-CNN) on data taken from large orthophotos. Let's say I have one class, and I label the entire orthophoto with bounding boxes. After labeling, is there a way to slice up the entire image into individual photos of specified pixel sizes (i.e. 416x416 pixels) while keeping the bounding boxes? I can easily slice the photo into the specified dimensions, but the problem I am having is keeping the bounding boxes in these new images.
That way, I would not be exhausting my GPU's memory requirements.

You can reduce your photo size and scale the corresponding boxes to the new dimensions (416x416).
Or if you want to go with your technique, you can slice the image and then, check if the bounding box lies in the slice, then, reorient it according to the slice you took.
Take a look at albumentations library for this.


Why evolutionary training of neural networks is not popular? Evolutionary algorithms are mentioned in some sources as a method to train a neural network (finding weights, not hyperparameters). However, I have not heard about one practical application of such an idea yet.
My question is, why is that? What are the issues or limitations with such a solution that prevented it from practical use?
I am asking because I am planning on developing such an algorithm and want to know what to expect and where to put most attention.

The main evolutionary algorithm used to train neural networks is Neuro-Evolution of Augmenting Topoloigies, or NEAT. NEAT has seen fairly widespread use. There are thousands of academic papers building on or using the algorithm.

NEAT is not widely used in commercial applications because if you have a clean objective function, a topology that is optimized for gradient decent via backpropogation, and an implementation that is highly optimized for a GPU, you are almost certainly going to see better, faster, results from a conventional training process. Where NEAT is really useful is if you want to do something weird, like train to maximize novelty, or if you want to try to train neurons that don't have cleanly decomposable gradients. Basically, you need to have any of the usual reasons you might prefer an evolutionary algorithm to hill-climbing approaches:

You don't have a clean mapping from loss function to individual model components.
Your loss function has many local maximia.


How can a specific connectivity pattern be stored in an optimally compact representation? I am interested in optimizing the memory capacity of an AGI. Given a specific complex input an AI can create a simplified model. This is a problem that can be solved using sparse coding [1]. However, this solves only the problem of encoding and not the maintenance of online representations—in cognitive terms: the state of mind.

A default cognitive model of short-term memory can be separated in three different stages:

Encoding → Maintenance → Retrieval

One solution is to use specialized hardware [2][3], but I am interested in software approaches to this problem and I would thus like to emphasize that it is the digital representation, which I am most interested in.

With the exception of qubits, the smallest possible representation are binary digits. However, additional architecture is required to represent phase spaces (i.e. floating point precision memory) and higher-order representations maybe include arrays and dictionaries. (Optimizing these are trivial... or simply to be postponed until needed, according to Knuth).

How can a specific connectivity pattern be stored in an optimally
compact representation?
Is there an implementation with concrete example code?
What is the state-of-the-art?*

* I will not specify "real-time" here, but the context is humanoid AGI.

[1] Papyan, V., Romano, Y., & Elad, M. (2017). Convolutional Neural Networks Analyzed via Convolutional Sparse Coding. Journal on Machine Learning Research, 18(83): 1–52. arXiv:1607.08194
http://jmlr.org/papers/volume18/16-505/16-505.pdf

[2] LeGallo et al. (2018). Mixed-precision in-memory computing. https://www.nature.com/articles/s41928-018-0054-8

[3] IBM. (2018). IBM Scientists Demonstrate Mixed-Precision In-Memory Computing for the First Time; Hybrid Design for AI Hardware. https://www.ibm.com/blogs/research/2018/04/ibm-scientists-demonstrate-mixed-precision-in-memory-computing-for-the-first-time-hybrid-design-for-ai-hardware/

Solution

In learning systems, the output of learning is a set of parameters. These parameters can be a tensor in the case of a CNN, a representation of a directed graph in the case of recursive network designs, or any other structure holding the results of training.

The general case of compactly storing the result of training involves four process elements. These elements can be combined into one algorithm if needed.

Encoding of ordinals
Lossless conversion of IEEE floats
Serialization of the structure
Lossless compression of the stream

Upon retrieval, the elements are reversed.

Decompression of the stream
Parsing to reassemble structure
Expansion of IEEE float representations
Decoding of ordinals

The encoding of ordinals requires the mapping of values to the minimal number of bytes or, most optimally, bits. The mapping must be reversible, meaning that there can be no information loss, so that decoding is completely reliable.

If the encoding and serialization packs the information tightly, there may be little gain with a general compression algorithm such as LZ4.

Notions of General Intelligence

Before devoting much time on developing humanoid AGI and modeling cognition and states of mind, it may be useful to curb expectations away from trendy ideas. There is much conflict between the naive conceptions of general intelligence and the realities of neurological systems in animals and people, which are far more sophisticated than implied in the trendy media hype. The most important considerations are these.

The complexity of networks in the brain
That there is a significant varieties of neurons in brains
The impact of neuroplasticity
Complexity of chemical signaling independent from nerves
Whether intelligence is in fact multi-dimensional
Large number of genes that correlate with intelligence
The academic slant of concepts regarding intelligence
Difference between brain signal topology and computing models
Data-centricity

There are many articles on the web on these topics, along with critiques of concepts such as the g-factor and the idea of general problem solving and general intelligence. There are several questions here on this topic.

Why did Artificial Intelligence projects fail?
Can the IQ of an AI program be measured?
Can a brain be intelligent without a body?
Is the singularity concept mathematically flawed?
Is there a ceiling to intelligence itself?
Self-conscious artifical intelligence
Is the Turing Test, or any of its variants, a reliable test of artificial intelligence?

There is no value in discouraging curiosity and serious research into understanding the nature of intelligence or efforts to expanding the boundaries of computing. The above comments are to assist in bringing the starting point of such inquiry into the realm of enthusiasm anchored by scientific methodology. Conjecture makes for great movie plots, but does not further science and does not lead to real world technology products and services.


Making a CNN for phoneme classification I was making a simple phoneme classification model for a 10 week-long class project and I ran into a small question. Is it possible, to create a model that takes a 1-second (the longest phoneme is 0.2 second but the large image is kept for context) spectrogram as input? Some people suggest creating an RNN for phoneme classification but can you build a pure CNN phoneme classification model?
Any responses would be great.

Yes you can, a few years ago I made a simple CNN for a single Arabic phoneme classification. You can use spectogram or using MFCC / MFSC as features, as long all data has the same size (use padding or cropping if needed).

You may need RNN if you want to combine some phonemes to recognize a single word or longer.


How to implement a neural network for Flappy Bird in Python? I am new in the field of AI. I am working to create the flappy bird using Genetic Algorithm. After reading and seeing some examples, I saw that most implementations use a Neural Network + Genetic Algorithm and after certain generations, you achieve a very good agent that survives very long.

I currently struggling to implement the Neural Network since I have never taken a Machine Learning course.
On many examples that I have read neural networks require training inputs and outputs. For the flappy bird, I can't think of output since you don't really know if the action of flapping will benefit you or not.

In the example that I followed, Synaptic.js is used and it is pretty straight-forward. However, in Python, I can't find a simple library that will initialize randomly and adjust the weights and biases depending on the good agents that survive longer.

What would be the right way to implement this Neural Network without having a training dataset?

Is there any way to create Flappy Bird without using Neural Networks, just Genetic Algorithm?

The example in Javascript that I am referring to: Flappy Bird Using Machine Learning

What you are trying to achieve, is a game that learns to play flappy bird. For doing this you need a neural network AND a genetic algorithm, those two things work together.
About your concerns on the output, you don't have to know if the action will benefit or not, i will soon explain why.

The neural network part

So, what you need is to know how to build a neural network, i don't know your knowledge about it, but i suggest starting from the basics. In this scenario, you need a feed forward neural network, because you just take the inputs from the current flappy bird scene/frame (such as the y position of the bird, the distance from the closes pipe ecc..) and feed it through a network that outputs either 1 or 0 (jump or don't jump) in the only output neuron we just decided it has.

In python you can implement a neural network from scratch, or using a neural network framework that does al the dirty work for you.

From scratch you would need to use numpy for matrix calculations, and you would need to learn matrix multiplication, dot products and all that fancy stuff (You can just let numpy taking care of the matrix calculations, but understanding how it works behind the scenes always helps understand new problems that you might come across when doing more advanced stuff)
Using a framework like Tensorflow for python, the only thing you need to do is find the right structure for the network you want to use. You will not have to worry about how activations work, or how the feed forward is performed (But again, it's a good thing to know when working with neural nets)

The genetic algorithm part or ""learning""

I say ""learning"" because at first sight it might look like learning, but really it is not.
The genetic algorithm works like "the survival of the fittest", where the "smarter" birds, which are the ones that reached the higher score on the current generation, will have a chance to have their child little birds, that have the same brain as their parent, with either some minimal modifications, or a mix of their parent brains.
The process of this ""learning"", so the genetic algorithm, works like so:

Create a generation of let's say 200 birds, every bird has a brain with random weights, so at the first run, they are all very...not smart
The game starts, and every frame of the game, the brain of the bird recieves as input some data that is taken from the current frame ( y pos of the bird, distance from pipe...)
The brain ( neural network ) of each bird, performs a feed forward with that data, and outputs what at the beginning is a very random result, let's say 0.75 for one bird
At this point you decided that 0.75 is greater than 0.5, so you take that as a 1, which stands for "jump", while if it was 0.3, so 0, the bird does nothing and keeps falling
Shortly the bird will die cause he has no idea of what he is doing, so he most likely collides with a pipe or the ground.
After all birds met their fate, you see that some birds reached further than others, so you choose, for example, 5 of the best performing ones.
Now you try to create a new generation of 200 birds using only the brains of those 5 that were choosen, by mixing and modifying theyr brains
Now the new birds have a brand new brain, that in some cases might be better than the previous one, so chances are that some of those birds will reach a higher score, therefore flap further into the level.
Repeat from point 6

In practice your "perform_genetic_algorithm" function in python, will have to choose the birds with the highest score, and as wild as it sounds, mix their brains and modify them, hoping that some modifications will improve the performance of the bird.

  I can't think of output since you don't really know if the action of flapping will benefit you or not

The mechanism above explains why you basically do not care at all about the output, except saying to the game engine: "hey the bird decided to flap, do it". Whether it's the right action or not, doesn't matter, as the smarter birds are naturally gonna get further and so be more likely to be choosen for next generation.

Hopefully now it's all more clear.

Here is some useful links for building a neural network and for understanding the genetic algorithm:

How to build a neural network: I am linking this because it contains all useful information about how to build a very basic neural network in python. In your case, you would have to ignore all the part about backpropagation, loss & error calculation and SGD, and just look at the feed forward part.
How to build a neural network - 2: This is another example of building a neural network that i found really useful, probably it's simpler and more straight forward than the previous link, but again, the backpropagation part is not needed for this genetic based learning.
Video tutorials on genetic algorithm: This is a very long but very explanatory playlist of videos that dives into the nature of genetic algorithms and how to implement one
Genetic algorithm optimization: Other source about genetic algorithms


How does maximum approximation of the posterior choose a distribution? I was learning about the maximum a posteriori probability (MAP) estimation for machine learning and I found a nice short video that essentially explained it as finding a distribution and tweaking the parameters to fit the observed data in a way that makes the observations most likely (makes sense).

However, in mathematical terms, how does it determine which distribution best fits the data?

There are so many distributions out there that it could be any of them and the parameters you could fit them could be infinitely large.

Introduction: MAP finds a point estimate!

As opposed to your apparently current belief, in maximum a posteriori (MAP) estimation, you are looking for a point estimate (a number or vector) rather than a full probability distribution. The MAP estimation can be seen as a Bayesian version of the maximum likelihood estimation (MLE). Therefore, I will first remind you of the objective of MLE.

Maximum likelihood estimation (MLE)

Let $\theta$ be the parameters you want to find. For example, $\theta$ can be the weights of your neural network. In MLE, we want to find a point estimate (rather than a full distribution). The objective in MLE is

\begin{align}
\theta^*
&= \operatorname{argmax}_\theta p(X \mid \theta) \tag{1}\label{1}
\end{align}

where $p(X \mid \theta)$ is the likelihood of the data $X$ given the parameters $\theta$. In other words, we want to find the parameters $\theta$ such that $p(X \mid \theta)$ is the highest, where $X$ is your given training data, so $X$ is fixed.

The notation $p(X \mid \theta)$ can be confusing because, in a conditional probability distribution, $p(a\mid b)$, we often assume that $b$ is given and $p(a\mid b)$ is a distribution over $a$. However, in the case of MLE, $\theta$ in $p(X \mid \theta)$ is not fixed, but it is a variable, while $X$ is given and fixed. Hence we call $p(X \mid \theta)$ a likelihood rather than a probability density or mass function. Moreover, we often denote the likelihood as $\mathcal{L}(\theta; X) = p_{\theta}(X)$ (and there are other notations, but this is, in my opinion, the least confusing one), because we want to emphasize that the likelihood is actually a function of the variable $\theta$. However, this is notation can also be confusing because we equate a function of a variable $\theta$ to a probability distribution over $X$. However, you should note that $p_{\theta}(X)$ is parametrized by $\theta$.

Therefore, the MLE estimation \ref{1} can also be written as follows

\begin{align}
\theta^*
&=
\operatorname{argmax}_\theta \mathcal{L}(\theta; X) \\
&=\operatorname{argmax}_\theta p_{\theta}(X)
\tag{2}\label{2}
\end{align}
where $\theta^*$ is the point estimate of the objective function.

This notation emphasizes the fact that we want to find $\theta$, such that the probability of the given data $X$ is maximized.

Maximum a posteriori (MAP)

MAP is similar to MLE, but the objective is slightly different. First of all, we assume that $\theta$ is a random variable, so we have an associated probability distribution $p(\theta)$.

Recall that the Bayes' rule is the following

\begin{align}
p(\theta \mid X) = \frac{p(X \mid \theta) p(\theta)}{p(X)}
\tag{3}\label{3}
\end{align}

The objective function in MAP estimation is

\begin{align}
\theta^*
&= \operatorname{argmax}_\theta \frac{p(X \mid \theta) p(\theta)}{p(X)} \\
&= \operatorname{argmax}_\theta p(\theta \mid X) \tag{4}\label{4}
\end{align}

Given that $p(X)$ does not depend on $\theta$, for the purposes of optimization, we can ignore it, so equation \ref{4} becomes

\begin{align}
\theta^*
&= \operatorname{argmax}_\theta p(X \mid \theta) p(\theta) \\
&= \operatorname{argmax}_\theta p(\theta \mid X)
\tag{5}\label{5}
\end{align}
which is the MAP objective.

What is the relationship between MLE and MAP?

In MAP, the objective is \ref{5}, which includes a prior over $\theta$, while, in MLE, equation \ref{1}, there is no such thing.

Therefore, in MAP, we can assume that the parameters $\theta$ follow a certain distribution, thanks to the usage of $p(\theta)$.

In both MAP and MLE, we want to find a point estimate (which can be a number, if you have just one parameter, or a vector of size $N$, if you have $N$ parameters).
MAP is equivalent to MLE if you use a uniform prior, that is, if $p(\theta)$ is a uniform distribution.

Which distribution fits the data $X$?

In MAP, the human (you, me, etc.) chooses the family of distributions. For example, you can assume that your parameters $\theta$ follow a Gaussian distribution, so $p(\theta)$ will be a Gaussian distribution over the parameters. Why do I say "family"? For example, in the case of a Gaussian distribution, you have two parameters that control the shape of the distribution, namely, the mean and variance. Depending on the concrete values of these two parameters, you will have different Gaussian distributions, so you call all these Gaussian distributions a family.

How do you find $\theta$?

To find $\theta^*$, you can use an optimization method like gradient descent or, in certain cases, you can find a closed-form solution. See also Which distributions have closed-form solutions for maximum likelihood estimation?.

Resources

The following blog post MLE vs MAP: the connection between Maximum Likelihood and Maximum A Posteriori Estimation, by Agustinus Kristiadi (a Ph.D. student in machine learning), might also be useful, so I suggest you read it. It will give you more details that I've left out on purpose to avoid cluttering this answer.


When to use NLP, NLG and NLU in conversation agents? I had read some blogs (like 1, 2 or 3) about what the difference between all three of them is. I am trying to build an open domain conversation agent using natural language AI. That agent can do casual conversation, like a friend. So, for that, I want to know what is the importance of NLP, NLG, and NLU, so that I can learn that part first.

They're all important. NLP is an umbrella term that includes the other two; NLG is only concerned with generating language, ie transforming some internal data structure into human language. NLU is about processing information contained in language, and putting it into relation with a knowledge base etc.

If you don't know anything about any of these fields, then I suggest your aim is far too optimistic. I work for a company that provides a conversational AI platform for businesses to develop their own agents, and it is a complex area.

If you want to have a quick go and pick up some experience, I suggest you start with ELIZA. Even though this is 'ancient', many modern chatbots still work on the same principles. There are many implementations in a number of programming languages, so you should be able to find one that suits you and you can tinker around with it.


Robot Arm Deep Q Learning Actions Hello I am new to reinforcement learning and robotics. So far I have an understanding of the concept on 2D world. You can make agent move one step in one direction. However, how do you define movement action of a robot arm? I am a bit lost over here. Any useful links or keywords would be very appreciated! :)

It depends a lot on the hardware of your robot arm.
Assuming that your servos have encoder information, if you have access to servos that have limited control like "rotate left/rotate right" functionality, you can phrase the your action space to be ["move left", "stop", "move right"]. In this way you can implement a discrete action space with 3 actions per servo and have an agent learn to move the servos around the space.

If your servos are connected to each other in an elbow/shoulder configuration, you can have a 9 discrete action setup essentially making a box of cardinal directions:

Up+Left----------Up-------Up+Right

Left--------------Stop---------Right

Down+Left-----Down-----Down+Right

If you have 3 or more servos, you can still use the same idea of discrete actions but the number of discrete actions grows by a factor of 3 with each servo as your action space is now the cross product of all of the other servos.

Alternatively you can use a "multi-headed" agent where each head chooses actions for a certain servo but there are pros and cons for both depending on your usecase.

If you have more advanced servos like Dynamixels which have high quality encoders, you'll have access to more advanced controls schemes. For instance, Dynamixels allow you to give actions in encoder space, angle space, and even velocity space. For example, you could give the action of "go to encoder value of 500" or "go to 90 degrees" or "move .5 radians/second". All of these approaches are useful for certain tasks. For humans controlling the arm using a joystick, the velocity based is the most intuitive and, in my experience, the same is true for RL agents using continuous control.

If you are using continuous control, you should normalize all of your action spaces within your agent then "unnormalize (?)" them before giving the actions to your servos.
For instance if your servo velocity ranges from -3.5 rad/sec to +3.5 rad/sec, have your agent select actions in the range of [-1,1] then multiply by 3.5 to get the velocity.

In either case, one thing that should be noted is that you give your robotic arm enough time to actually perform the action that the agent selected. If not you will see your robot "jitter" back and forth quickly as your agent selects actions randomly. This is bad for a few reasons but most importantly because it might break your servos. To overcome this issue, give each action a little more time to be "actualized" by your robot. This can either come in the form of adding a delay in your code while your servos do "the thing" or by using an "iterations since last action update" counter to get a new action from the agent after a certain number of iterations. Not only is this better for your hardware, this also leads to better exploration of your state space as your agent can move through the state space encountering similar states more frequently.

A last thing to be aware of is to set hard coded limits on the servos so that your agent doesn't kill your servos by banging the arm against a table or something. Finding these safe bounds isn't always easy as multiple jointed configurations can have multiple ways of hitting the bounds and it might require a forward dynamics model to find these limits.
If you have only 2 servos, it should be pretty easy to find though :).


Are Autoencoders for noise-reduction only suited to deal with salt-and-pepper kind of noise? I'm currently looking at NN to deal with noisy data. I like the Autoencoder approach https://medium.com/@aliaksei.mikhailiuk/unsupervised-learning-for-data-interpolation-e259cf5dc957 because it seems to be adaptive and does not require to be trained on specific training data.
However, as it is described in this article it seems to rely on having none-noise samples in the input data that are true to the ground truth, so I wonder if an autoencoder also could work in the case of white or blue noise instead of salt-and-pepper noise?

The autoencoder proposed in the link is similar to a Denoising autoencoder (DAE), in sense that both starting from a noisy image try to reconstruct the original image. The difference is that the noisy pixel are ignored during the backpropagation. The DAE takes in input an image, where before the forward phase noise is applied on the image. On output the DAE have to reconstruct the image without noise.
So, given an image $I$ and $\epsilon \sim \mathcal{N}(0,\,\sigma^{2})$ the noise is applied to the image obtaining the image $\hat{I}$. Now, $\hat{I}$ is the DAE input and the autoencoder is trained to minimize the function $L(I,g(f(\hat{I})))$. Where $f$ is the encoder and $g$ is the decoder.

The image on the left is an image of the MNIST dataset where $\epsilon \sim \mathcal{N}(0,\,0.3)$ and on the right there is the reconstruction of a deep DAE. You can change the type of noise in the image, the results should be the same. So the answer to your question is no, autoencoder are suited also for other type of noise.
If interested you can read the paper "What Regularized Auto-Encoders Learn from the Data-Generating Distribution" Guillaume Alain, Yoshua Bengio where is shown that the DAE is learning an approximation of the gradient of the data distribution.


How can I detect moving objects in a video by OpenCV without using deep learning techniques? I want to detect moving objects in a surveillance video without using machine learning tools (like neural networks).
Is there a simple way in the OpenCV library? What is an efficient solution for this purpose?

After a quick scan, it would seem that, in the history of object detection, machine learning has always been at the forefront. Before then, it would just be a heuristic approach.

For a quick answer, here: https://towardsdatascience.com/real-time-object-detection-without-machine-learning-5139b399ee7d

That goes over object detection without using machine learning in OpenCV

That being said, there are some segments of computer vision that are not strictly machine learning. For example, a commonly used algorithm Selective Search for region proposal doesn't use machine learning, and the TF Algorithm for background generation doesn't either.

But when it comes to object detection, the gap between machine learning and all other methods is so sheer that ML is the only method really considered today.


Why we don't use importance sampling in tabular Q-Learning? Why don't we use an importance sampling ratio in Q-Learning, even though Q-Learning is an off-policy method?

Importance sampling is used to calculate expectation of a random variable by using data not drawn from the distribution. Consider taking a Monte Carlo average to calculate $\mathbb{E}[X]$.

Mathematically an expectation is defined as
$$\mathbb{E}_{x \sim p(x)}[X] = \sum_{x = \infty}^\infty x p(x)\;;$$
where $p(x)$ denotes our probability mass function, and we can approximate this by
$$\mathbb{E}_{x \sim p(x)}[X] \approx \frac{1}{n} \sum_{i=1}^nx_i\;;$$
where $x_i$ were simulated from $p(x)$.

Now, we can re-write the expectation from earlier as

$$\mathbb{E}_{x \sim p(x)}[X] = \sum_{x = \infty}^\infty x p(x) = \sum_{x = \infty}^\infty x \frac{p(x)}{q(x)} q(x) = \mathbb{E}_{x\sim q(x)}\left[ X\frac{p(X)}{q(X)}\right]\;;$$
and so we can calculate the expectation using Monte Carlo averaging
$$\mathbb{E}_{x \sim p(x)}[X] \approx \frac{1}{n} \sum_{i=1}^nx_i \frac{p(x)}{q(x)}\;;$$
where the data $x_i$ are now simulated from $q(x)$.

Typically importance sampling is used in RL when we use off-policy methods, i.e. the policy we use to calculate our actions is different from the policy we want to evaluate. Thus, I wonder why we don't use the importance sampling ratio in Q-learning, even though it is considered to be an off-policy method?

In Tabular Q-learning the update is as follows

$$Q(s,a) = Q(s,a) + \alpha \left[R_{t+1} + \gamma \max_aQ(s',a) - Q(s,a) \right]\;.$$

Now, as we are interested in learning about the optimal policy, this would correspond to the $\max_aQ(s',a)$ term in the TD target because that is how the optimal policy chooses its actions - i.e. $\pi_*(a|s) = \arg\max_aQ_*(s,a)$, so eventually the greedy TD update would be greedy with respect to the optimal state-action value function due to the guaranteed convergence of Q-learning.

The action $a$ in the update rule, i.e. the action we chose in state $s$ to receive the reward $R_{t+1}$, was chosen according to some non-optimal policy, e.g. $\epsilon$-greedy. However, as the $Q$ function is defined as the expected returns assuming we are in state $s$ and have taken action $a$ -- we thus don't need an importance sampling ratio for the $R_{t+1}$ term, even though it was generated from an action that the optimal policy might not have taken, because we are only updating the $Q$ function for state $s$ and action $a$, and by the definition of a $Q$ function it is assumed that we have taken action $a$ as we condition on this.


What is the difference between batches in deep Q learning and supervised learning? How is the batch loss calculated in both DQNs and simple classifiers? From what I understood, in a classifier, a common method is that you sample a mini-batch, calculate the loss for every example, calculate the average loss over the whole batch, and adjust the weights w.r.t the average loss?
(Please correct me if I'm wrong)
But is this the same in DQNs? So, you sample a batch from your memory, say 64 transitions. Do I iterate through each transition and adjust the weights "on the fly", or do I calculate the average loss of the batch and THEN in a big step adjust the weights w.r.t the average batch loss?

From what I understood in a classifier a common method is that you sample a mini-batch, calculate the loss for every example, calculate the average loss over the whole batch and adjust the weights w.r.t the average loss? (Please correct me if I'm wrong)

You are wrong.
The weights are adjusted w.r.t. the average gradient, and this must be calculated using individual loss function results. The average loss (or cost function when considering the whole dataset) is a useful metric for current performance, and it is the measure being minimised. But you cannot calculate meaningful gradients against the average loss directly.

But is this the same in DQNs?

The batch process is not as you described, but an experience replay minibatch in RL and a sampled minibatch in supervised learning can be very similar. The main difference in RL is that your prediction targets must be recalculated as part of the sampling process (using $G_{t:t+1} = R_{t+1} + \gamma \text{max}_{a'}\hat{q}(S_{t+1},a', \theta)$ to calculate the TD target, assuming you are using single step Q learning), whilst in most supervised learning the target values are fixed for each example.
In theory you could use repeated single item stochastic gradient descent in DQN, it doesn't break any theory, and it would work. However, it will usually be more efficient to use a standard minibatch update, combining all gradients into one average gradient for the minibatch and making a single update step.
If you are using a high level library for your neural network model in DQN, you usually don't need to worry about this detail. You can use the .fit function or whatever the library provides. In that case the only difference between a supervised learning update and an experience replay DQN update is what you get from the sampling. In supervised learning you get a set of $(\mathbf{x}_i, \mathbf{y}_i)$ examples directly by sampling a minibatch. In RL you get $(\mathbf{s}_i, \mathbf{a}_i, r, \mathbf{s'}_i, done)$ and must construct the $(\mathbf{x}_i, \mathbf{y}_i)$ minibatch from these before passing to your .fit function


Why do most deep learning papers not include an implementation? I'm a novice researcher, and as I started to read papers in the area of deep learning I noticed that the implementation is normally not added and is needed to be searched elsewhere, and my question is how come that's the case? The paper's authors needed to implement their models anyway in order to conduct their experimentations, so why not publish the implementation? Plus, if the implementation is not added and there's no reproducibility, what prevents authors from forging results?

The paper's authors needed to implement their models anyway in order to conduct their experimentations, so why not publish the implementation?

Some papers and authors actually provide a link to their own implementation, but most of the papers (that I have read) don't provide it, although some third-party implementations may already be available on Github (or other code-hosting sites) when you are reading the paper. There may be different reasons why the author(s) of a paper don't provide a reference implementation

They use some closed-source software or maybe it makes use of other resources that cannot be shared
Their implementation is a mess and, so, from a pedagogical point of view, it's quite useless
This may encourage other people to try to reproduce their results with different implementations, so it may indirectly encourage people to do research on the same topic (but maybe not providing a reference implementation could actually have the opposite effect!)

Plus, if the implementation is not added and there's no reproducibility, what prevents authors from forging results?

Actually, that's a good question that only researchers that have already published something or some reviewers can say something about it. I would also like to know their take on this!
Nevertheless, from some reviews of papers I have read (e.g. on OpenReview), in most cases, the reviewers are interested in the consistency of the results, the novelty of the work, the clarity and structure of the paper, etc. I think that, in most cases, they probably trust the provided results, also because, often, for reproducibility, researchers are expected to describe their models and parameters in detail, provide plots, etc., but I don't exclude that there are cases of people that try to fool the reviewers. For example, watch this video where Phil Tabor comments on ridiculous attempts to fool people and plagiarism by Siraj Raval.


What approach would work well for predicting earthquake intensity based on historical data? My problem: I own warning system where I collect data from institutions and send them over through various ways to users. I would like to hear your advice on what approach I can use for solving my problem with earthquake intensity far from epicenter. Since seismogical institutions mostly issue info about intenstiy of an earthquake for the epicenter, I would need to predict and classify what intensity the earthquake can have for places distant of several km/miles from the epicenter.
As an input/training set, I can use data of historical earthquakes and their magnitudes in an epicenter. Then I would need to fill mostly "by hand" an information about intensity based on seismological records, historical testimonies, chronicles atc.
What I need from AI: I need "something" that would predict earthquake intensity based on dataset of historical earthquakes.
Example/TLDR: There is an earthquake with magnitude 3.8, distant 80 km with depth 6 km. Based on dataset of historical earthquakes (with same type of information + witnessed and collected intensity), and output, I would need prediction of intensity of an eartquake 80 km from the epicenter.

You can do prediction like this with regression. Take a look at scikit-learn models for regression: https://scikit-learn.org/stable/modules/classes.html#module-sklearn.linear_model
Here's an example:
# This is made up data!
#        epicenter magnitude, depth_m, distance_km, local_magnitude
data = [[3.5,                 6000,    61.3,        2.1],
        [9.1,                 16000,   261.0,       6.0],
        [9.1,                 16000,   96.5,        8.2],
        [3.8,                 6000,    80.0,        2.6],
        [2.5,                 4000,    101.3,       0.4],
        [7.0,                 9000,    81.9,        6.1],
        [5.1,                 7000,    21.3,        4.1]]

from sklearn.linear_model import LinearRegression
import numpy as np

data = np.asarray(data) # to numpy array
X=data[:,:3] # the input variables
y=data[:,3] # the output variable (local magnitude)
reg = LinearRegression().fit(X, y) # fit the model based on the data
reg.predict(np.array([[6.6, 8000, 27]])) # Get a prediction based on new inputs

>>> 6.1511027


Why do we update the weights of the target network in deep Q learning? I know we keep the target network constant during training to improve stability, but why exactly are we updating the weights of our target network? In particular, if we've already reached convergence, why exactly are we updating the weights of our target network?

If you are certain that you reached convergence then there is no point in continuing to train your agent, because of that there is also no point in discussing why is target network being updated after convergence is reached. You should simply stop training if you converged. During training we obviously need to keep updating target network to improve correctness of Q-value estimates.


Neural Network that Predicts Game State Based on Actions I am trying to find literature on a network architecture that takes the following as in input:

Action (like 'Up', 'Down', etc)
Image of current state

and outputs:

Image of next state

I already have a lot of training data for the inputs. However, I am trying to find relevant literature/architecture for this problem.

You'll probably want to start out with "Action-Conditional Video Prediction using Deep Networks in Atari Games" (arXiv link: https://arxiv.org/abs/1507.08750). That's from 2015 though, I'm sure there have been lots of other interesting developments since then. This paper may still be a good starting point though, and provide you with the correct terminology to plug into google / google scholar to find more recent papers that build on top of this. Google scholar also provides functionality to automatically find papers that cite this one (interesting recent papers will probably cite this one).

As an additional point, you may want to reconsider your desired output. Given a current state-image and action, it may be easier to train a network to predict only the change in image (i.e., predicting NEW_IMAGE - OLD_IMAGE), rather than predicting the full image. You can then always still manually reconstruct the predicted new image simply by adding that output to the old image again. I'm quite sure I've seen this being done in a more recent paper too, but don't remember exactly the title / authors.


In this implementation of the Information Set Monte Carlo Tree Search, why can't the players see the cards of each other? After reading this paper about Monte Carlo methods for imperfect information games with elements of uncertainty, I couldn't understand the application of the determinization step in the author's implementation of the algorithm for the Knockout game.
Determinization is defined as the transformation from an instance of an imperfect information game to ab instance of a perfect one. It means that all players should see the cards of each other after the determinization step.
Why can't the players see the cards of each other in the code above?

All the methods in the GameState class that is used to represent state, are stubs, and without these, the MCTS algorithm won't do anything at all. In particular, the DoMove method just changes who's turn it is, without actually taking any action.

Probably the reason the players can't see each other's cards is that this is not a completed implementation. Someone is either still working on this, or gave up half way.


Deep Q-Learning: why don't we use mini-batches during experience reply? In examples and tutorial about DQN, I've often noticed that during the experience replay (training) phase people tend to use stochastic gradient descent / online learning. (e.g. link1, link2)

# Sample minibatch from the memory
minibatch = random.sample(self.memory, batch_size)
# Extract informations from each memory
for state, action, reward, next_state, done in minibatch:
    # if done, make our target reward
    target = reward
    if not done:
      # predict the future discounted reward
      target = reward + self.gamma * \
               np.amax(self.model.predict(next_state)[0])
    # make the agent to approximately map
    # the current state to future discounted reward
    # We'll call that target_f
    target_f = self.model.predict(state)
    target_f[0][action] = target

Why can't they use mini batches instead?
I'm new to RL, but in deep learning people tends to use mini-batches as they would result in a more stable gradient. Doesn't the same principle apply to RL problems? Is the randomness/noise introduced actually beneficial to the learning process? Am I missing something, or are these sources all wrong?

Note:

Not all the sources rely on stochastic gradient descent: e.g. keras-rl seems to rely on minibatches (https://github.com/keras-rl/keras-rl/blob/master/rl/agents/dqn.py)

DQN "library" implementations that I have seen do use mini-batches to train, and I would generally recommend this, as it usually strikes a reasonable balance between number of weight updates and accuracy of the gradients.

In your first link, and the code excerpt, the sample list is literally called minibatch. However, the developer then goes on to make a series of individual SGD steps per item within that sample.

  Why can't they use mini batches instead?

They can, and it works just fine. In my DQN explorations I tend to use a single mini-batch per timestep, and the Adam optimiser to accelerate learning. I also use a mini-batch to calculate TD target values.

  I'm new to RL, but in deep learning people tends to use mini-batches as they would result in a more stable gradient. Doesn't the same principle apply to RL problems?

It does, but I don't know if this is the most sample efficient that it could be. I prefer to use mini-batches and more complex optimisers because this runs faster (for a given number of replay training samples per time step), and I can run more time steps of the environment in less time.

It is possible for instance that a replay sample size of 32 per time step, with 4 mini-batch updates of 8 each would be the most sample efficient approach for a specific problem (because it makes more updates from the given samples), but it might not be the most CPU efficient.

  Is the randomness/noise introduced actually beneficial to the learning process?

I don't think so, but there are plenty of other tuning issues to consider in RL.

  Am I missing something, or are these sources all wrong?

They are not necessarily wrong, but may be CPU inefficient.

More likely in my opinion, your sources are explaining algorithms in principle, and did not want to add too many layers of optimisation, or too many components that may need tuning to work. There is a lot going on in a typical DQN agent, and having many performance optimisations can hide the details that the person is trying to teach.


What does 'democratizing AI' exactly mean? In my AI literature research, I often notice authors use term 'democratizing AI', especially in the AutoML area. I think I have an idea of what this means, but I would like to ask you for some more specific answers.

In this particular context, "Democratize" means to make more accessible to people.

Thus, "Democratizing AI" means to make AI softwares and AI programming available, accessible and easy to use for the vast majority of people.


Is the advent of a technological singularity a solution to the Fermi Paradox? It is really all in the title.

For those less familiar, the Fermi Paradox broadly speaking asks the question "where is everybody". There's an equation with a lot of difficult to estimate parameters, which broadly speaking come down to this (simplification of the Drake equation):

(Lots stars in the universe) * (non-zero probability of habitable planets around each star) * (lots of time spanned) = It seems there really should be somebody out there.

There are, of course, plenty of hypotheses as to why we haven't seen/observed/detected any sign of intelligent life so far, ranging from "well we're unique deal with it" to "such life is so advanced and destroys everything it comes across, so it's a good thing it didn't happen".

The technological singularity (also called ASI, Artificial Super Intelligence) is basically the point where an AI is able to self-improve. Some think that if such AI sees the light of day, it may self-improve and not be bound by biological constraints of the brain, therefore achieve a level of intelligence we cannot even grasp (let alone achieve ourselves).

I certainly have my thoughts on the matter, but interested to see if there is already an hypothesis revolving around the link between the 2 out there (I never came across but could be). Or perhaps an hypothesis as to why this cannot be.

For references to those not familiar with the Fermi paradox

If the technological singularity always leads to the extinction of all intelligent life, then yes.

If the technological singularity always leads to intelligent life migrating into higher planes of existence that aren't accessible to us right now, then also yes.

Otherwise it is exactly the assumption of unbridled technological progress that makes the Fermi paradox perplexing. A post-singularity culture should have the ability to spread through the galaxy. If there are a lot of post-singularity cultures some of them should have spread through the galaxy. And if there are enough cultures that are spreading through the galaxy, we should notice them.


Autoencoder for MobileNetV2 I have way more unlabeled data than labeled data. Therefore I would like to train an Autoencoder using MobileNetV2 as the encoder. Then I will use the pretrained model for the classification of the labeled data.

I think it is rather difficult to "invert" the MobileNet architecture to create a Decoder. Therefore, my question is: can I use a different architecture for the decoder, or will this introduce weird artifacts?

can I use a different architecture for the decoder, or will this
  introduce weird artifacts?

If you are using U-net -like architecture with skip connection from corresponding encoding to decoding layer outputs of corresponding layers should have the same spatial resolution. There is no other commonly recognized limitations on decoder architecture for convolutional networks.


What's the purpose of layers without biases? I noticed that the TensorFlow library includes a use_bias parameter for the Dense layer, which is set to True by default, but allows you to disable it. At first glance, it seems unfavorable to turn off the biases, as this may negatively affect data fitting and prediction.
What is the purpose of layers without biases?

Bias is one of the hyperparameters in neural networks, which let you shift activation function. Disabling bias means setting bias to be zero.
Even though, in many cases, bias is a big help for successful learning, in some cases, you may want to add an extra constraint to your neural network in finding the objective function. For example, in the paper below, a zero-bias layer as the last layer helps increase the output's interpretability.
https://ieeexplore.ieee.org/abstract/document/9173537


What is local consistency in constraint satisfaction problems? In the Constraint Propagation in CSP, it is often stated that pre-processing can solve the whole problem, so no search is required at all. And the key  idea is local consistency. What does this actually mean?

If we can do some reduction in the search space using CSP (constraint propagation) we can drastically reduce the search space or sometimes completely avoid the need for a search by directly reaching the solutions (for e.g. with variables having their domains reduced to size one). It could also happen that we come to a point when a variable domain size becomes zero, in that case no solution exists, given the constraints, so no need for a search.

Constraint propagation basically involves the concept of enforcing local consistency (this is done by enforcing node-consistency, arc-consistency, path-consistency and also global constraints using Alldiif or Atmost).

The terms: nodes, arc, path, etc. basically reflects a CSP problem represented as a graph with nodes as the variables and the arcs/edges as constraints. The process is simply to remove values from the domains of the variables that are inconsistent. Algorithms such as AC-3, PC-2, etc. precisely are for these purposes.


What framework for a project with a custom environment? I'm planning an RL project and I have to decide which RL framework do I use if any at all. The project has a highly custom environment, and testing different algorithms will be required to obtain optimal results. Furthermore, it will use a custom neural network, not implemented in the popular TensorFlow/PyTorch ML frameworks. Therefore, the framework should allow for customization with regard to approximation function (1) and the environment (2). The problem is that to my current knowledge, most of the framework allows only to work with a built-in environment. Does anybody know a framework that meets the two conditions (1) and (2)? Or anybody knows a review that contains information about framework in the context of those conditions?

You can modify the environments and create custom models using Ray RLlib: https://docs.ray.io/en/latest/rllib.html


Do convolutional neural networks perform convolution or cross-correlation? Typically, people say that convolutional neural networks (CNN) perform the convolution operation, hence their name. However, some people have also said that a CNN actually performs the cross-correlation operation rather than the convolution. How is that? Does a CNN perform the convolution or cross-correlation operation? What is the difference between the convolution and cross-correlation operations?

Short answer
Theoretically, convolutional neural networks (CNNs) can either perform the cross-correlation or convolution: it does not really matter whether they perform the cross-correlation or convolution because the kernels are learnable, so they can adapt to the cross-correlation or convolution given the data, although, in the typical diagrams, CNNs are shown to perform the cross-correlation because (in libraries like TensorFlow) they are typically implemented with cross-correlations (and cross-correlations are conceptually simpler than convolutions). Moreover, in general, the kernels can or not be symmetric (although they typically won't be symmetric). In the case they are symmetric, the cross-correlation is equal to the convolution.
Long answer
To understand the answer to this question, I will provide two examples that show the similarities and differences between the convolution and cross-correlation operations. I will focus on the convolution and cross-correlation applied to 1-dimensional discrete and finite signals (which is the simplest case to which these operations can be applied) because, essentially, CNNs process finite and discrete signals (although typically higher-dimensional ones, but this answer applies to higher-dimensional signals too). Moreover, in this answer, I will assume that you are at least familiar with how the convolution (or cross-correlation) in a CNN is performed, so that I do not have to explain these operations in detail (otherwise this answer would be even longer).
What is the convolution and cross-correlation?
Both the convolution and the cross-correlation operations are defined as the dot product between a small matrix and different parts of another typically bigger matrix (in the case of CNNs, it is an image or a feature map). Here's the usual illustration (of the cross-correlation, but the idea of the convolution is the same!).

Example 1
To be more concrete, let's suppose that we have the output of a function (or signal) $f$ grouped in a matrix $$f = [2, 1, 3, 5, 4] \in \mathbb{R}^{1 \times 5},$$ and the output of a kernel function also grouped in another matrix $$h=[1, -1] \in \mathbb{R}^{1 \times 2}.$$  For simplicity, let's assume that we do not pad the input signal and we perform the convolution and cross-correlation with a stride of 1 (I assume that you are familiar with the concepts of padding and stride).
Convolution
Then the convolution of $f$ with $h$, denoted as $f \circledast h = g_1$, where $\circledast$ is the convolution operator, is computed as follows
\begin{align}
f \circledast h = g_1
&=\\
[(-1)*2 + 1*1, \\
(-1)*1 + 1*3, \\
(-1)*3 + 1*5, \\
(-1)*5+1*4]
&=\\
[-2 + 1, -1 + 3, -3 + 5, -5 + 4]
&=\\
[-1, 2, 2, -1] \in \mathbb{R}^{1 \times 4}
\end{align}
So, the convolution of $f$ with $h$ is computed as a series of element-wise multiplications between the horizontally flipped kernel $h$, i.e. $[-1, 1]$, and each $1 \times 2$ window of $f$, each of which is followed by a summation (i.e. a dot product). This follows from the definition of convolution (which I will not report here).
Cross-correlation
Similarly, the cross-correlation of $f$ with $h$, denoted as $f \otimes h = g_2$, where $\otimes$ is the cross-correlation operator, is also defined as a dot product between $h$ and different parts of $f$, but without flipping the elements of the kernel before applying the element-wise multiplications, that is
\begin{align}
f \otimes h = g_2
&=\\
[1*2 + (-1)*1, \\
1*1 + (-1)*3, \\
1*3 + (-1)*5, \\
1*5 + (-1)*4]
&=\\
[2 - 1, 1 - 3, 3 - 5, 5 - 4]
&=\\
[1, -2, -2, 1] \in \mathbb{R}^{1 \times 4}
\end{align}
Notes

The only difference between the convolution and cross-correlation operations is that, in the first case, the kernel is flipped (along all spatial dimensions) before being applied.

In both cases, the result is a $1 \times 4$ vector. If we had convolved $f$ with a $1 \times 1$ vector, the result would have been a $1 \times 5$ vector. Recall that we assumed no padding (i.e. we don't add dummy elements to the left or right borders of $f$) and stride 1 (i.e. we shift the kernel to the right one element at a time). Similarly, if we had convolved $f$ with a $1 \times 3$, the result would have been a $1 \times 3$ vector (as you will see from the next example).

The results of the convolution and cross-correlation, $g_1$ and $g_2$, are different. Specifically, one is the negated version of the other. So, the result of the convolution is generally different than the result of the cross-correlation, given the same signals and kernels (as you might have suspected).

Example 2: symmetric kernel
Now, let's convolve $f$ with a $1 \times 3$ kernel that is symmetric around the middle element, $h_2 = [-1, 2, -1]$. Let's first compute the convolution.
\begin{align}
f \circledast h_2 = g_3
&=\\
[(-1)*2 + 1*2 + (-1) * 3,\\ (-1)*1 + 2*3 + (-1) * 5,\\ (-1)*3 + 2*5 + (-1) * 4]
&=\\
[-2 + 2 + -3, -1 + 6 + -5, -3 + 10 + -4]
&=\\
[-3, 0, 3]
\in \mathbb{R}^{1 \times 3}
\end{align}
Now, let's compute the cross-correlation
\begin{align}
f \otimes h_2 = g_4
&=\\
[(-1)*2 + 1*2 + (-1) * 3, \\ (-1)*1 + 2*3 + (-1) * 5, \\ (-1)*3 + 2*5 + (-1) * 4]
&=\\
[-3, 0, 3]
\in \mathbb{R}^{1 \times 3}
\end{align}
Yes, that's right! In this case, the result of the convolution and the cross-correlation is the same. This is because the kernel is symmetric around the middle element. This result applies to any convolution or cross-correlation in any dimension. For example, the convolution of the 2d Gaussian kernel (a centric-symmetric kernel) and a 2d image is equal to the cross-correlation of the same signals.
CNNs have learnable kernels
In the case of CNNs, the kernels are the learnable parameters, so we do not know beforehand whether the kernels will be symmetric or not around their middle element. They won't probably be. In any case, CNNs can perform either the cross-correlation (i.e. no flip of the filter) or convolution: it does not really matter if they perform cross-correlation or convolution because the filter is learnable and can adapt to the data and tasks that you want to solve, although, in the visualizations and diagrams, CNNs are typically shown to perform the cross-correlation (but this does not have to be the case in practice).
Do libraries implement the convolution or correlation?
In practice, certain libraries provide functions to compute both convolution and cross-correlation. For example, NumPy provides both the functions convolve and correlate to compute both the convolution and cross-correlation, respectively. If you execute the following piece of code (Python 3.7), you will get results that are consistent with my explanations above.
import numpy as np

f = np.array([2., 1., 3., 5., 4.])

h = np.array([1., -1.])
h2 = np.array([-1., 2., 1.])

g1 = np.convolve(f, h, mode="valid")
g2 = np.correlate(f, h, mode="valid")

print("g1 =", g1) # g1 = [-1.  2.  2. -1.]
print("g2 =", g2) # g2 = [ 1. -2. -2.  1.]

However, NumPy is not really a library that provides out-of-the-box functionality to build CNNs.
On the other hand, TensorFlow's and PyTorch's functions to build the convolutional layers actually perform cross-correlations. As I said above, although it does not really matter whether CNNs perform the convolution or cross-correlation, this naming is misleading. Here's a proof that TensorFlow's tf.nn.conv1d actually implements the cross-correlation.
import tensorflow as tf # TensorFlow 2.2

f = tf.constant([2., 1., 3., 5., 4.], dtype=tf.float32)
h = tf.constant([1., -1.], dtype=tf.float32)

# Reshaping the inputs because conv1d accepts only certain shapes.
f = tf.reshape(f, [1, int(f.shape[0]), 1])
h = tf.reshape(h, [int(h.shape[0]), 1, 1])

g = tf.nn.conv1d(f, h, stride=1, padding="VALID")
print("g =", g) # [1, -2, -2, 1]

Further reading
After having written this answer, I found the article Convolution vs. Cross-Correlation (2019) by Rachel Draelos, which essentially says the same thing that I am saying here, but provides more details and examples.


Which is the most successful AI project so far? I'm looking for good examples of successful AI projects and theories that had a relatively good impact on society, economics and military field.

So many years have passed after the first AI researches; hence I'm wondering if it has really increased the quality of our lives.

I'd say the most successful are the ones so commonly used that we don't even notice them:

The mail systems that automatically decipher handwritten addresses on your packages, they use machine vision and have probably been doing it since mid-90s.
Algorithmic trading bots on stock markets - they handle something like 85% of all trades.
Many modern CPUs use AI techniques, including neural networks, to guess what your program is going to do next and optimize branch prediction and memory fetches.
Most good modern fraud and spam detectors use some combination of AI techniques (clustering, decision trees, SVMs, even some machine vision to check out attached pictures) - and, in the opposing BlackHat camp, the latest automatic CAPTCHA breakers use all the latest advancements in deep learning too).
And of course there's Google, Facebook and US DoD who try to put AI into anything they can think of.


What is "natural image domain"? I see some papers use the term "natural image domain". I googled that but didn't find any explanation of it.

I guess I understand the normal meaning of "natural image", such as the image people take by phones. The images in ImageNet database are all natural images.

Is "natural image domain" a subfield of computer vision?

An image of a natural domain refers to any kind of image that has some variance in its structure, so it does not always present the same peculiarities.

For example, in a digits classification task, you can train your network on the MNIST dataset, that is a simple black & white set of images that are always well defined: the digit is in darker pixels, and the background is always white, they just have different shapes.

For the same task but in the natural image domain, the digits may vary in color, noise, and even position inside of a single image. As you may imagine this does not apply to digits only, but it includes any image that was not specifically created or modified to be fed into a neural network.

So, to put it simply, a neural network trained for natural images is able to perform classification in any real-life situation or scenario, without the need to edit the image to adapt it to what the network was trained for (for example, changing the color or cropping so the object you want to recognize is exactly in the center).

Here a similar question that may fill some doubts you may have.


How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function? I've seen plenty of examples of people doing Sigmoid + MSE backpropagation implementations, yet I do not seem to understand how to implement backpropagation as stated in the title in the case of multi-class classifications.
What confuses me mainly are the matrix-vector shapes and their multiplications, and their implementations in code.

Alright. Consider an ordinary neural network, so, in the last layer, we have, $z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]}$, where $a^{[L]} = \sigma(z^{[L]})$, where $\sigma$ is the softmax activation:
$$
\sigma(\mathbf z)_{i} = \frac{e^{z_i}}{\sum_k e^{z_k}}
$$
I think, one of the most effective ways of not to get confused about all these matrices with different dimensions, is to simply get rid of the matrices and do all calculations componentwise. So, calculating the $ij$ component of the jacobian matrix, we have:
$$
J_{ij} = \frac{\partial\sigma_i}{\partial z_j} = \frac{e^{z_i}}{\sum_k e^{z_k}}\delta_{ij} - \frac{e^{z_i}e^{z_j}}{(\sum_k e^{z_k})^2}
$$
Consider a cost $C$, and we wish to calculate the derivative with respect to the weights of the last layer, that is, we are running the first step of backpropagation. Applying chain rule to the $ij$-component of the weight:
$$
\frac{\partial C}{\partial W^{[L]}_{ij}}
= \sum_k\frac{\partial C}{\partial a_{k}}\frac{\partial a^{[L]}_{k}}{\partial W^{[L]}_{ij}}
$$
And, with one more chain rule:
$$
\frac{\partial a^{[L]}_{k}}{\partial W^{[L]}_{ij}} =
\sum_s\frac{\partial a^{[L]}_{k}}{\partial z^{[L]}_s}\frac{\partial z^{[L]}_s}{\partial W^{[L]}_{ij}}
$$
Now we can identify exactly where the jacobian matrix is:
$$
J_{ij} = \frac{\partial\sigma_i}{\partial z_j} = \frac{\partial a^{[L]}_{i}}{\partial z^{[L]}_j}
$$
Thus, the back-propagation step is just:
$$
\frac{\partial C}{\partial W^{[L]}_{ij}}
= \sum_k\sum_s\frac{\partial C}{\partial a_{k}}\frac{\partial a^{[L]}_{k}}{\partial z^{[L]}_s}\frac{\partial z^{[L]}_s}{\partial W^{[L]}_{ij}}
= \sum_k\sum_s\frac{\partial C}{\partial a_{k}}J_{ks}\frac{\partial z^{[L]}_s}{\partial W^{[L]}_{ij}}
$$
Now, in component language, it is much simpler to identify where are all the matrix multiplications: The transpose of the gradient with respect to the cost function, multiplied by the jacobian matrix of the softmax activation (or whatever activation), and finally, the last derivative, which will evaluate to something depending on the activation of the previous layer.
=].


Bayes error rate formula clarification My questions concern a particular formulation of the Bayes error rate from Wikipedia, summarized below.

  For a multiclass classifier, the Bayes error rate may be calculated as follows:
  $$p = 1 - \sum_{C_i \ne C_{\max,x}} \int_{x \in H_i} P(C_i|x)p(x)\,dx$$
  where $x$ is an instance, $C_i$ is a class into which an instance is classified, $H_i$ is the area/region that a classifier function $h$ classifies as $C_i$.

We are interested in the probability of misclassifying an instance, so we wish to sum up the probability of each unlikely class label (hence we want to look at $C_i \ne C_{\max, x}$). However, the integral is confusing me. We want to integrate an area corresponding to the probability that we choose label $C_i$ given $x$. But we drew $x$ from $H_i$, the region covered/classified by $C_i$, so wouldn't $P(C_i|x) = 1$?

I think most of my confusion will be resolved if someone can help clarify the intention of the integral. Is it to draw random samples from the total space of $h$ (the classifier function), and then sum the probabilities from each classified $C_i \ne C_{\max, x}$? How does $x$ exist in the outer summation before it has been sampled from $H_i$ in the integral?

Bayes Error Rate
For the general case of K different classes, the probability of classifing x instance correctly is:
\begin{equation} \label{eq1}
\begin{split}
P(correct) & = \sum_{i=1}^{K} p(x \in H_i, C_i) \\
 & = \sum_{i=1}^{K} \int_{x \in H_i} p(x,C_i) \, dx\\
 & = \sum_{i=1}^{K} \int_{x \in H_i} P(C_i|x)p(x)\,dx \\
\end{split}
\end{equation}
where $H_i$ is the region where class $i$ has the highest posterior. So the Bayes Error Rate is:
\begin{equation} \label{eq2}
\begin{split}
P(error) & = 1 - p(correct) \\
 & = 1 - \sum_{i=1}^{K} \int_{x \in H_i} P(C_i|x)p(x)\,dx
\end{split}
\end{equation}
Be careful if we drew $x$ from $H_i$, the region covered/classified by $C_i$, $P(C_i|x) \ne 1$, because that would mean that we surely predict correctly all the time. If $x$ belongs to a decision area, this does not imply that it belongs to the corresponding class.


Is it possible to train a neural network to learn something via video footage? Is it possible to train a neural network to learn something via video footage (which is essentially a sequence of images)?

In other words, if I have a video teaching me how to draw an animal from scratch, can I then use this video to teach the computer to draw the animal in the same way?

There are many machine learning and image processing techniques (such as RNNs) that can be applied to sequences of images or videos. So, I guess the difficult part becomes mapping the activity to an action like moving a pen or something.

Short answer: No.

Long answer: A neural network is a function that maps input data (e.g. a picture) to output data (e.g. probability that the picture contains a dog). What you propose does not seem like the sort of task this tool is suited for, though I'm not willing to claim that it can't be done.

If you are interested in computer-generated artwork, I encourage you to learn about Generative Adversarial Networks and style transfer.


Backpropagation of neural nets with shared weight I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. However, I am stuck at some point.

Consider a simple fully connected neural network with two hidden layers. For simplicity, choose linear activation function (${f(x) = x}$) at all layer. Now consider that this neural network takes two $n$-dimensional inputs $X^{1}$ and $X^{2}$. However, the first hidden layer only takes $X^1$ as the input and produces the output of $H^1$. The second hidden layer takes $H^{1} $and $X^2$ as the input and produces the output $H^{2}$. The output layer takes $H^{2}$ as the input and produces the output $\hat{Y}$. For simplicity, assume, we do not have any bias.

So, we can write that, $H^1 = W^{x1}X^{1}$

$H^2 = W^{h}H1 + W^{x2}X^{2} = W^{h}W^{x1}X^{1} + W^{x2}X^{2}$ [substituting the value of $H^1$]

$\hat{Y} = W^{y}H^2$

Here, $W^{x1}$, $W^{x2}$, $W^{h}$ and $W^{y}$ are the weight matrix. Now, to make it more interesting, consider a sharing weight matrix $W^{x} = W^{x1} = W^{x2}$, which leads, $H^1 = W^{x}X^{1}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$

I do not have any problem to do forward propagation by my hand; however, the problem arises when I tried to make backward propagation and update the $W^{x}$.

$\frac{\partial loss}{\partial W^{x}} = \frac{\partial loss}{\partial H^{2}} . \frac{\partial H^{2}}{\partial W^{x}}$

Substituting, $\frac{\partial loss}{\partial H^{2}} = \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$

$\frac{\partial loss}{\partial W^{x}}= \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y} . \frac{\partial}{\partial W^{x}} (W^{h}W^{x}X^{1} + W^{x}X^{2})$

Here I understand that, $\frac{\partial Y}{\partial H^{2}} = (W^y)^T$ and $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$ and we can also calculate $\frac{\partial Y}{\partial H^{2}}$, if we know the loss function. But how do we calculate $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$?

If we write $ H^2 = W^{h}H1 + W^{x}X^{2} $ then it will be better to understand the backward propagation step.

Now,

$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$ can be written as:
$\frac{\partial H^2}{\partial H^1}\frac{\partial H^1}{\partial W^{x}} $

$\frac{\partial H^2}{\partial H^1} =  (W^h)^T$ and
$\frac{\partial H^1}{\partial W^{x}} = (X^{1})^T $

Therefore,

$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = (W^h)^T(X^{1})^T  $

I hope it has solved your problem.


How can I determine if an input sentence is consistent with a certain subject? How can I determine if an input sentence is consistent with a certain subject?

For example, suppose I am given the following dataset.

| Subject       | User input           | Output |
|---------------|----------------------|--------|
| Dog ownership | I own a dog          | Yes    |
| Dog ownership | My dog is called Joe | Yes    |
| Dog ownership | I don't have a dog   | No     |

In the examples above, the subject "dog ownership" is consistent with the input sentences "I own a dog" and "My dog is called Joe" (because, if your dog is called Joe, then you also own a dog).

If you have lots of training data, ANN's (deep learning) could quite possibly get you there.  But I have a hunch you might also get some mileage out of using something like a rule induction approach.  Maybe something like CN2. I'd suggest at least reading up on those and see if you can see a way to apply that to your system.