Can an object's movement (instead of its appearance) be used to classify it? I know that it is very common for machine learning systems to classify objects based on their visual features such as shapes, colours, curvatures, width-to-length ratios, etc.

What I'd like to know is this: Do any techniques exist for machine learning systems to classify objects based on how they move?

Examples:

Suppose that in a still image, 2 different classes of objects look identical. However, in a video Class A glides smoothly across the screen while Class B meanders chaoticaly across the screen.
When given multiple videos of the same person walking, classify whether he's sober or drunk.

Machine learning systems work based on input data. The form of that data is irrelevant. It may need some configuring on the technical side of things, but the general ability to learn from the dataset remains the same.
ML systems are built to learn how to interpret data, without you needing to predefine exactly what it should be looking for.

But even then, you can reason that videos are no different from images. A video is essentially nothing but a flipbook of images. If you were to paste every frame of the video in a long sequence, you would have one big picture.
The rest should follow as usual: if a machine learning system can be taught to interpret images, it can evidently also interpret these "frame sequence" images, which means it's able to interpret the dataset that we colloquially call a video.

That being said, since a video contains much more information (a 5 second 30fps video is 150x as much data as a single image with the same resolution), the required learning process increases exponentially.

So yes, it's perfectly possible, but it will require more processing power and training as there are more input variables to account for.

There are ways to reduce this increase in complexity. For example, to check if someone is drunk or not, rather than push the entire video through the learning algorithm, you instead preprocess the video to figure out the person's gait or even just their footsteps (relative location and timing) and only push this processed data through.

This dramatically cuts down on the complexity of the network needed; at the cost of requiring preprocessing which may introduce issues (bad preprocessing = bad learning).


How do I compute the variance of the return of an evaluation policy using two behaviour policies? Suppose there is an evaluation policy called $\pi_{e}$ and there are two behavior policies $\pi_{b1}$ and $\pi_{b2}$. I know that it is possible to estimate the return of policy $\pi_{e}$ through behavior policies via importance sampling, which is unbiased. But I do not know about the variance of return estimated through two behavior policies $\pi_{b1}$ and $\pi_{b2}$. Does anybody know about the variance or any bound on the variance of estimated return?

Let $G_{0}^{b1}=\sum_{t=1}^{T}\gamma^{t-1}r_{t}^{b1}$ represent the total return for an episode through behavior policy $\pi_{b1}$ and $G_{0}^{b2}=\sum_{t=1}^{T}\gamma^{t-1}r_{t}^{b2}$ represent the total return for an episode through behavior policy $\pi_{b2}$.
It is possible to estimate the return of policy $\pi_{e}$ as follows:

$$G_{0}^{(e,b1)}=\prod_{t=1}^{T}\frac{\pi_{e}(a_{t}|s_{t})}{\pi_{b1}(a_{t}|s_{t})}*G_{0}^{b1}$$

$$G_{0}^{(e,b2)}=\prod_{t=1}^{T}\frac{\pi_{e}(a_{t}|s_{t})}{\pi_{b2}(a_{t}|s_{t})}*G_{0}^{b2}$$

I want to compare the variance of $G_{0}^{(e,b1)}$ and $G_{0}^{(e,b2)}$. Is there any formulation to compute the variance $G_{0}^{(e,b1)}$ and $G_{0}^{(e,b2)}$?

If you simulate many trajectories and receive many estimates of the two returns you're interested in, you could empirically compare their sample variances.

However, the variance of ordinary importance sampling is in general unbounded.
If you're wanting some theoretical bounds on the variance of importance sampling estimates, I'd start with weighted importance sampling, whose variance converges to zero (Sutton and Barto, section 5.5).


How to classify when the label is seldom known When I think about classification I think of the cancer/not cancer example. You have a bunch of attributes and you know whether the person had cancer during the relevant time period and you determine which attributes predict that result.
I work in a highly-regulated industry that serves the public. There are certain people we are not allowed to do business with, let's say because they will use our service for illegal purposes. Sometimes we tell the (potential) customer "yes" and sometimes "no".
When we say "yes" and the potential customer intends to use our service for illegal activity, they certainly won't inform us of the mistake.
Likewise, when we say "no" the potential customer will sometimes go away, will sometimes complain, but  that potential customer will not self-identify and say "yes, you are correct, I intended to use your service for illegal activity".
Occasionally we will receive a report from a 3rd-party that will label a customer, but these reports are  a tiny fraction of the number of customers. Unlike the cancer classification we almost always don't know the actual label, we only know what we guessed.
What techniques should we consider to measure our accuracy?

Domain knowledge and cluster interpretation
When you have no (or very limited) gold standard labels for your dataset, traditional performance metrics that require knowing how often you're correct (like accuracy, sensitivity, or specificity) simply won't work. If this is the case, you'll need to examine your outliers to see if they make sense in the context of your particular field. You should look at what features are responsible for a sample being classified as an anomaly, for example, you might see that agents classified as spam send a huge number of emails per hour, or that a suspected money laundering bank account interacts frequently with many offshore accounts. In most cases, you will never know with absolute certainty that your classification is correct, but if you can justify the classification in terms of what is known about your particular domain, it can lend credence to a model.
Another approach is to perform unsupervised clustering on your dataset. Hopefully, you should see that your method identifies anomalies which are somehow different from other datapoints, appearing as outliers or a distinct group of samples unlike the others. You will need to do more investigation to be confident that the clusters represent normal people and fraudsters, but clustering can help you identify that there are indeed some samples that appear markedly different from the others. If you can show that certain samples show similar hallmarks of illegal activity, it also helps to gain confidence in what your model is doing.


Is the size of a neural network directly linked with an increase in its inteligence? Just came across this article on GPT-3, and that lead me to the question:
In order to make a certain kind of neural network architecture smarter all one needs to do is to make it bigger?
Also, if that is true, how does the importance of computer power relates with the importance of fine-tuning/algorithmic improvement?

First of all, there is no real 'intelligence' innate to artificial Neural Networks (NNs).
All they do is trying to approximate a mathematical function with a certain degree of generalization (hopefully without learning a given dataset by heart, i.e. hopefully without overfitting).
The more nodes (or neurons) you include into the network, the more complex a function can be that a network can learn to approximate. It's similar to high-school math: The higher the degree of some polynomial, the better the polynomial can be adjusted to fit some observation to be modeled; with the only difference being that NNs commonly include non-linearities and are trained via some kind of stochastic gradient descent.
So, yes. The more nodes a model possesses, the higher the so-called model capacity, i.e. the higher the degree of freedom a NN-model has to fit some function. After all, NN are said to be universal function approximators - given they have enough internal nodes in their hidden layer(s) to fit some given function.
In practice, however, you don't want to blow up a model architecture unnecessarily, since this commonly results in overfitting if it doesn't cause some instabilities of the training procedure instead.
Generally, the larger the model to be trained, the higher the computational cost to train the network.
A common suggestion is to reduce the number of nodes in a network at the expense of increasing a network's depth, i.e. the number of hidden layers. Often, that can help reduce the demand for excessively many nodes.


When should Monte Carlo Tree search be chosen over MiniMax? I would like to ask whether MCTS is usually chosen when the branching factor for the states that we have available is large and not suitable for Minimax. Also, other than MCTS simluates actions, where Minimax actually 'brute-forces' all possible actions, what are some other benefits for using Monte Carlo for adversarial (2-player) games?

Some basic advantages of MCTS over Minimax (and its many extensions, like Alpha-Beta pruning and all the other extensions over that) are:

MCTS does not need a heuristic evaluation function for states. It can make meaningful evaluations just from random playouts that reach terminal game states where you can use the loss/draw/win outcome. So if you're faced with a domain where you have absolutely no heuristic domain knowledge that you can plug in, MCTS is likely a better choice. Minimax must have a heuristic evaluation function for states (exception: if you game is so simple that you can afford to compute the complete game tree and reach all terminal game states immediately from the the initial game state, you don't need heuristics). If you do have strong evaluation functions, you can still incorporate them and use them to improve MCTS too; they're just not strictly necessary for MCTS.

MCTS has simpler anytime behaviour; you can just keep running iterations until you run out of computing time, and then return the best move. Typically we expect the performance level of MCTS to grow with computatinon time / iteration count relatively smoothly (not always 100% true, but intuitively you can usually expect something like this). You can sort of achieve anytime behaviour in minimax with iterative deepening, but that's usually a bit less "smooth", a bit more "bumpy"; this is because every time you increase the search depth, you need significantly more processing time than you did for the previous depth limit. If you run out of time and have to abort your current search at your current depth limit, that last search will be completely useless; you'll have to discard it and stick to the results from the previous search with the previous depth limit.

A difference, which is not necessarily an advantage or disadvantage either way in the general case (but can be in specific cases):

The computation time of MCTS is generally dominated by running (semi-)random playouts. This means that functions for computing legal move lists, and applying moves to game states, typically dictate how fast or slow your MCTS runs; making these functions faster will generally make your MCTS faster. On the other hand, the computation time of Minimax is generally dominated by copying game states (or "undoing" moves, which is an operation that in most games will require additional memory usage for game states to be possible) and heuristic evaluation functions (though the latter are likely to also become important in terms of computation cost in MCTS if you choose to include them there). In some games it will be easier to provide efficient implementations for one of these, and in other games it may be different.

A basic advantage of Minimax over MCTS:

In settings where MCTS can only run very few iterations relative to the branching factor (or in the extreme case, fewer iterations than there are actions available in the root node), MCTS will perform extremely poorly / close to random play. We've noticed this being the case for quite a decent number of games in our general game system Ludii (where the "general game system" often implies that games are implemented less efficiently than they could be in a dedicated single-game-specific program) with low time controls (like 1 second per move). This same general game setting often makes it difficult to find super strong heuristics, but it's generally still possible to come up with some relatively simple ones (like just a simple material heuristic in chess). An alpha-beta search with just a couple of search plies and a basic, simple heuristic will often outperform a close-to-random MCTS if the MCTS can't manage to run significantly more iterations than it has legal moves in the root node.


Giraffe Chess - High Level Assessment My high-level takeaway from Matthew Lai's Giraffe Chess Paper is that one would want to use broad, shallow game trees, with some method of evaluating the probability of a favorable outcome for a given board position.  Is this correct?

(Still working my way though the AlphaGo paper, but the method seems to be similar.)

If you mean high level assessment of self-learned evaluation functions in chess, then no, the advantage of a better evaluation function lies in the ability to prune the search tree more aggressively. So you would on the contrary try to search narrowly but deeply.

(In reality neural network based evaluation functions are so slow, that you would search narrowly and still not get very deep. Nor very strong.)

If you mean chess programming in general, than the answer is also no. In chess you have to go deep, at least selectively, because tactical possibilities that occur deep in some variations are important.


The best way of classifying a dataset including classes with high similarity? I have a dataset which has two very similar classes (men wrestling, women wrestling). I've used InceptionV3 as a classifier to solve the problem of classifying this dataset. Unfortunately, the accuracy of this classifier doesn't hit more than 70%. Is there any suggestion about how I can overcome this problem or any other similar problems?

If you want to classify data with similar characteristics, it would often helps if you hand craft features. For classifying women or men wrestling, you may want to try using cv2 to track human faces and feed that to your CNN as input. Example: https://realpython.com/face-recognition-with-python/

If the data is not images, you may want to do a analysis to see which feature have no clear relationship with whether it is men or women wrestling and remove them. Example: https://towardsdatascience.com/a-feature-selection-tool-for-machine-learning-in-python-b64dd23710f0

Hope that I can help you and have a nice day!


I have 5000 html files (structured text), how can I generate a new one that "resembles" those? I don't know anything about ML or NLP, but I was asked by someone to create brand new statutes (written laws) that resemble the ones currently in effect in my country. I have already gathered the laws, and have 5000 html files now, one per law.
The average size of each html file is 49 kB. The entire corpus is 300 MB.
I have two alternative goals (doing both would be perfect of course):

Generate a new, complete HTML file, that would imitate the 5000 existing ones (it would typically have 1 big heading at the top, sub-headings, articles with their own title and number, etc.)

Generate sentences that sound as if they could be found in a typical law (the laws are written in French)

Is any of those goals feasible, with such a small corpus (~300 MB total)?
Should I try and fine-tune an existing model (but in that case, wouldn't the small size of my corpus be a problem? Wouldn't it be "drowned out" in the rest of the training data?), or should I create one from scratch?
I've tried following guides on huggingface, but between the obsolete files, the undocumented flags and my general lack of knowledge of the subject, I'm completely lost.
Thanks in advance.
BTW, if you want to take a peek at the data, there it is: https://github.com/Biganon/rs/

You could try to train a recurrent neural net on char level. Basically, you took GRU or LSTM and use a sequence of characters, not tags or words. In the blogpost "The Unreasonable Effectiveness of Recurrent Neural Networks" there are examples for Shakespeare, Linux source code on C and for papers in latex code, and the results are quite valid, produced from similar to your size training sets. The good thing in case of html pages is that modern browsers are very good at handling a slightly broken HTML, so this approach could work for both of your tasks.


Why do self-driving cars use a radar? The slideshow 10 astonishing technologies that power google’s
self-driving cars documents some of the technologies used in Google's self-driving car. It mentions a radar.

Why does Google use radar? Doesn't LIDAR do everything radar can do? In particular, are there technical advantages with radar regarding object detection and tracking?

To clarify the relationship with AI: how do radar sensors contribute to self-driving algorithms in ways that LIDAR sensors do not?

The premise is AI algorithms are influenced by inputs, which are governed by sensors. For instance, if self-driving cars relied solely on cameras, this constraint would alter their AI algorithms and performance.

LIDARs, especially cheap LIDARs, have problems with reflective surfaces (like metallic paint on cars), strong lights like car headlights, weather(rain, snow, hail, fog), and have a considerably shorter range than comparable in price radars. Of course, they have much better precision, so some hardware stacks for cars are using both.


Aesthetics analysis with deep learning I'm trying to score video scenes in terms of aesthetics and cinematography features. Basically, how "interesting" a scene or video frame can be for a viewer. Simpler, how attractive a scene is. My final goal is to tag intervals of video which can be more interesting to viewers. It can be a "temporal attention" model as well.

Do we have an available model or prototype to score cinematographic features of an image or a video? I need a starter tutorial on that. Basically, a ready-to-use prototype/model that I can test as opposed to a paper that I need to implement myself. Paper is fine as long as the code is open-source. I'm new and can't yet write a code given a paper.

Aesthetics of images has a strong subjective element and possibility of multiple dimensions depending on purpose of the media. That means:

It is hard to define what we mean by scoring aesthetics.
Given any well-constrained definition, it is then time-consuming to collect relevant data.

However, there is some interest in the machine-learning community, as media quality would be a very useful metric to sort and filter data on (provided the metric is close enough to the end user who wants to select it). As a result, there are data sets, research papers and pre-built models for this.

Media quality training data can be crowdsourced in a variety of ways, including looking at popularity of items on social media, to paying experts to assess large numbers of images. An example of one open dataset compiled by researchers for this purpose is called AVA.

This data might be reduced to image/quality pairs which you can then train a CNN model to predict the quality metric (score out of 10 for example). This might just be a regression, but other more complex loss functions are also considered.

A quick search for existing models brings up Google's NIMA project, which has more than one implementation available as open-source code. NIMA appears to use multiclass classification approach to predict which ratings humans would most likely give the image, and the resulting score is then a weighted average of the predicted scores - the claimed benefit of that seems to be that it better matches how the quality ratings are sourced, and it will better capture split opinions (e.g. half of people think the image is terrible, but half think it is great is a different type of image to one where everyone thinks it is just average).

Here is an implementation of NIMA by Github account "idealo" looks complete with documentation, and ready to use with pre-built scripts.

Just to show this is not a one-off, here's a blog by Andrej Karpathy about using CNNs to rate selfies which includes some introduction to core CNN concepts.


Techniques for sentence transformation and consistency I'm looking for NLP techniques to transform sentences without affecting their meaning, and measures of textual consistency. For example, techniques that could transform active voice into passive voice, such as

  The cat was chasing the mouse.

to

  The mouse was being chased by the cat.

I can think of a number of heuristics one could implement to make this happen for specific cases, but would assume that there is existing research on this in the field of linguistics or NLP. My searches for "sentence transformation" and similar terms didn't bring up anything though, and I'm wondering if I simply have the wrong search terms.

Related to this, I'm also looking for measures of text consistency, e.g., an approach that could detect that most sentences in a corpus are written in active voice and detect outliers written in passive voice. I'm using active vs. passive voice as an example here and would be interested in more general approaches. Any pointers would be much appreciated, thanks!

Strictly speaking, this is impossible. Changing the form of a sentence also changes its meaning. Even active-passive can be important, as you would use it to emphasise what is important: was it relevant what the cat was doing, or was it more relevant what happened to the mouse? True, the purely propositional meaning is not affected by this, but that is only one component of a sentence's meaning.

There has been a lot of work in traditional linguistics about sentence form. You could look at one of the seminal works, Syntactic Structures by Noam Chomsky, where he introduces the concept which later lead to transformational grammar. This influenced a lot of subsequent linguistic approaches, but as far as I am aware transformations are not really that much in the linguistic focus anymore.

For your second question, stylistic consistency, you could look at the work of Douglas Biber. His book Variation across speech and writing introduces a number of (easily extractable) linguistic features that you could use to quantify consistency.


Using an 'operation ID' as a neural network input Sorry if this is basic or covered elsewhere, I am just starting here and I wasn't able to find an answer, but I might have not been searching for the right thing. So:

I am training a neural network to predict current draw in a system. There are a number of obvious numerical inputs, like temperature, counting rate, voltage, etc.

The most predictive thing, however, is what operation the system is doing. So like, if it's doing a 'calibration' then the current profile is much different than if it's in 'standby'. I know that I can just use a different network for each operation, but in this case I have a couple hundred different macros defined and I don't want to have 200+ neural networks retrain all the time.

I also know that I can have a digital value as an input, but my understanding is that it has to be either 0/1. Also, the relationship to operation is not at all correlated - so operation 100 is not necessarily more current draw than 99 or less than 101.

So, is there a way to have an operation ID or something factor in, but not have it be in the linear combination mathematically? So, basically, tell the system to do a different training based on ID or something? I'll be using python and scikit-learn.

Thanks!

Great question.

The operational ID is actually a category, or categorical input. Each category requires its own input signal into the neural network. Each signal is actually an activation which is multiplied by a weight and added to all the other weights and inputs. This means that if you use the same input signal for all your operational IDs then 101 is more important than 50. As you have mentioned these inputs are very different, and therefore must have a binary input for each operation id. If you add more IDs then you have to retrain the neural network.

You also mentioned you will be using scikit-learn, which has built in functions for category data. Have a look at this link on the one hot encoder.


Are there any conferences dedicated to artificial general intelligence? Similarly to What are the scientific journals dedicated to artificial general intelligence?, are there any conferences dedicated to artificial general intelligence?

There are several conferences dedicated to AGI or human-level intelligence, such as

AGI Conferences (organized by AGI society)
Biologically Inspired Cognitive Architectures (organized by BICA Society)
Advances in Cognitive Systems
IEEE Task Force on Towards Human-like Intelligence

The conferences focus on topics such as cognitive architectures, autonomy, creativity, lifelong learning, and formal models of general intelligence. There are also journals associated with these conferences (see [1], [2], [3]). If you want to know more about them, I suggest that you go to their websites.


Are self-driving cars using single frame or multiple frame to make decision? This might be a trivial question but I couldn't find any reliable answers on the internet.

Almost all the neural network architectures for self-driving cars that I have seen on the internet have a feedforward network, previous frames will not help in making the current decision.

I have read somewhere that Tesla uses two last frames captured to make a decision, even then 2 frames will not be that useful in this case.
This might not be very helpful when predicting things ie.. lane cut-ins, as the system needs to observe the vehicle (that is going to cut in) behavior such as turn indicator, vehicle veering towards center lane over time in order to predict.

Can someone explain if this is the way production self-driving card such as Tesla work?

Or Is it something like the below?

Or are they using something like Many to one Recurrent net where inputs are CNN vectors of previous few frames and output is the control?

It varies quite substantially between different self-driving paradigms(rather obviously) but for the most part the vast majority of implementations are using a variety of different reference frames in order to make predictions.

For example, Tesla's Autopilot is being fed many different camera feeds as well as radar and ultrasonic signals that are processed in a variety of temporal contexts.

While, for the most part, all of these programs are very tight-lipped, we can make a variety of assumptions based on the information available and educated assumptions.

As with many large, complex ML/AI systems, there is a large amount of compartmentalization where many different connectionist(or sometimes classic) models are combined(à la youtube recommendation system). Tesla is likely utilizing recurrent and convolutional networks where particular modules(combinations of models) are deciding on specific contexts(temporal or signal-based). These outputs are then most likely fed into an actor network which makes real time decisions.


Why not teach to a NN not only what is true, but also what is not true? I'm not a person who studies neural networks, or does anything that is related with that area, but I have seen a couple of seminars, videos (such as 3Blue1Brown's Series), and what I am always told is that we trying the network over some huge collection of data about what is right. For example, when we are training an AI in order for it to recognise hand written words, what we do is that we give it some hand-written letters, and let it guess the letter. If the guess is wrong, by some means, we adjust the neural network in a way that, next time it will give us the correct result with more probability (the basic description of the "learning" process might not be accurate, but it is not important for sake of the question.)

But it is like teaching some mathematical subject to a student without saying him/her the boundaries of the theorems that we supply; for example, if we teach A implies B, student might be tend to relate A with B, and when he/she has B, s/he might be tempted to say we also have A, so to make sure he/she will not do such a mistake, what we do is to show him/her  a counterexample where we have B, but not A.

This - i.e teaching not only what is true, but also what is not true - especially important in the process of "learning" of a neural network, because the whole process is in a sense "unbounded" (please excuse my vagueness in here).

So, what I would do if I was working on neural networks is that; for example in the above recognition of hand written letter case: I would also show the NN some non-letter images, and also put an option in the last layer as "non-letter" with all those other letters, so that the NN should not always return a letter just to sake of producing a result for a given input, it needs to also have to option to say that "I do not know", in which case it produces the result Not a Letter.

 Question

Is there anyone that has ever applied above method to a NN, and got results ? if so, what were the result compare to the case where there is not option as "I do not know".

Yes this is done routinely. For example this is how the YOLO object detection and classifier system works, to give a real-world for example. In YOLO, the "non-object" classification is "background" i.e. any image segment that doesn't contain one of the types of object we are interested in.

In general, you can add an "other" class to any classifier, provided you have data examples that fit into "other" class to learn and some sense of how often "other" will occur in the production system you are aiming for. Whether you choose to do so depends on the purpose of the model.

Many toy and test models do not include an "other" category, because they are used in a closed way to assess how each machine learning system works. That includes the famous MNIST handwritten digits data set for instance, so if you read tutorials about that, there is an underlying assumption that the trained network will only be presented with other handwritten digits and its only task is to classify them. However, this is not a general assumption for machine learning classifiers in general, just related to the goal of using the MNIST data set.

Adding a new "I do not know" category does not increase the accuracy or performance of a system when that category is not important in the target production system. When such a category is required due to the nature of a task, then the performance metrics will be different - a system that has been trained with some negative examples will likely perform better in that case.


What is the status of the capsule networks? What is the status of the capsule networks?
I got an impression that capsule networks turned out not to be so useful in applications more complicated than the MNIST (at least according to this reddit discussion).
Is this really the case? Or can they be a promising research direction (and if so, is there any specific application for which they seem the most promising)?

ML is full with things that supposed to work better (in theory).

Sigmoid function seems better than ReLu.

L1 seems way better than L2.

Spikes neural network seem to be better than standard neural network.

A shallow neural network with a lot of neurons has more parameters than a deep one with the same amount of neurons. So, in theory, has to be more powerful.

Is common to forget the big impact of the training capacity. And easy to get charmed by its theoretical capacity.
Training process are still in diapers. We lack a good understating of it.
Today, Capsules neural network are a very powerful architecture, hard to train.


What is happening when a reinforcement learning agent trains itself out of desired behavior? I have a reinforcement learning agent with both a positive and a negative terminal state.  After each episode during training, I am recording whether a success or failure occurred, and then I can compute a running ratio of success to failure.

I am seeing a phenomenon where, at some point in time, my agent achieves a reasonably high success rate (~80%) for a 100-episode running average.  However, with further training, it seems to 'train itself out' of this behavior and ends the training sequence with a very low success rate (~10-20%).

I am using an epsilon-greedy strategy whereby epsilon decays linearly from 1.0 to 0.1 for the first 10% of episodes and then remains at 0.1 for the remaining 90%.  As such, the 'training out' appears to occur some time where exploration only occurs with 10% probability.

What could be causing this undesirable behavior? How can I combat it?

This is a common problem and does have a name. It is called "catastrophic forgetting" (link is just to a paper I found randomly when searching for the term).

  What could be causing this undesirable behavior?

It happens only when using function approximation for value functions (e.g. a neural network trained to learn Q values), and is caused by the agent's own success. After a while the only samples that you will train with will be near-optimal high return cases, and a neural network will optimise the Q value approximation only for that recent data in order to get the best loss. This will usually mean much poorer predictions on unseen but different data, as the network overfits to states and actions only seen in the optimal policy. Eventually the agent will explore into an area where its predictions are way off. Then, because Q learning also uses its own predictions to bootstrap new Q values, this can start a runaway feedback process where the agent starts to choose a suboptimal path through the environment.

Inside the hidden layers and weights, the neural network may have lost the ability to differentiate well between the states on the old, worse path and the newer better ones. It didn't need to differentiate, because it stopped needing to reduce loss on any data about the old states. So it will also start incorrectly associating the now poor results with the more optimal paths. It will behave at least partially as if the correct policy was set by the overfit Q predictions, but the values need adjusting - so as well as (correctly) reducing its value predictions of the suboptimal paths it has just encountered, it will also (incorrectly) reduce the value predictions of the optimal paths.

Sometimes, the swing back to receiving lower returns during this process is so strong and/or incorrectly associated with the high return states along with the low ones, that the agent never properly recovers. Other times, the agent can re-learn fully and you get random oscillations over time between optimal and non-optimal agent behaviour.

  How can I combat it?

This is still an ongoing area of research. Here are a couple of things that have worked for me:

Low learning rates, and defences against sudden large gradients (e.g. gradient clipping).
Regularisation. Sadly dropout seems not to work in RL, but weight decay is still useful to prevent over-fitting, and it also helps combat catastrophic forgetting because it prevents bootstrap estimates of long-unseen state/action combinations from returning radically different Q values to the rest of the system.
Keep some early experience around from when the agent was still performing badly - this allows the agent to still train with some bad cases and prevents the Q function predicting that "everything is awesome" because it still has examples to learn from where this is not the case.

For simple environments, such as inverted pendulum, just keeping some very early fully random behaviour in the experience replay table is enough. For instance if you have a table with 10000 observations (of $s, a, r, s'$), keep 1000 of the first experiences in that table and don't discard them when the table is full. For more complex environments, this is not so useful, as the early random behaviour is too far removed from what the agent learns.
The DQN "rainbow" paper uses prioritized experience replay to focus on areas where Q value predictions from the NN are not matching the observations.


How can I learn a graph given nodes with features in a supervised fashion? I have a dataset and want to be able to construct a graph from it in a supervised fashion.

Let's assume I have a dataset with N nodes, each node has e.g. 10 features. Out of these N nodes, I want to learn a graph, i.e. an $N \times N$ adjacency matrix. So, I start with $N$ nodes and all I know is a 10-dimensional feature vector for each node. I have no knowledge about the relation between these nodes and want to figure it out.

Here is an example for $N=6$, but in practice $N$ is not fixed.

So the output I would like to get here is a $6\times6$ adjacency matrix, representing the relations between the nodes (undirected).

Note: N is arbitrary and not fixed. So an algorithm should be able to perform on any given N.

My dataset is labeled. For the training dataset, I have the desired adjacency matrix for each collection of input nodes, which is filled with $0$s and $1$s.

However, the output of the algorithm could also be an adjacency matrix filled with non-integer numbers in $[0,1]$, giving some kind of probability of the nodes being connected (preferably close to $0$ or $1$ of course). So I could easily give a number as the label for each node. In the above example, the labels for the three connected nodes could be class $1$, and so on.

Is there any kind of supervised learning algorithm (e.g. some sort of graph neural network) that can perform these tasks?

It's perfectly reasonable to apply 'traditional' Deep Learning approaches to try and learn an adjacency matrix (a matrix is just a vector of vectors, which can be flattened into a single output vector) but you might need a lot of training data as N gets larger.

Your outputs could certainly have the form of an adjacency matrix, as you describe. Whether it's more useful to have 'boolean' (either 0 or 1)
or 'probabalistic' entries in the matrix depends both on the data and the specifics of your end application.


Why does Clipped Surrogate Objective works in Proximal Policy Optimization In ''Proximal Policy Optimization Algorithms'' , Schulman et al. (2017), page 3

I don't understand why the clipped surrogate objective works.
As written in the article : "With this scheme, we only ignore the change in probability ratio when it would make the objective improved, and we include it when it makes the objective worse".
I feel confused : how can it works if it doesn't take account of objective improvements?

Ok, so I think I have a better understanding of this now.

Firstly, let's remind the main idea of the PPO : staying close to the previous policy. It's the same idea than in TRPO, but the L function is improved.

So, you wanna make "small but safe steps". With clipped surrogate objective, you don't give too much importance to promising actions. You learn that bad actions are bad, so you decrease their probability according to "how bad" they are. But for good actions, you only learn that they are "a little bit good", and their probability will be just slightly increased.

This mechanism allows you to perform small but relevant updates of your policy.

hope this will help someone :)


Is the test time the phase when the model's accuracy is calculated with test data set? When papers talk about the "test time", does this mean the phase when the model is passed with new data instances to derive the accuracy of the test data set? Or is "test time" the phase when the model is fully trained and launched for real-world input data?

If it is not defined otherwise, testing is the phase where the model is passed with new data instances to derive the score of the test set. It should not be confused with validation set.

A validation dataset is a sample of data held back from training your model that is used to give an estimate of model skill while tuning model’s hyperparameters during training. There are a lot of methods of validation with k-fold cross-validation being one of the most popular.

In k-fold cross-validation, the original training set is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for validating the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times, with each of the k subsamples used exactly once as the validation data for the each epoch. The k results can then be averaged to produce a single estimation. The advantage of this method over repeated random sub-sampling is that all observations are used for both training and validation, and each observation is used for validation exactly once.

The validation dataset is different from the test set that is also held back from the training of the model, but is instead used to give an unbiased estimate of the skill of the final tuned model when comparing or selecting between final models.


Why is dropout favoured compared to reducing the number of units in hidden layers? Why is dropout favored compared to reducing the number of units in hidden layers for the convolutional networks?

If a large set of units leads to overfitting and dropping out "averages" the response units, why not just suppress units?

I have read different questions and answers on the dropout topic including these interesting ones, What is the "dropout" technique? and this other Should I remove the units of a neural network or increase dropout?, but did not get the proper answer to my question.

By the way, it is weird that this publication A Simple Way to Prevent Neural Networks from Overfitting (2014), Nitish Srivastava et al., is cited as being the first on the subject. I have just read one that is from 2012:
Improving neural networks by preventing co-adaptation of feature detectors.

Dropout only ignores a portion of units during a single training batch update.  Each training batch will use a different combination of units which gives them the best chance of that portion of them working together to generalize. Note the weights for each unit are kept and will be updated during the next batch in which that unit is selected. During inference, yes, all units are used (with a factor applied to activation...the same factor that defines the fraction used)...this becomes essentially an ensemble of all the different combinations of units that were used.
Contrasted with fewer units, the fewer units approach will only learn what those fewer units can be optimized for.  Think of dropout as an ensemble of layers of fewer units.(with the exception that there are partial weight sharing between the layers)


What are the flaws in Jeff Hawkins's AI framework? In 2004 Jeff Hawkins, inventor of the palm pilot, published a very interesting book called On Intelligence, in which he details a theory how the human neocortex works.
This theory is called Memory-Prediction framework and it has some striking features, for example not only bottom-up (feedforward), but also top-down information processing and the ability to make simultaneous, but discrete predictions of different future scenarios (as described in this paper).
The promise of the Memory-Prediction framework is unsupervised generation of stable high level representations of future possibilities. Something which would revolutionise probably a whole bunch of AI research areas.
Hawkins founded a company and proceeded to implement his ideas. Unfortunately more than ten years later the promise of his ideas is still unfulfilled. So far the implementation is only used for anomaly detection, which is kind of the opposite of what you really want to do. Instead of extracting the understanding, you'll extract the instances which your artificial cortex doesn't understand.
My question is in what way Hawkins's framework falls short. What are the concrete or conceptual problems that so far prevent his theory from working in practice?

The short answer is that Hawkins' vision has yet to be implemented in a widely accessible way, particularly the indispensable parts related to prediction.

The long answer is that I read Hawkins' book a few years ago and was excited by the possibilities of Hierarchical Temporal Memory (HTM). I still am, despite the fact that I have a few reservations about some of his philosophical musings on the meanings of consciousness, free will and other such topics. I won't elaborate on those misgivings here because they're not germane to the main, overwhelming reason why HTM nets haven't succeeded as much as expected to date: to my knowledge, Numenta has only implemented a truncated version of his vision. They left out most of the prediction architecture, which plays such a critical role in Hawkins' theories. As Gerod M. Bonhoff put it in an excellent thesis1 on HTMs,

  "In March of 2007, Numenta released what they claimed was a “research
  implementation” of HTM theory called Numenta Platform for Intelligent
  Computing (NuPIC). The algorithm used by NuPIC at this time is called
  “Zeta1.” NuPIC was released as an open source software platform and
  binary files of the Zeta1 algorithm. Because of licensing, this paper
  is not allowed to discuss the proprietary implementation aspects of
  Numenta’s Zeta1 algorithm. There are, however, generalized
  concepts of implementation that can be discussed freely. The two most
  important of these are how the Zeta 1 algorithm (encapsulated in each
  memory node of the network hierarchy) implements HTM theory. To
  implement any theory in software, an algorithmic design for each
  aspect of the theory must be addressed. The most important design
  decision Numenta adopted was to eliminate feedback within the
  hierarchy and instead choose to simulate this theoretical concept
  using only data pooling algorithms for weighting. This decision is
  immediately suspect and violates key concepts of HTM. Feedback,
  Hawkins’ insists, is vital to cortical function and central to his
  theories. Still, Numenta claims that most HTM applicable problems can
  be solved using their implementation and proprietary pooling
  algorithms."

I am still learning the ropes in this field and cannot say whether or not Numenta has since scrapped this approach in favor of a full implementation of Hawkins' ideas, especially the all-important prediction architecture. Even if they have, this design decision has probably delayed adoption by many years. That's not a criticism per se; perhaps the computational costs of tracking prediction values and updating them on the fly were too much to bear at the time, on top of the ordinary costs of processing neural nets, leaving them with no other path except to try half-measures like their proprietary pooling mechanisms. Nevertheless, all of the best research papers I've read on the topic since then have chosen to reimplement the algorithms rather than relying on Numenta's platform, typically because of the missing prediction features. Cases in point include Bonhoff's thesis and Maltoni's technical report for the University of Bologna Biometric System Laboratory2. In all of those cases, however, there is no readily accessible software for putting their variant HTMs to immediate use (as far as I know). The gist of all this is that like G.K. Chesterton's famous maxim about Christianity, "HTMs have not been tried and found wanting; they have been found difficult, and left untried." Since Numenta left out the prediction steps, I assume that they would be the main stumbling blocks awaiting anyone who wants to code Hawkins' full vision of what an HTM should be.

1Bonhoff, Gerod M., 2008, Using Hierarchical Temporal Memory for Detecting Anomalous Network Activity. Presented in March, 2008 at the Air Force Institute of Technology, Wright-Patterson Air Force Base, Ohio.

2Maltoni, Davide, 2011, Pattern Recognition by Hierarchical Temporal Memory. DEIS Technical Report published April 13, 2011. University of Bologna Biometric System Laboratory: Bologna, Italy.


Methods for fast image comparison I want to implement a real-time system for image comparison (e.g. compare a face with a reference one) on an Odroid. I would like to know what are the most suitable architectures for this task. I started with methods based on triplet loss (like Facenet) but I realized that a real-time solution is not feasible. Are there good, light alternatives?

The problem might not be caused by your loss function. Deep learning models tend to be computationally demanding. Mobile devices are not usually prepared for handling models with high throughput. That being said, you might try to:

Prepare smaller model - less layers, less computation
Mobile models optimization - Google provides some materials on optimizing Tensorflow models for mobile inference:

https://www.tensorflow.org/mobile/prepare_models
https://www.tensorflow.org/mobile/optimizing


How to deal with huge branching factors in real-time? Some time ago playing chess was challenging for algorithms, then Go game which is vastly more complex than compared to chess.

How about playing RTS game which have enormous branching factors limited by its time and space (like deciding what to do next)? What are the successful approaches to such problems?

As you mentioned in the question, you cannot solve all problems with decision trees.  Decision trees usually works well in a turn-based game with a good heuristic function, but in RTS games takes a different approach.

In the case of a very complex RTS game, one could implemented a rule-based AI. For example

given it is the early game use all units to scout for resources
if a certain criteria is met build the base a certain way
if another criteria is met build an army
if the army is big enough, attack
if being attacked by the enemy, bring the units back to the base to defend

Each of these rules could implement various other AI technique, for example use A-star to find the optimal path between a unit's current location and destination.

Further optimization could be done by "grouping" similar units to act like one unit. e.g. calculate the path for the entire group instead of each individual unit.

You could also add finer grained rules, like if enemy is a certain distance from a unit, move closer and attack or retreat to the base (depending on health, ammo, abilities, etc).

The benefit of this approach is that a rule-based system executes very fast as no training or decision trees are necessary and this frees up a lot of system resources for visuals like physics and graphics.

The disadvantage is that if the rule system is not complex enough, the player will easily recognize the pattern and the game will become predictable and boring.  You will also notice that the more different units you add to the game, the exponentially more complex the rule system becomes as you have to cater and test interaction between each type of unit in the game otherwise players might find a weakness in the game design and exploit to complete missions in ways it was not designed to be completed.

One of the reasons why multi-player games are so popular is that you do not play against set rules, but against creative people who have the ability to comes up with new strategies you had never seen before.


Is there a connection between the bias term in a linear regression model and the bias that can lead to under-fitting? Here is a linear regression model
$$y = mx + b,$$
where $b$ is known as $y$-intercept, but also known as the bias [1], $m$ is the slope, and $x$ is the feature vector.
As I understood, in machine learning, there is also the bias that can cause the model to underfit.
So, is there a connection between the bias term $b$ in a linear regression model and the bias that can lead to under-fitting in machine learning?

In machine learning, the term bias can refer to at least 2 related concepts

A (learnable) parameter of a model, such as a linear regression model, which allows you to learn a shifted function. For example, in the case of a linear regression model $y = f(x) = mx + b$, the bias $b$ allows you to shift the straight-line up an down: without the bias, you would only be able to control the slope $m$ of the straight-line. Similarly, in a neural network, you can have a neuron that performs a linear combination of the inputs, then it uses a bias term to shift the straight-line, and you could also use the bias after having applied the activation function, but this will have a different effect.

Anything that guides the learning algorithm (e.g. gradient descent with back-propagation) towards a specific set of solutions. For example, if you use regularization, you are biasing the learning algorithm to choose, typically, smoother or simpler functions. The bias term in the linear regression model is also a way of biasing the learning algorithm: you assume that the straight-line function does not necessarily go through zero, and this assumption affects the type of functions that you can learn (and this is why these two concepts of bias are related!). So, there are many ways of biasing a learning algorithm.

The bias does not always lead you to the best solutions and can actually lead to the wrong solutions, but it is often useful in many ways, for example, it can speed up learning, as you restrict the number of functions that can be learned and searched. The bias (as described in point 2) is often discussed in the context of the bias-variance trade-off, and both the bias and the variance are related to the concepts of generalization, under-fitting, and over-fitting. The linked Wikipedia article explains these concepts quite well and provides an example of how the bias and variance are decomposed, so you should probably read that article for more details.


What should the target be when the neural network outputs multiple Q values in deep Q-learning? I have some gaps in my understanding regarding the performing of the gradient descent in Deep - Q networks. The original deep q network for Atari performs a gradient descent step to minimise $y_j - Q(s_j,a_j,\theta)$, where $y_j = r_j + \gamma max_aQ(s',a',\theta)$.

In the example where I sample a single experience $(s_1,a_2,r_1,s_2)$ and I try to conduct a single gradient descent step, then feeding in $s_1$ to the neural network outputs an array of $Q(s_1,a_0), Q(s_1,a_1), Q(s_1,a_2), \dots$ values.

When doing gradient descent update for this single example, should the target output to set for the network be equivalent to $Q(s_1,a_0), Q(s_1,a_1), r_1 + \gamma max_{a'}Q(s_2,a',\theta), Q(s_1,a_3), \dots$ ?

I know the inputs to the neural network to be $s_j$, to give the corresponding Q values. However, I cannot concretize the target values that the network should be optimized.

When doing gradient descent update for this single example, should the target output to set for the network be equivalent to $Q(s_1,a_0), Q(s_1,a_1), r_2 + \gamma max_aQ(s',a',\theta) , Q(s_1,a_3),...$ ?

Other than what looks like a couple of small typos, then yes.

This is an implementation issue for DQN, where you have decided to create a function that outputs multiple Q functions at once. There is nothing about this in Q learning theory, so you need to figure out what will generate the correct error (and therefore gradients) for an update step.

You don't know the TD targets for actions that were not taken, and cannot make any update for them, so the gradients for these actions must be zero. One way to achieve that is to feed back the network's own output for those actions. This is common practice because you can use built-in functions from neural network libraries to handle minibatches*.

There are some details worth clarifying:

You have substituted the third entry in the array with the calculated TD target because the action from experience replay is $a_2$. In general you substitute for the action taken. Looks like you have this correct.
You have $r_1$ in your experience replay table, but put $r_2$ in your TD target formula. Looks like a typo. Another typo is that you maximise over $a$ but reference $a'$. Also, you reference $s'$ but don't define it anywhere. Fixing these issues gives $r_1 + \gamma \text{max}_{a'}Q(s_2,a',\theta)$
For the TD target it is often worth using a dedicated target network that every N steps is copied from the learning network. It helps with stability. This can be noted as a "frozen copy" of $\theta$ noted $\theta^-$, and the neural network approximate Q function often noted $\hat{q}$ giving formula of $r_1 + \gamma \text{max}_{a'}\hat{q}(s_2,a',\theta^-)$ for your example.

* If you want you can also calculate the gradient more directly from the single action that was taken, and back propagate from there, knowing that all the other outputs will have a gradient componnet of zero. That requires implementing at least some of the back propagation yourself.


Can a LFSR be approximated by a Neural Network? I was wondering whether a LFSR could be approximated by a NN (output or current state). We know that a LFSR is called linear in some sort of mathematical sense, but is that true? Considering it follows Galois field mathematics. So can a Neural Network approximate a LFSR?

Answers with mathematical proof or actual experience is preferred.

According to the abstract of the following book, yes it's possible:

"Prediction of Sequences Generated by LFSR Using Back Propagation MLP"

Abstract:
Prediction of the next bit in pseudorandom sequences is one of the main issues in cryptology in order to prove the robustness of the systems. Linear complexity has served as a reference measurement to evaluate the randomness of the sequences, comparing them with the shortest LFSR that can generate those sequences. Several tools based on artificial intelligence have also been used for the next bit prediction, such as the C4.5 classifier. In this paper, we apply a different approach, the back propagation neural networks, to predict the sequences generated by LFSR. The results confirm that these networks can predict the entire sequence knowing less input patterns than techniques based on classifiers.


Can an AI be trained to generate the outline of a story? I know that one of the recent fads right now is to train a neural network to generate screenplays and new episodes of e.g. the Friends or The Simpsons, and that's fine: it's interesting and might be the necessary first steps toward making programs that can actually generate sensible/understandable stories.

In this context, can neural networks be trained specifically to study the structures of stories, or screenplays, and perhaps generate plot points, or steps in the Hero's Journey, etc., effectively writing an outline for a story?

To me, this differs from the many myriad plot-point generators online, although I have to admit the similarities.  I'm just curious if the tech or the implementation is even there yet and, if it is, how one might go about doing it.

As far as I am aware, this has not been done yet.

I see several problems with this. A neural network is basically a classifier, which matches an input to an output. Both input and output are usually numerical values, though they could be matched to concepts or words.

To train a NN you provide an appropriately encoded input, and the corresponding output. The NN learns the associations between the two, and can then classify unseen input accordingly. This has recently been used to transform images in a particular style etc.

What would the input and output be for generating screenplays? You could use previous scripts as inputs, but what would the output be? It could be narrative 'moves' of some sort, perhaps. So you could train an NN to recognise narrative elements from screenplays.

However, you are still not creating anything, but just recognising stuff. You would need some other input. I guess you could train an NN on "The Simpsons", get a narrative structure, and then present it with an Episode of "Friends" and see what happens. It won't be a new episode of a screenplay, though.

The other way round might work: you feed it narrative moves (a kind of story skeleton), and get a script out. But it would need a lot of (human) post-editing to be at all useful.

I think an NN is the wrong tool to use here. There has been work done with generating stories and screenplays, even way back in the early days of AI. But that was all based on symbolic AI, not on the kind of ML which seems to currently be en vogue. Have a look at James Ryan's website; he has recently written an overview over historic approaches to story (and screenplay) generation.


What is the purpose of the GAN? The Generative Adversarial Network (GAN) is composed of a generator $G$ and a discriminator $D$. How do these two components interact? What is the intuition behind the GAN, its purpose, and how it is trained?

GANs were invented in a bar somewhere in Montreal, Canada. At said bar, the idea was that neural networks could be used for generating new examples from an existing distribution. This was the problem:

Given an input set $X$, can we make a new $x'$ that looks like it should be in $X$?

The classic description of a GAN is a counterfeiter (generator) and a cop (discriminator). The counterfeiter has the same problem, make a piece of paper look like a real currency.
In training a GAN, the input to the generator is random noise, a starting seed so that no 2 results are the same. The generator then makes a new $x'$. The input to the discriminator alternates between an actual $x$ and an $x'$ that the generator made. The discriminator then takes the $x'$ and decides whether it is part of the set $X$. The discriminator is then trained using its answer to ensure that it can properly tell the difference between bad counterfeits and elements of $X$. When the discriminator makes a decision on an $x'$ that the generator made, the generator is updated as well, in order to increase its ability to make new $x'$s that the discriminator will think are in $X$.
Using this simple framework, these 2 networks work against each other (adversarially) to train each other until the generator is making $x'$s so well that the discriminator can't tell the difference between them and the real thing. At this point, the generator can be used to make new pictures of cats or whatever was the goal in the first place.
The primary usage of $D$ is training $G$ in a (sorta) supervised way but after $G$ is trained, it isn't needed for generation. That's not to say having a good discriminator isn't useful. There are many applications where that's exactly what you'd want, but, if the goal is generation, then probably not. The generator is trained in a (sorta) supervised fashion because there is actually a label associated with the $x'$ it generates (whether the discriminator could detect that it was a counterfeit or not).  Unsupervised methods are used when there are no labels and one wants to understand how their data either by clustering or finding a good distance function. Refer to this answer to see the difference between supervised and unsupervised.


How to calculate multiobjective optimization cost for ordinary problems? What I did:
Created a population of 2D legged robots in a simulated environment. Found the best motor rotation values to make the robots move rightward, using an objective function with Differential Evolution (could use PSO or GA too), that returned the distance moved rightward. Gradient descent used for improving fitness.

What I want to do:
Add more objectives. To find the best motor rotation, with the least motion possible, with the least jittery motion, without toppling the body upside down and making the least collision impact on the floor.

What I found:

Spent almost two weeks searching for solutions, reading research
papers, going through tutorials on Pareto optimality, installing
libraries and trying the example programs.
Using pairing functions to create a cost function wasn't good
enough.
There are many multi-objective PSO, DE, GA etc., but they seem
to be built for solving some other kind of problem.

Where I need help:

Existing multi objective algorithms seem to use some pre-existing
minimization and maximization functions (Fonseca, Kursawe, OneMax,
DTLZ1, ZDT1, etc.) and it's confusing to understand how I can use my
own maximization and minimization functions with the libraries.
(minimize(motorRotation), maximize(distance),
minimize(collisionImpact), constant(bodyAngle)).
How do I know which is the best Pareto front to choose in a
multi-dimensional space? There seem to be ways of choosing the
top-right Pareto front or the top-left or the bottom-right or
bottom-left. In multi-dimensional space, it'd be even more varied.
Libraries like Platypus, PyGMO, Pymoo etc. just define the problem using
problem = DTLZ2(), instantiate an algorithm algorithm =
NSGAII(problem) and run it algorithm.run(10000), where I assume
10000 is the number of generations. But since I'm using a legged
robot, I can't simply use run(10000). I need to assign motor values
to the robots, wait for the simulator to make the robots in the
population move and then calculate the objective function cost. How
can I achieve this?
Once the pareto optimal values are found, how is it used to create a
cost value that helps me determine the fittest robot in the
population?

I eventually used the keep-efficient function from this answer to calculate the Pareto front and used the k-means function to calculate the centroid of the front. This gave me the approximate knee-point of the front, which is usually the optimal solution. One of the calculations was to maximise the distance moved in x direction (dx) vs. minimising the energy consumed (e), so since the x axis needed positive maximization and the y axis needed minimization, I inverted the y axis, since min(f(y)) = - max(-f(y)). This helped get the pareto front toward the top right side of the graph and both the x and y axes were maximization objectives. The optimal point calculated was the robot that had the best fitness.


What is the most biologically plausible representation for the actor and critic? Which representation is most biologically plausible for actor nodes? For example, actions represented across several output nodes which may be either

mutually exclusive with each other (e.g., go north, go south, etc),
achieved by winner-takes-all.
NOT mutually exclusive with each other (e.g. left leg forward, right leg forward); these actions may occur concurrently. To go north, the correct combination of nodes must be active.

Similarly which representation is most plausible for critic output nodes?

A single output node that outputs a real number representing the
reward.
A set of output nodes each representing a separate value, achieved by winner-takes-all.

Or do other representations better align with real brains ?

For the actor, I'd say the 'not mutually exclusive' option is more biologically plausible in the context of muscle systems, where the actions can be seen as simultaneous muscle activations. Maybe at a higher level, an agent thinks of the action as 'go north' or 'go south', but the final outputs which have to control muscles at a lower level have to represent simultaneous muscle activations.

For the critic, I'd say the 'single output node' is more biologically plausible. Agents perceive the world in the form of high dimensional inputs, such as images. The approach where a value function is learned in a tabular fashion where you know the value for every single state doesn't really scale very well and is limited to small discrete state spaces. For biological agents, it makes sense to have a function that senses the current state of the environment and outputs a single number that represents the value, which gives the agent an idea of how things are going so far given the actions it took in the past.


What is the difference between the definition of a stationary policy in reinforcement learning and contextual bandit? A stationary policy is a function that maps a state to a probability distribution of actions.

In a contextual bandit problem, a state itself does not include the history. But in a reinforcement learning problem, the history can be used to define a state. In this case, does the history include the rewards revealed thus far as well? If so, the policy is not stationary anymore I guess.

As the title says, I think I am confused about the difference in the definitions of stationary (and/or non-stationary) policy between reinforcement learning and contextual bandit.

What is the difference between the definition of a stationary policy in reinforcement learning and contextual bandit?

There is no difference. A policy decides which action to take in each state. This is usually split into deterministic policies of the form $\pi(s) : \mathcal{S} \rightarrow \mathcal{A}$ and stochastic policies of the form $\pi(s|a) : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}, \text{Pr}[A_t=a |S_t=s]$.

When we say that a policy is static, it means that the mapping from state to action - or distribution of actions - does not change over the time that we are interested in. This definition applies equally in the Reinforcement Learning (RL) and Contextual Bandit settings.

  But in a reinforcement learning problem, the history can be used to define a state.

It can be used this way, but is not required. What is required is the Markov property i.e. that identifying the state also determines all the allowed transitions and rewards, plus their probabilities of occurring given an action.

  In this case, does the history include the rewards revealed thus far as well?

If those can affect the future state transitions and rewards yes. If they generally do not, then you would usually exclude them from the state description.

  If so, the policy is not stationary anymore I guess.

What you are proposing is a system that changes its state depending on rewards seen so far. As above, this is not necessary to define a RL problem, but it is allowed in RL. Whilst in a contextual bandit you assume no rules apply to state transitions, and this is a key difference between contextual bandit and RL settings.

The policy is stationary if its mapping rules remain unchanged.

Your proposed addition to the state does not require changing the policy. The policy is a function of the state. It may choose a different action depending on this historical aspect of the state, but it can still remain a fixed function - its input may change, but the policy function itself does not need to change to account for that.


What kind of data structures are needed to efficiently do back-propagation in a feedforward neural network? In a feed-forward neural network, in order to efficiently do backpropagation, what kind of data structure is needed?

I know the weights can just be stored in an array, and you need pointers of some kind to represent connections from one layer to the next (or just a default scheme/pattern), but is anything else needed for backpropagation to work?

TL;DR: You'll need to store a little bit more to perform backward passes. You'll need to store data from the forward pass. This stored information is used for calculating the gradient.

Overview (warning: not trivial)

  I know the weights can just be stored in an array

You'll need a little more:

To update the weights you need to keep a "cache" of the forward pass intermediate terms. That is, forward propagation can be seen as a series of transformations on your input $X$:
$$X\xrightarrow{\Theta^{[1]}+b^{[1]}} [ Z^{[1]} \xrightarrow{\alpha^{[1]}} A^{[1]}] \xrightarrow{\Theta^{[2]}+b^{[2]}}
\dots
\xrightarrow{\Theta^{[L]}+b^{[L]}} [ Z^{[L]} \xrightarrow{\alpha^{[L]}} A^{[L]}]\xrightarrow{\frac{1}{m}\sum\limits_m\sum\limits_{n_L} loss\{A^{[L]},y\}} J
$$

where:

$Z^{[1]}=\Theta^{[1]}X+b^{[1]}$ (ie the linear part)

$A^{[l]}=\alpha^{[l]}(Z^{[l]})$ (ie element wise activation over linear part)

You need to store the $Z^{[l]}$ & $A^{[l]}$ terms in said "cache." You could store these in an array or some other similar data structure. You need these for calculating the gradient during the backwards pass.

Syntax

$A^{[k]}$ - this means we are indexing by layer (eg $\alpha^{[k]}$ is the activation for k-th layer)

$m$ - is the number of examples in the batch

$n_k$ - denotes the number of neurons in the k-th layer

$L$ - the number of layers (so $n_L$ is the number of neurons in last layer)

$\Theta$ - The set of all weights (notice no superscript)

Backprop

In the case of neural networks the cost is a scalar function of inputs and parameters. To get backprop started calculate the scalar by matrix derivative of the cost with respect to the activations of the last layer call this matrix $dA^{[L]}$. Observe:

$dA^{[L]} = \frac{\partial J(\Theta,X)}{\partial A^{[L]}}$

Next, we calculate scalar-by-matrix derivative of $Z^{[L]}$. Doing this one realizes:

$dZ^{[L]} = \frac{\partial J(\Theta,X)}{\partial Z^{[L]}} = dA\odot\alpha'^{[L]}(Z^{[L]})$

Where $\odot$ denotes element wise (Hadamard) product.

With the above one can make use of the matrix definitions for back propagation:

$\text{(A)}\quad d\Theta^{[l]} = \frac{1}{m}dZ^{[l]}\times (A^{[l-1]})^T$

$\text{(B)}\quad db^{[l]} = \frac{1}{m}\sum_{c=1}^m dZ^{[l](c)}$ (where the new superscript in $dZ^{[l](c)}$ denotes summing along the batch dimension )

$\text{(C)} \quad dZ^{[l]}= dA^{[l]}\odot \alpha^{'[l]}(Z^{[l]})$

$\text{(D)}\quad dA^{[k]} = (\Theta^{[k+1]})^T\times dZ^{[k+1]}$

And of course the wight updates are:

$\Theta^{[L]} \leftarrow \Theta^{[L]} - \frac{\eta}{m}d\Theta $

$b^{[L]} \leftarrow b^{[L]} - \frac{\eta}{m}db $

(where $\eta$ is the learning rate)

Observe, how the forward pass terms are used during the backprop calculations.

A recommendation

Take the A. Ng deep learning specialization. He does a good job explaining the intuition and even has a project to implement this. Though, he does not derive the back propagation equations. You can find a not so easy derivation here.


Neural networks for sports betting I want to design a neural network that can be used for predicting sports scores for betting, specifically for American football.  What I’d like to do is create a kind of profile for each game based on the specific strengths and weaknesses of each team.

For example, let’s say two teams have the following characteristics:

Team A:

Passing Offense Rating: 5
Rushing Offense Rating: 2

Team B:

Passing Defense Rating: 3
Rushing Defense Rating: 4

I’d like to be able to search for historical games where two teams have similar profiles.  I could perhaps then narrow it down to games with profiles that have statistically significant historical outcomes (i.e., certain types of matchups are likely to result in similar results).

In reality, I’d have dozens of team characteristics to compare.  I would then need to assign weights of importance to each characteristic, which could be used to further ensure effective selection of similar games.

I think I could do this like a convolutional neural network where there is an additional filter applied to the characteristics for the weights.  Are there any other ways that are specifically applicable to this strategy?

This is probably not going to work well as a way to make money. People with far larger budgets, and far more training, are already milking out any money to be made this way. This is probably their day job, and they are good at it.

That said, here are some ideas:

You do not need or want to use a convolutional network for this. Convolutional networks are useful when you want to detect a pattern invariant to translation in complex inputs. The only translation you have is that the teams could appear in either order. Just input them in both possible orders if this is a concern.
You want to find similar games. You do not need or (generally) want to use a neural network to do this. As a starting point, normalize the features and compute the nearest neighbors of an input point directly. Algorithms for doing this are fast and will return all similar points. You can even (with small modifications) output the degree of similarity as a numeric value.
If you want to predict who will win, you will probably have much better luck doing that directly. Build a feed-forward neural network that predicts which team won from the features you gathered. This is a routine classification task, and the resulting model will probably work better than finding similar games and then trying to manually determine what to do.
If you care about scores instead of just who won, use a feed forward network for regression instead of classification. The task is almost identical.


Why is the validation loss less than the training loss, and what can be said about the effect of the learning rate? I have the following results I am trying to make sense of. I have attached the loss curves here for reference.

As you can see, the first issue is that the validation loss is lower than the training loss. I think this is due to using a pre-trained model with a high dropout rate (please correct me if I am wrong here).

As one can see, the mean_auc score is increasing consistently, and so it seems that the network is indeed learning something and the validation loss is also better behaved relatively.

The training loss is what bugs me a lot. It is not at all consistent and varies a lot. This is a naive question, but is this graph giving me any sort of information about the learning rate, etc, or am I in a situation wherein everything is incorrect essentially?

Any response would be really appreciated.

This is very difficult to tell with the information provided, but the phenomenon  is something that I have encountered many times before. Sometimes this is not a bad thing, here are some possible considerations/explanations:

Data from the training set could be identical or leaking in to the validation set.
Using a high dropout rate can cause this as well as other generalization techniques.
The training set may contain outliers, contributing to higher loss and difficulty in learning.

Also, note that the there is not so much an issue with the training loss bouncing around as long as it decreases over time. You may want to try different optimizers as well to see if that changes this behavior.


Are probabilistic models dead ends in AI? I am a strong believer of Marvin Minsky's idea about Artificial General Intelligence (AGI) and one of his thoughts was that probabilistic models are dead ends in the field of AGI.

I would really like to know the thoughts and ideas of people who believe otherwise.

When considering effective approaches to AGI, one must extrapolate outwards to the types of modelling (and therefore inputs) that would be necessary to achieve any general utility. One consideration might be the fundamental "building blocks" of our physical world, and understanding the movements of these, can lead to accurate predictions of (all) occurrences. These fundamental elements are called (generally) subatomic particles and are anything but discrete values. In quantum field theory, the more accurate you are able to measure position, the less accurate you may know a quarks momentum (and vice versa). Our world, at the most fundamental layer, is probabilistic when observed. This is all not to say that an understanding of quantum mechanical kinematic descriptions is the only methodology to achieve true AGI, but to say probabilistic models, and therefore uncertainty, is a dead-end seems radically inaccurate.

That said, Dr.Minsky didn't really feel probabilistic models were dead ends. The emerging view in the field, one that Dr.Minsky urged for years, is that connectionism alone couldn't exclusively lead to AGI due to its uniformed structure. If you are unaware, connectionism is the concept of creating models around discrete units of representation (neurones in a neural net for example). You see the problem we have identified isn't that probabilistic models are inaccurate, it's that our current approach doesn't express the biological realism necessary for AGI (although sufficient for specific intelligence).

[I briefly worked with Dr.Minsky at the AI Lab before his passing last year, a hilarious man and brilliant scientist.]


Which of these two numerical methods for z-score normalisation is preferable, in multivariate linear regression? In the exercise Exercise 3: Multivariate Linear Regression, by Andrew Ng, the author suggests to "scale both types of inputs by their standard deviations and set their means to zero".

$$x_{n e w}=\frac{x-\mu}{\sigma}$$

Method 1

The author provides the following Matlab (and Octave) code to scale the inputs.

x = [ones(m, 1), x];
sigma = std(x);
mu = mean(x);
x(:,2) = (x(:,2) - mu(2))./ sigma(2);
x(:,3) = (x(:,3) - mu(3))./ sigma(3);

Method 2

But why not simply scale the inputs between zero and one, or, divide by the maximum?

x_range=max(x)
x(:,2) = (x(:,2)/x_range(2));
x(:,3) = (x(:,3)/x_range(3));

I have done the exercise with method 2 and these are the results.

Question

Is there a computational advantage with the first method over the second method?

No, there is no computational advantage of the second method over the first, if you neglect the computational requirements for the calculation of $\sigma$ and $\mu$.

We generally use the first method for better results. This is because if you separate your dataset into train and test data, then you may normalise the train data perfectly between $0$ and $1$ by taking $max$ and $min$ from the train dataset. Your algorithm will only see data between $0-1$, but there is no way to ensure normalising the test data on the same $max$ and $min$ will result in the data being between $0$ and $1$..

So a better way is to use the Gaussian Distribution method, as the only assumption it makes is the data distribution is Gaussian which is just about true for any phenomenon (after some functional transformations which might be handled by the ML algorithm). I answered a similar question here.


How can the A* algorithm be optimized? How can the A* algorithm be optimized?

Any references that shows the optimization of A* algorithm are also appreciated.

The first step of optimisation is to measure where inside the implementation most time is spent -- you don't actually optimise the algorithm itself, but a specific implementation of it. This step should give you an overview of where you can make improvements. Speculative changes usually don't do much.

There are several aspects of A* which would be candidates. You are keeping a list of paths currently under investigation. If it is very long, processing might take more time. You could try and prune this list (though it might mean that you won't actually find the best path anymore). Or you could investigate efficient ways of storing it.

Each step you need to compute the 'cost' of each path. This could be cached to stop you from repeating calculations. The heuristic function could be slow -- again something to investigate.

Let me stress again that you need to measure the performance first. There is no point in randomly trying to make things faster; you could waste a lot of time for very little gain. And typically with optimisation, readability of the code suffers, and it becomes harder to maintain. You might find that other parts of your system take a lot longer than the actual path finding, so your time might better be spent on those.


How do I predict if it is rainy or not? I'm building a weather station, where I'm sensing temperature, humidity, air pressure, brightness, $CO_2$, but I don't have a raindrop sensor.

Is it possible to create an AI which can say if it's raining or not, with the help of the given data above and maybe analyzing the slope from the last hour or something? Which specific technology should I use and how can I train it?

This sounds like a great project, although this exact setup limits your options somewhat.  Supervised machine learning approaches are effectively ruled out because you don't have the necessary training data to develop a model (i.e. the dependent variable: whether it is raining or not).  You could look at accessing similar data (from what source depends on your geographical location) that includes precipitation information, and attempt to train a model based on that, before applying it to your own data to 'predict' the current conditions.

If you want a model to determine whether it's raining or not (rather than attempting to define the type, rate or other characteristics of precipitation) then you need a binary classification model.  If you can get the data to train a supervised model, there are many such models available.  Which one to choose depends on a number of factors – enough factors that it is as much an art as a science (and often the best way to choose is just to try several and pick the best one).  XGBoost is a popular choice for a wide range of problems, including binary classification.  State Vector Classifiers (SVCs, a subtype of State Vector Machine, SVM) are also common.  Of course there are neural networks also.  Depending on how the dependent variable correlates with the independent ones, you may find a simpler model to be more effective.  If you choose to try a neural network, I suggest keeping it simple and starting with a Multi-Layer Perceptron (MLP).

Alternatively, you could try an unsupervised approach, which commonly takes the form of clustering.  In this instance, you would look to try clustering algorithms, such as DBSCAN, to see if the independent variables naturally form groups (clusters) which differentiate between when it is raining or not.  Of course you can cluster the data using an unsupervised model, but you would still need to then correlate that with rainfall in order to make sense of the model.

There are a number of challenges you may need overcome.  For example, depending on the frequency with which the data is sampled and stored, you may need to downsample it to a manageable size.  Presumably you also have the time at which the data was collected.  If not, you would also need to make use of the brightness sensor to differentiate between night and day, which may confound or over-power any cooling effect from rainfall.  (You could look at clustering the data, based on temperature and brightness, in order to create a new feature which indicates night and day.)

To summarise, there are numerous approaches you could take, all of which will, at some point, require data on whether it is/was raining or not in order to train a model.  If you have suitably localised weather data available, you may be able to train a model using that, synchronised via time and/or the other data you'll be collecting.  The problem, as stated, is a binary classification one (as opposed to a time series prediction one, i.e. weather forecasting, for example).  So look into classification and clustering models for supervised and unsupervised machine learning respectively.  Scikit Learn for python or Caret for R are commonly used packages for the types of models I've described.


Are these two definitions of the state-action value function equivalent? I have been reading the Sutton and Barto textbook and going through David Silvers UCL lecture videos on YouTube and have a question on the equivalence of two forms of the state-action value function written in terms of the value function.

From Question 3.13 of the textbook I am able to write the state-action value function as
$$q_{\pi}(s,a) = \sum_{s',r}p(s',r|s,a)(r + \gamma v_\pi(s')) = \mathbb{E}[r + \gamma v_\pi(s')|s,a]\;.$$
Note that the expectation is not taken with respect to $\pi$ as $\pi$ is the conditional probability of taking action $a$ in state $s$. Now, in David Silver's slides for the Actor-Critic methods of the Policy Gradient lectures, he says that
$$\mathbb{E}_{\pi_\theta}[r + \gamma v_{\pi_\theta}(s')|s,a] = q_{\pi_\theta}(s,a)\;.$$

Are these two definitions equivalent (in expectation)?

These two definitions are not exactly the same, even though they have a very similar formulation. David Silver's notation is probably an abuse of notation.

The first difference between those two definitions is that, in the case of David Silver's slides, the policy is parametrized by $\theta$ (i.e. the policy could be represented e.g. by a neural network), while, in the first case, the policy is not parametrized, which is a more general way of writing the definition of the state-action value function, because you do not assume how you will represent the policy.

The second apparent difference (as you have noticed) is that, in the first case, the expectation is taken with respect to transition probability function, while, in the second case, it seems to be taken with respect to the parametrized policy. In the second case, you do not probably know the transition probability function, so you will compute your state-action value function based on your current best estimate of the policy.

Now, as Brale pointed out in the comments (please, read the comments under this answer!), the definition of the state-action value function is always the same, even if you don't know the MDP (because that's a definition and it doesn't matter if you know the transition probability or not). However, in the case of David Silver's slides, by that notation, I think he means he will compute the state-action value function based on the current estimate of the policy.


What do the terms "front-end" and "back-end" refer to in this article? I found the terms front-end and back-end in the article (or blog post) How to Develop a CNN for MNIST Handwritten Digit Classification. What do they mean here? Are these terms standard in this context?

I do not think these are formally defined.

The distinction is just to facilitate discussion of the NN architecture: e.g., you may have a few convolutional layers with pooling as a front-end, and a different architecture as a back-end (in a text-book architecture, just a fully-connected layer. But to get wild, maybe LSTM? To really get wild, BERT?).

In the end (no pun intended), computers do not care if a layer is seen by humans as a front-end or a back-end.


Why are denser layers needed in computer vision neural nets? Many neural net architectures for computer vision tasks use several convolutional layers and then several fully-connected (or dense) layers. While the reasons for using convolutional layers are clear to me, I don't understand why the dense layers are needed. Can't high accuracy be achieved with only convolutional layers?

Convolutional layers are added in order to extract features from the image (like edges, corners, textures). After extracting those features, you feed them to a fully connected neural network to get the prediction.

Let's take an example, consider you want to classify the cat's image. But you decided to do this by only using the convolutional layer. So, you feed the image to a convolutional layer. After passing through some layer it extracted the key features from the cat's image and at the end of the layer, you add a single neuron(since we have a single class to predict). Now it's ready to classify it as a cat. But unfortunately, it's misclassified the image. Why? Now let us answer the question.

Since convolutional layers are not fully connected layers, the neuron added at the last layer is only connected with a handful of neurons from the previous layer. So it misses the key features (encapsulated by some other neurons it is not connected with them) in order to detect this image as a cat. In order to get the prediction right, you have to add a dense layer(a fully connected layer) at the end of the convolutional layer so that it can get all of those extracted features.

If this does not satisfy you, please, ask the question more specifically and also edit the question because linear layer does not call fully connected layer.


Why is it called Latent Vector? I just learned about GAN and I'm a little bit confused about the naming of Latent Vector.

First, In my understanding, a definition of a latent variable is a random variable that can't be measured directly (we needs some calculation from other variables to get its value). For example, knowledge is a latent variable. Is it correct?
And then, in GAN, a latent vector $z$ is a random variable which is an input of the generator network. I read in some tutorials, it's generated using only a simple random function:

z = np.random.uniform(-1, 1, size=(batch_size, z_size))

then how are the two things related? why don't we use the term "a vector with random values between -1 and 1" when referring $z$ (generator's input) in GAN?

It is called a Latent variable because you cannot access it during train time (which means manipulate it), In a normal Feed Forward NN you cannot manipulate the values output by hidden layers. Similarly the case here.

The term originally came from RBM's (they used term hidden variables). The interpretation of hidden variables in the context of RBM was that these hidden nodes helped to model the interaction between 2 input features (if both activate together, then the hidden unit will also activate). This principle can be traced to Hebb's rule which states "Neurons that fire together, wire together." Thus RBM's were used to find representation of models, in a space (generally lower dimensional than than the original). This is the principal used in Auto Encoder's also. Thus as you can see we are explicitly, not modelling the interaction between 2 features, how the process is occurring is "hidden" from us.

So, the term latent basically can be attributed to the following ideas:

We map higher dimensional data to a lower dimensional data with no prior convictions of how the mapping will be done. The NN trains itself for the best configuration.
We cannot manipulate this lower dimensional data. Thus it is "hidden from us.
As we do not know what each dimension means, it is "hidden" from us.


Does IBM Cloud Private for Data run on public clouds like AWS or Azure? I just started using IBM Cloud Private for Data this week and I wasn't sure if I can use other public clouds to connect with my ICP for data account. So I spoke with an IBM representative and I wanted to share their responses...

Yes, ICP for Data can run on any public cloud infrastructure as long as it supports Docker containers and Kubernetes.

Already tested: Amazon AWS, IBM Cloud, Azure, and Red Hat OpenStack


Is it possible to combine two neural networks trained on different tasks into one that knows both tasks? I'm relatively new to artificial intelligence and neural networks.
Let's say I have two different fully trained neural networks. The first one is trained for mathematical addition and the second one on mathematical multiplication. Now, I want to combine these two neural networks into one that knows about both operations.
Is this possible? Is there a representative name for this kind of technique?
I had read something about bilinear CNN models that sounds similar to what I'm looking for, right?

Combining two different fully trained neural networks is not only feasible, it is commonly done. Let's look at the example given as two concepts involving integers, $C_a$ and $C_m$.
$$ C_a: \mathcal{Y} = f_a (\mathcal{X}) = x_0 + x_1 $$
$$ C_m: \mathcal{Y} = f_m (\mathcal{X}) = x_0 \, x_1 $$
Now let's define a palette of operations, including these two binary operations, that can be used to construct, a concept $C_e$, an expression comprised of an arbitrary hierarchy of addition, multiplication, constants, and substitution.
$$ C_e: \mathcal{Y} = f_e (\mathcal{X}) \; \text{, where} $$
$$ f_e \in \{f_a, f_b\} \; \land \; i \in \{0, 1\} \; \land \; ( \, x_i \in \mathbb{I} \; \lor \; x_i \in \mathcal{Y} \, ) \; \text{.} $$
Now, one artificial network can be trained to approximate $f_a$ within a concept class $\mathbb{C}$ of which $C_a$ and $C_m$ are members, using labeled examples of correct integer additions and another artificial network can be trained to approximate $f_b$ within that same concept class, using labeled examples of correct integer multiplications.
An expression involving both can be trained to approximate arbitrary product of sums or sum of products under specific conditions. It's unclear though if this is what you are looking for.
Normally, one wouldn't train a network to perform operations that are already known. Training is normally used to model operations that are not known.
(Bilinear Convolutional Neural Networks (B-CNNs), introduced in Bilinear CNNs for Fine-grained Visual Recognition, 2017, Tsung-Yu Lin, Aruni RoyChowdhury, Subhransu Maji, is an approach to using two CNNs in conjunction to provide two fine and course visual recognition in the same way the human visual system can have a dual awareness of detail and panorama. B-CNNs probably don't apply to the scenario given in the question.)


How does a single neuron in hidden layer affect training accuracy I'm currently a student learning about AI Networks. I've came across a statement in one of my Professor's books that a FFBP (Feed-Forward Back-Propagation) Neural Network with a single hidden layer  can model any mathematic function with accuracy dependant on number of hidden layer neurons. Try as I might I cannot find any explanation as to why that occurs - could someone maybe explain the question why that is?

The claim that Neural Network with a single hidden layer can model any functions is proven in Cybenko's Approximation by superpositions of a sigmoidal function.
https://link.springer.com/article/10.1007/BF02551274
check also: https://en.wikipedia.org/wiki/Universal_approximation_theorem
The thing is that the neural network using sigmoidal functions, which are non-linear functions can.


Why would the reward of A3C with LSTM suddenly drop off after many episodes? I am training an A3C with stacked LSTM.
During initial training, my model was giving descent +ve reward. However, after many episodes, its reward just goes to zero and is continuing for a long time. Is it because of LSTM?
Is it normal?
Should I expect it to work after the training is over or just terminate the training and increase the density of my network?

The thing you're explaining is not impossible for a RL model, but it's rare. That's a known thing that some RL algorithms work or don't work depending on a random seed. I implemented the same model once to play KungFuMaster-v0. It was during a university RL course and the code seemed fine (actually 2 people including teacher looked at it very carefully and they didn't find any bugs). I remember calling it 10 times a row and one time out of ten it showed that nasty behavior. There were teacher's notes in the task: if the reward suddenly drops to 0 and stay there for a long time, check your code, there's a high probability of a bug. So I'd say if your net works just fine 9 times out of 10, probably there are no bugs, otherwise if I were you, I'd carefully check the code.