Assuming an AI can die, who manages the state? AI death is still unclear a concept, as it may take several forms and allow for "coming back from the dead". For example, an AI could be somehow forbidden to do anything (no permission to execute), because it infringed some laws.

"Somehow forbid" is the topic of this question. There will probably be rules, like "AI social laws", that can conclude an AI should "die" or "be sentenced to the absence of progress" (a jail). Then who or what could manage that AI's state?

Following on from your own software verification-based answer to this question, it seems clear that ordinary (i.e. physical), notions of death or imprisonment are not strong enough constraints on an AI (since it's always possible that a state snapshot has been or can be made).

What is therefore needed is some means of moving the AI into a 'mentally constrained' state, so that (as per the 'formal AI death' paper) what it can subsequently do is limited, even if escapes from an AI-box or is re-instantiated.

One might imagine that this could be done via a form of two-level dialogue, in which:

The AI is supplied with percepts intended to further constrain it
("explaining the error of it's ways", if you like).
Its state snapshot is then examined to try and get some indication of whether it is being appropriately persuaded.

In principle, 1. could be done by a human programmer/psychiatrist/philosopher while 2. could be simulated via a 'black box' method such as Monte Carlo Tree Search.

However, is seems likely that this would in general be a monstrously lengthy process that would be better done by a supervisory AI which combined both steps (and which could use more 'whitebox' analysis methods for 2.).

So, to answer the question of "who manages the state", the conclusion seems to be: "another AI" (or at least a program that's highly competent at all of percept generation/pattern recognition/AI simulation).


When is it time to switch to deep neural networks from simple networks in text classification problems? I did an out of domain detection task (as a binary classification problem) and tried LR and Naive Bayes and BERT but the deep neural network didn't perform better than LR and NB. For the LR I just used BOW and it beats the 12-layer BERT.

In a lecture, Andrew Ng suggests "Build First System Quickly, Then Iterate", but it turns out that sometimes we don't need to iterate the model into a deep neural network and most of the time traditional shallow neural networks are good/competitive enough and much simpler for training.

As this tweet (and its replies) indicate, together with various papers [1, 2, 3, 4 etc], traditional SVM, LR, and Naive Bayes can beat RNN and some complicated neural networks.

Then my two questions are:

When should we switch to complicated neural networks like RNN, CNN, and transformer and etc? How can we see that from the data set or the results (by doing error analysis) of the simple neural networks?
The aforementioned experiments may be caused by the simple test set, then (how) is it possible for us to design a test set that can fail the traditional models?

Source: https://blog.easysol.net/building-ai-applications/

When data is too big, complex and nonlinear it's time to try deep learning. It's always good to try to add some layers to see it can eliminate bias and don't lead to high variance.

Deep learning models can be tweaked(hyperparameters) and regularized(parameters), and it is worth the work.


Is it possible to restore scrambled/damaged text? Given an ease with which a human can read a text scrambled in a special way, like the following passage popular a while ago

  Aoccdrnig to a rscheearch at Cmabrigde Uinervtisy, it deosn't mttaer
  in waht oredr the ltteers in a wrod are, the olny iprmoatnt tihng is
  taht the frist and lsat ltteers be at the rghit pclae. The rset can be
  a toatl mses and you can sitll raed it wouthit porbelm. Tihs is
  bcuseae the huamn mnid deos not raed ervey lteter by istlef, but the
  wrod as a wlohe.

could you train a computer to restore the original? Are there any studies/known instruments that can achieve this?

I am talking about mildly scrambled/damaged text, like the above, not a totally random permutations or random omission of letters.

You can rephrase your question as a statistical problem, where you for each word or n-gram try to predict the most likely correctly spelled version of each word or n-gram based on your knowledge of the distributions of these across a corpus of text in the target language.

This is how spell-checkers work.

Peter Norvig (who wrote Artificial Intelligence: A Moderne Approach with Stuart Russell) has an article on the this with a short, easy-to-read code example and a description of the statistics behind it on his web page: How to write a Spelling Corrector.


How to take the optimal batch_size for training a model? I have an image dataset, which is composed of 113695 images for training and 28424 images for validation. Now, when I use ImageDataGenerator and flow_from_dataframe, it as the parameter batch_size.
How can I take the correct number for batch_size because both numbers cannot be divided by the same number? Should I need to drop four images in the validation data to make them batch_size of 5? Or is there another way?

This Cross Validated post might answer your question.
In a nutshell:

A single batch (that is all your data in one batch) will result in a smooth trajectory on the loss surface. The drawback is that all your data might not fit into your memory. Which is highly likely for ~100k images.

One image per batch (batch size = no. examples) will result in a more stochastic trajectory since the gradients are calculated on a single example. Advantages are of computational nature and faster training time.

The middle way is to choose the batch size in such a way that your batch fits into memory and gradients behave less 'noisy'. To be honest there is no 'golden' number, personally I like to choose powers of two.
Don't worry that your data is not divisible by the batch size. Libraries will take care about that internally, the last batch will just be a smaller than the defined batch size ($N \text{ mod } b$).


Is it a great misconception that the softmax is an activation function? An activation function is a function from $R \rightarrow R$. It takes as input the inner products of weights and activations in the previous layer. It outputs the activation.

A softmax however, is a function that takes input from $R^p$, where $p$ is the number of possible outcomes that need to be classified. Therefore, strictly speaking, it cannot be an activation function.

Yet everywhere on the net it says the softmax is an activation function. Am I wrong or are they?

I see no problem in regarding the softmax as a particular activation function which takes a vector input and produces a vector output. In fact, the sigmoid function can be viewed as a two-dimensional softmax in which one of the two inputs is hardwired to zero while the corresponding output is neglected.


How can I classify houses given a dataset of houses with descriptions? I have a dataset with a number of houses, for each house, I have a description. For example "The house is luxuriously renovated" or "The house is nicely renovated". My aim is to identify for each house whether it is luxuriously, well or poorly renovated. I am new to NLP so any tips on how to approach this problem would be much appreciated.

It all depends on what kind of annotations or other variables are present in your dataset. I see 2 possible scenarios here:

your dataset is made only of houses descriptions, without any indication of their luxury level.

you have annotations regarding the luxury level or another similar variables from which you can infer the luxury level (like the house price for example).

In the first case there's not much you can do except for trying to apply some unsupervised algorithms or transfer learning. Usually in NLP unsupervised techniques are used for tasks like Topic or language modeling, both of which are not really helpful for your specific application since they work at a really abstract level trying to learn relationships between words or documents in corpora containing huge variety of texts. The best you could try could be preprocessing the data to extract specific terms like entities (cities names, furnitures names, etc.) and adjectives from each description, and then try cluster them into n clusters were n is the number of classes you're interest in by applying for example Latent Dirichlet Allocation. Even though everything is possible with the right time and patience, I would never follow this road, especially because it relies on a perfect preprocessing, that involves already transfer learning, for the name entity recognition part for example. And even tough libraries like SpaCy offer really good models to perform these tasks, once you have these kind of annotations a rule based approach would probably be faster and easier to build than another unsupervised model, e.g. creating a simple dictionary contacting names of luxurious furnitures name and adjectives that indicate if the house is expensive would probably be sufficiently good.
In the second case, the story change completely because if you have annotations you could rely on supervised learning. If you already have explicit annotations like "luxuriously renovated" "nicely renovated" nothing stops you from trying to train whatever architecture you feel more comfortable with on this classification task. Even simple architecture like CNN that are easy and fast to train usually achieve good results in classification, and you definitely want to leverage some pre-trained embedding vectors like GloVe as an input feature (every deep learning framework like tensorflow or pytorch already implement the possibility to use them).
To conclude, if you don't have annotations you might try LDA just to check if you're lucky but if I were you I would start annotating data as quick as I can. A good practice in this case is to also ask someone else to perform the annotations, not only to be faster in the creation of the dataset but also to then calculate the Inter Annotator Agreement score, that gives an indication about the quality of the annotations (if the score is low, the dataset quality is poor and no model will be able to learn something from it).


Is it legal to license and sell the output of a neural network that was trained on data that you don't own the license to? Is it legal to license and sell the output of a neural network that was trained on data that you don't own the license to?  For example, suppose you trained WaveNet on a collection of popular music. Could you then sell the audio that the WaveNet produces? There are copyright restrictions on using samples to produce music, but the output of a generative neural network might not include any exact replicas from the training data, so it's not clear to me whether those laws apply.

Disclaimer: I am not an attorney and this does not constitute formal legal advice.

If the output is novel the copyright resides with the creator

In this case, almost certainly the human who utilizes the algorithm†.  There was a recent US patent case "Dabus" [U.S. Patent Application No.: 16/524,350] where the human programmers tried to claim an AI as an inventor, which was rejected by USPTO.  This is largely because an inventor is defined as a "natural person".  But it's an interesting challenge to the notion of inventorship and authorship.
An argument regarding output specifically as a salable commodity would be that most human creative endeavor is result of previous work that forms the basis for the novel arrangement of elements in the form of an original work.
Music is more specific because melodies are mathematical, and what are traditionally protected, although Blurred Lines case blurred this line, in that the ruling there was the production aspect, the "feel", was what was plagiarized.
And, in fact, algorithmic music generation, which is heavily utilized in pop music, depends on combinatorial functions producing novel sequences, not protected by prior copyright.
Samples are clear cut—it is excerpting from copyrighted work.  A wave form is then copyrightable, in that a work of recorded music is simply an arrangement of waveforms in some combination and sequence.
If the output is procedural, and a process that can be automated, that would enter the domain of patent law, utility patents specifically.
Design output would also be in the realm of design patents, which are specific/generalized arrangements of elements in products that are not processes.

A note on liability:
Law is a process, where precedent plays a major factor.  Because law uses natural language, there is ambiguity, and jurisprudence is the process of clarifying the meaning and application via challenges.  Especially when in uncharted territory "no one knows" until a case has been ruled on, and rulings can be challenged all the way up to the Supreme Court, which may or may not accept a suit.
Damages for copyright are 3x, but damages can be very hard to prove.  Intellectual Property litigation is also enormously expensive, thus unlikely to be pursued unless there is a potential financial benefit.  (The first step in any potential copyright violation is typically a "cease and desist" letter, with no legal action if acceded to. In a case such as "Blurred Lines", where there is significant financial return to the alleged infringer, the award or settlement presumably exceeds the cost of litigation.)
However, deep-pocket players can use litigation, or threat of litigation, to disincentivize competitors, and, if the target has limited resources, can produce the desired outcome regardless of the strength of the claim.  (It's not uncommon in law in general to file frivolous suits as a strategy, although there is typically a financial penalty  such as reimbursement of the defendant's legal fees if the suit is found to be frivolous and dismissed.) "Patent trolling" became so much of a problem before 2013, the entire philosophy of patentability, what is patentable, had to be revisited.

†Who has rights to an algorithmic process is also a legal question.
Software is patentable, being simply a type of computer, regardless of medium, but most software is not patented, and protected instead as "trade secrets" and via non-disclosure agreements.  (This may be due to the glacial pace of the patent process compared to the software development process, but also per the from the necessity of making patents public.  i.e. if Google patented their search algorithm, it would be an instruction manual on how to exploit it.)
Software is copyrightable as the specific lines of code.  Copyright naturally resides with the creator, but the process of copyright is be utilized to document the right against potential infringement.


Using DDPG for control in multi-dimensional continuous action space? I am relatively new to reinforcement learning, and I am trying to implement a reinforcement learning algorithm that can do continuous control in a custom environment. The state of the environment is composed of 5 sequential observations of 11 continuous values that represent cells (making the observation space 55 continuous values total), and the action is 11 continuous values representing multipliers of those cell totals.
After preliminary research, I decided to use Deep Deterministic Policy Gradient (DDPG) as my control algorithm because of its ability to deal with both discrete states and actions. However, most of the examples, including the one that I am basing my implementation off of, have only a single continuously valued action as the output. I have tried to naively change the agent network from outputting a single value to output a vector of values, but the agent does not improve as all, and the  set of outputs seems to split into two groups near either the maximum value or the minimum value (I believe the tanh activation on the output has something to do with it) with the values in those groups changing in unison.
I have two questions about my problems.

First, is it even possible to use DDPG for multi-dimensional continuous action spaces? My research leads me to believe it is, but I have not found any code examples to learn from and many of the papers I have read are near the limit of my understanding in this area.

Second, why might my actor network be outputting values clustered near its max/min values, and why would the values in either cluster all be the same?

Again, I am fairly new to reinforcement learning, so any advice or recommendations would be greatly appreciated, thanks.

First, is it even possible to use DDPG for multi-dimensional
continuous action spaces?

Yes, DDPG was primarily developed to deal with continuous action space you can find out more here, here and here.

I have not found any code examples to learn from and many of the
papers I have read are near the limit of my understanding in this
area.

You can check kerasRL ddpg implementation it is quite easy to understand and can solve your problem of a model with multiple outputs..

why might my actor network be outputting values clustered near its
max/min values, and why would the values in either cluster all be the
same?

I think it is just a convergence problem try out small learning rates and/or small target_model_update param.
If this doesn't work for you check TRPO and/or PPO..


What are the mathematical prerequisites to be able to study artificial general intelligence? What are the mathematical prerequisites to be able to study artificial general intelligence (AGI) or strong AI?

I always recommend starting with game theory, combinatorial game theory, and algorithmic combinatorial game theory, (but I'm potentially biased;)

Combinatorics is a given--discrete mathematics is heavily utilized in computer science, and, with the advent of Combinatorial Game Theory (CGT), ability to determine if a given choice can be deemed optimal ("perfect play").  CGT arises out of traditional Game Theory, which we sometimes term "economic game theory" to make the distinction.  Out of Game Theory also arises subfields such as Evolutionary Game Theory, which is important in AI.

These fields relate to rationality, which is the basis for optimized decision making. Decision making algorithms seems to be the fundamental distinction of what constitutes an Artificial Intelligence.

From minimax to gametrees, it's probably a good idea to have a basic grounding in these fields, even if the problem you're AI is trying to solve isn't formally defined as a game.

All problems, from a fundamental standpoint, can be regarded either as puzzles--non-competitive context--or games--competitive context.  This distinction is based on whether there is a single agent (puzzles) or multiple agents (games.)


Is it possible to spot photoshoped or edited photos using AI? I have this question in my head: does the current level of AI development allow us to spot faked or photoshoped images? (i.e forged ID card or personal documents).

If it is possible, what is such a process to follow in order to build an AI that achieves this task?

It is possible! Here is an article by Adobe where they explain how they do it: https://theblog.adobe.com/spotting-image-manipulation-ai/

The algorithm for this would almost certainly be a Convolutional Neural Net trained on a dataset of real and manipulated images (labeled as such).


What's the difference between content-based attention and dot-product attention? I'm following this blog post which enumerates the various types of attention.
It mentions content-based attention where the alignment scoring function for the $j$th encoder hidden state with respect to the $i$th context vector is the cosine distance:
$$
e_{ij} = \frac{\mathbf{h}^{enc}_{j}\cdot\mathbf{h}^{dec}_{i}}{||\mathbf{h}^{enc}_{j}||\cdot||\mathbf{h}^{dec}_{i}||}
$$
It also mentions dot-product attention:
$$
e_{ij} = \mathbf{h}^{enc}_{j}\cdot\mathbf{h}^{dec}_{i}
$$
To me, it seems like these are only different by a factor. If we fix $i$ such that we are focusing on only one time step in the decoder, then that factor is only dependent on $j$. Specifically, it's $1/\mathbf{h}^{enc}_{j}$.
So we could state: "the only adjustment content-based attention makes to dot-product attention, is that it scales each alignment score inversely with the norm of the corresponding encoder hidden state before softmax is applied."
What's the motivation behind making such a minor adjustment? What are the consequences?

Follow up question:
What's more, is that in Attention is All you Need they introduce the scaled dot product where they divide by a constant factor (square root of size of encoder hidden vector) to avoid vanishing gradients in the softmax. Any reason they don't just use cosine distance?

The Attention is All you Need has this footnote at the passage motivating the introduction of the $1/\sqrt{d_k}$ factor:

To illustrate why the dot products get large, assume that the components of $q$ and $k$ are independent random variables with mean 0 and variance 1. Then their dot product, $q \cdot k = \sum^{d_k}_{i=1}q_ik_i$ has mean 0 and variance $d_k$.

I suspect that it hints on the cosine-vs-dot difference intuition. The cosine similarity ignores magnitudes of the input vectors - you can scale $h^{enc}$ and $h^{dec}$ by arbitrary factors and still get the same value of the cosine distance.
The footnote talks about vectors with normally distributed components, clearly implying that their magnitudes are important. This suggests that the dot product attention is preferable, since it takes into account magnitudes of input vectors.  And the magnitude might contain some useful information about the "absolute relevance" of the $Q$ and $K$ embeddings.
Edit after more digging: Note that transformer architecture has the Add & Norm blocks after each
attention and FF block. At first I thought that it settles your question: since
every input vector is normalized then cosine distance should be equal to the
dot product. That's incorrect though - the "Norm" here means Layer
Normalization - analogously to batch normalization it has trainable mean and
scale parameters, so my point above about the vector norms still holds.
Finally, since apparently we don't really know why the BatchNorm works
same thing holds for the LayerNorm. Interestingly, it seems like (1) BatchNorm
(2) LayerNorm and (3) your question about normalization in the attention
mechanism - all of it look like different ways at looking at the same, yet
undiscovered and clearly stated thing.


How do neural networks manage to do regression? I'm trying to learn about neural networks, and I'm interested in gaining a better conceptual understanding of how they work to solve certain problems. I'm having trouble in conceptually understanding how they succeed in doing regression (i.e. predicting continuous variables), however, and wondered if anybody has a good explanation. I know the mathematics of how NNs work, but a clearer conceptual understanding would be helpful.

To give an idea of what I mean by a "conceptual understanding", here's the one that I have for how multilayer NN's with a sigmoid activation function are able to be effective classifiers. Each NN takes the scalar product of its inputs and a set of weights. The weights define a plane in the input space, and the sign of the scalar product indicates which side of the plane the point defined by the inputs is on. The sigmoid activation function outputs 1 if the point is on one side and 0 or -1 (depending on which function is used) if the point is on the other. So the first hidden layer of neurons can be considered to identify which side of each of a group of planes the input point is on. The neurons can also act as AND and OR gates, so subsequent layers of neurons give an output indicating whether the point lies in a region bounded by several of the planes (e.g. a neuron activates only if the point is above one plane and below another, indicating it is in a region of the space associated with one class of points). So if the network learns an appropriate set of planes to bound regions containing different classes of inputs and the AND/OR relations that determine whether a point is in a given region, then it can classify the input points, and this can work for regions with arbitrarily shaped boundaries.

I've not found or been able to think of a similar way of explaining why an NN can perform well in general regression problems (if it's big enough). Does anyone here know a way to explain this?

I came across the article A visual proof that neural nets can compute any function that explains how a neural network can approximate any function, which pretty much answers my question.

I suppose connections between neurons do not generally arrange themselves in the way described, though, so it would still be interesting to get insight into whether there are patterns of how the inter-neuron connections arrange themselves in neural networks trained to solve actual problems.


Has the Lovelace Test 2.0 been successfully used in an academic setting? In October 2014, Dr. Mark Riedl published an approach to testing AI intelligence, called the "Lovelace Test 2.0", after being inspired by the original Lovelace Test (published in 2001). Mark believed that the original Lovelace Test would be impossible to pass, and therefore, suggested a weaker, and more practical version.

The Lovelace Test 2.0 makes the assumption that for an AI to be intelligent, it must exhibit creativity. From the paper itself:

  The Lovelace 2.0 Test is as follows: artificial agent $a $ is challenged as follows:

  $a$ must create an artifact $o$ of type $t$;
  $o$ must conform to a set of constraints $C$ where $c_i ∈ C$ is
  any criterion expressible in natural language;
  a human evaluator $h$, having chosen $t$ and $C$, is satisfied
  that $o$ is a valid instance of $t$ and meets $C$; and
  a human referee $r$ determines the combination of $t$ and $C$
  to not be unrealistic for an average human.

Since it is possible for a human evaluator to come up with some pretty easy constraints for an AI to beat, the human evaluator is then expected to keep coming up with more and more complex constraints for the AI until the AI fails. The point of the Lovelace Test 2.0 is to compare the creativity of different AIs, not to provide a definite dividing line between 'intelligence' and 'nonintelligence' like the Turing Test would.

However, I am curious about whether this test has actually been used in an academic setting, or it is only seen as a thought experiment at the moment. The Lovelace Test seems easy to apply in academic settings (you only need to develop some measurable constraints that you can use to test the artificial agent), but it also may be too subjective (humans can disagree on the merits of certain constraints, and whether a creative artifact produced by an AI actually meets the final result).

No.

TL;DR: The Lovelace Test 2.0 is very vague, making it ill-suited for  evaluation of intelligence. It is also generally ignored by researchers of Computational Creativity, who already have their own tests to evaluate creativity.

Longer Answer:
According to Google Scholar, there are 10 references to the "Lovelace Test 2.0" paper. All of those references exist merely to point out that the Lovelace Test 2.0 exists. In fact, at least two of articles I consulted (A novel approach for identifying a human-like self-conscious behavior and FraMoTEC: A Framework for Modular Task-Environment Construction for Evaluating Adaptive Control Systems) proposed their own tests instead.

One of the authors who wrote the FraMoTEC paper also wrote his thesis on FraMoTEC, and indirectly critiqued the Lovelace Test 2.0 and other similar such tests:

  The Piaget-MacGyver Room problem [Bringsjord and Licato, 2012], Lovelace Test 2.0 [Riedl, 2014] and Toy Box problem [Johnston, 2010] all come with the caveat of being defined very vaguely — these evaluation methods may be likely to come up with a reasonable evaluation for intelligence, but it is very difficult to compare two different agents (or controllers) that partake in the their own domain-specific evaluations, which is what frequently happens when agents are tailored to pass specific evaluations.

Another major issue with the Lovelace Test 2.0 is that there is a proliferation of other tests to "measure" the creativity of AI. Evaluating Evaluation: Assessing Progress in Computational Creativity Research, published by
Anna Jordanous in 2011 (3 years before the invention of the Lovelace Test 2.0) analyzed research papers about AI creativity and wrote:

  Of the 18 papers that applied creativity evaluation methodologies to evaluate their system’s creativity, no one methodology emerged as standard across the community. Colton’s creative tripod framework (Colton 2008) was used most often (6 uses), with 4 papers using Ritchie’s empirical criteria (Ritchie 2007).

That leaves 10 papers with miscellaneous creativity evaluation methods.

The goal of "Evaluating Evaluation" was to standardize the process of evaluating creativity, to avoid the possibility of the field stagnating due to the proliferation of so many creativity tests.  Anna Jordanous still remained interested in evaluating creativity tests, publishing articles such as "Stepping Back to Progress Forwards: Setting Standards for Meta-Evaluation of Computational Creativity" and Four PPPPerspectives on Computational Creativity.

"Evaluating Evaluation" does provide some commentary to explain the proliferation of systems to evaluate creativity:

  Evaluation standards are not easy to define. It is difficult to evaluate creativity and even more difficult to describe how we evaluate creativity, in human creativity as well as in computational creativity. In fact, even the very definition of creativity is problematic (Plucker, Beghetto, and Dow 2004). It is hard to identify what ’being creative’ entails, so there are no benchmarks or ground truths to measure against.

The fact that so many tests of creativity already exist (to the extent that Jordanous can make an academic career in studying them) means that it's very difficult for any new test (such as the Lovelace Test 2.0) to even be noticed (much less cited). Why would you want to use something like the Lovelace Test 2.0 when there's so many other tests you could use instead?


What constitutes a large space state (in Q-learning)? I know this might be specific to different problems, but does anyone know if there is any rule of thumb or references on what constitutes a large state space?
I know that, according to multiple papers, tabular Q-learning is not suitable for these problems with large state spaces, but I can't find any indication of what would constitute a large state space
I've been working on a problem where my state space is about 30x30 and updating my Q-learning with the tabular method runs and works well. Would 100x100 start to become too big or 400x400?

I know this might be specific to different problems but does anyone know if there is any rule of thumb or references on what constitutes a large state space?

Not really, it is all relative. There are two main ways in which the scale of a value table might be too much:

Memory required to represent the table. This is relatively simple to calculate for any size.

Time required to populate the table with accurate estimates. This depends on how you are collecting that data, and how much variance there is given the same state, action choices.

If you are using action values you need to allow for the fact that the table size is not just $|\mathcal{S}|$, but $|\mathcal{S} \times \mathcal{A}|$.
If you are running fast, local simulations of the environment, then table sizes of a million or even a hundred million are not unreasonable. Software language or library choice for the simulation and agent can have a significant impact at these larger scales.
If any of the state description is a continuous variable, then the table size would become infinite in theory, so no finite table that could actually be realised on a computer could fully capture it. You must use some form of approximation to get practical results. However, even then you can still use tabular approaches when approximation involves discretising the state variable directly - there are a few different methods used in machine learning to do that - e.g. tile coding. You can also use tile coding in large discrete spaces.

I've been working on a problem where my state space is about 30x30 and updating my Q-learning with the tabular method runs and works well. Would 100x100 start to become too big or 400x400?

It depends on the action space too, but 400x400 gives state space of 160,000. Assuming < 10 actions, then this is still well within the realm of tabular methods. You probably have less than 1,000,000 parameters to handle for a tabular method, which compares well to a moderately-sized neural network, with the advantage of better stability and possibility to create a fully optimal agent. If simulating the environemnt is fast, something at that scale may only take a few minutes to a few hours to fully optimise in tabular form.
As with most machine learning though, if you care about some optimisation, such as memory size of the agent, speed at which it solves the problem, or some other metric, then you will need to try experiments with different approaches. Then you will have some knowledge about best approach for your problem, given your definition of "best agent". The experience you gain with that might extend and apply to similar problems in future.


How do you generate the transition probabilities of a non-trivial MDP? I understand an MDP (Markov Decision Process) model is a tuple of $\{S, A, P, R \}$ where:

$S$ is a discrete set of states
$A$ is a discrete set of actions
$P$ is the transition matrix ie. $P(s' \mid s, a) \rightarrow [0,1]$
$R$ is the reward function id. $R(s, a, s') \rightarrow \mathbb{R}$

For a non-trivial MDP, say $1000$ states and $10$ actions, the transition matrix has theoretically $S \times A \times S = 10,000,000$ entries (though many entries will be $0$).

I understand that one way of generating the $P$ matrix is to estimate it via Monte Carlo sampling, by simulating the environment. However, with non-trivial state space and simulation costs, this could be prohibitively expensive.

In practice, when a non-trivial MDP is being formulated, what are the different ways an accurate $P$ matrix can be produced?

In non-trivial cases, the transition matrix is (generally) not maintained in the traditional tabular form. If the representation used factored notation (Factored MDP) then Dynamic Bayesian networks can be used.
Another approach would be to abstract state spaces so that you have reduced number of states in the representation of P. These can be roughly classified as model minimization techniques.
Hierarchical representations can be adopted if some tasks can be split. E.g., the classical "taxi problem" has an explicit Get-passenger and Put-passenger task, that form the hierarchy. Now each of the smaller tasks can possibly have a well formed transition function.
It may also be prudent to use model free learning methods in such cases.


Is there a neural network with a varying number of neurons? Is there some type of neural network that changes the number of neurons while training?

Using this idea, the network can increase or decrease the number of neurons when the complexity of the inputs increases or decreases.

Yes, NEAT (NeuroEvolution of Augmenting Topologies) increases the number of neurons during training. More specifically, NEAT uses evolution to introduce new neurons and connections during training, and - just as evolution - if the mutation performs poorly, gets eliminated after a few generations. This way overall performance increases over time while it keeps your network size (and computing power to run it) minimal. There's also a way to add convolution to this algorithm.

In the paper that introduced NEAT, the authors mention a completely different algorithm they tried, optimizing network hyper-parameters by choosing reasonable random values and training it multiple times. That could decrease that number as well.

Also, there is a trick to temporarily turn off specific parts of a network, which supposedly helps with overfitting.

A ReLU+backpropagation based network can "turn off" parts of the network during training, because constant 0's derivative is constant 0. In practice, you decrease the number of neurons (that is considered bad though, that's when leaky ReLU and PReLU is used instead).


Convolutional Sequence to Sequence Learning: Training vs Generation I am struggling to understand the use of the Convolutional Sequence to Sequence (Conv-Seq2Seq) model.  The image below is take directly from the paper and is the nearly canonical diagram of the parallel training procedure.  After puzzling over it for quite some time, it has come to seem straight forward to me:

An input sentence of N tokens can be encoded in one step because the input sentence exists prior to the start of training, and therefore the token-wise convolution can be trivially parallelized. (Compare to RNN encoders, which require N steps)
During training, an output sentence can similarly be parallelized in the decoder because during training, the entire output sentence is known.
Therefore, during training, the attention function can be fully parallelized in the two dimensional array of dot products shown below
Finally, during training, the attention is used to weight the input embeddings and encodings, combined with the output training encodings (as such) and the final output assembled.

This is clearly not the case after the network is trained and evaluation input sequences are translated without reference outputs.  I understand from various resources (including but not limited to Gehring's conference presentation) that post-training, output sequences are generated token by token in a fashion vaguely similar to earlier architectures, but I cannot find a clear description of that process.

(I speculate that this is because the parallel training routine was so revolutionary at the time, that the focus of the publications was rightly on the training routines.)

Can someone please help me understand the post-training generation algorithm, if possible in terms of the training diagram?

My current non-confident understanding is that the sentence below would be handled something like the following:

Prime the decoder with a default token string of <p> <p> <s>, this results (due to convolution) in a single decoder encoding (as such) input into the attention function, and would hopefully generate the single token <'Sie'> as output
Restart the decoder with the token string <p> <p> <s> <'Sie'> which would generate two inputs into the attention function and hopefully output <'Sie'> <'stimmen'>
Proceed with lengthening input sentences until the final token generated output ends with a </s> token signifying the end of the sentence.

If that is a correct understanding, can someone confirm it?  If close, can someone correct me?

Convolutional Sequence to Sequence Learning, Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin, 2017

I know it's too late to answer your query after 1.5 years
The inference algorithm is explained at https://charon.me/posts/pytorch/pytorch_seq2seq_5/#inference
Implementation of the algorithm is available at: https://github.com/pytorch/fairseq.
One could get a complete understanding from their source code.


What parameters to tweak to improve reliability (machine learning)? Lets say I have a limited amount of training data (1000 documents each having 10000 values) and after learning with those, the program is
 basically not allowed to fail.

From what I've read, even very easy tasks take a machine learning algorithm several thousand learning cycles until it is about 98% reliable.

Which parameters need to be tweaked so that even with many many values, the program can make reliable decisions?

If your question is about supervised learning,  than ten million training points are enough in most of the cases. Besides, I think your question is too broad. There are a lot of  parameters which can influence the accuracy. Like which algorithm you want to use. Which approximator and its topology, the hyperparameters etc.


Is the new Alpha Go implementation using Generative Adversarial Networks? I read through the publication Mastering the game of Go without Human Knowledge. It doesn't seem to use GANs, just a new form of search and reinforcement learning.

No, GANs are not used. It's reinforcement learning at what it does best. The tree search is an interesting addition and assists with navigating the sheer scale of the game.

Although the agent was playing itself to become better, there wasn't 2 separate networks (generator and discriminator). The agent learned through RL and didn't have the error calculations that generators and discriminatory have.


Why can't my implementation of the Actor-Critic algorithm achieve good results in the 2048 game? I implemented the Actor-Critic with n-step TD prediction to learn to play the 2048 game
For the environment, I don't use this 2048 implementation. I use a simple one without any graphical interface, just pure matrices. The input for the neural network is the log2 of the game board.
The structure of my neural network is:

Input layer
Hidden layer with 16 units
Softmax layer with 4 units (up, down, left, right) for the actor
Linear regression for the critic

The hidden layer is shared between the actor and critic, so the same hidden layer (point 2) is connected to both the softmax layer of the actor and the linear regression layer of the critic.
The reward in the original game is the value of the merged cells. For example, if two 4s merged, then the reward is 8. My reward function is almost the same, except I take the log2 of it.
I tried these parameters and I also tweaked the learning rate, the $\gamma$, but I couldn't achieve any good result.
Could you recommend what should I change?

Interesting project. First thing I'd do is normalize your state by the maximum cell value. This way you can represent multiple situations at once (eg A grid of all 4s and 8s would look the same as a grid of all 16s and 32s).
Also making the reward = max_cell/2048 might do better as ActorCritic methods seem to do better with rewards within 0-1.

Another reward setup is giving +1 per timestep. It's simple but it also means that maximizing the time to stay alive is the best, which is what I end up doing most of the time when I play anyway.

Good luck!


If a neural network is a universal function approximator, can it have any prior beliefs? Let us confine ourselves to the case where we have a $n$ dimensional input and a $+1$ or $-1$ output. It can be shown that:

For every $n$, there exists a dense NN of depth 2, such that it contains all functions from ${±1}^n$ to ${±1}$. (given sign activation functions and some other very simple assumptions).

Check section 20.3 for the proof.
So, if a neural net can approximate any function, then it has a $\mathcal V \mathcal C $ dimension of infinite (considering the $n$ dimensional set of points as our universe).
Thus, it can realize all types of functions (or its hypothesis set contains all set of functions), and hence cannot have prior knowledge (prior knowledge in the sense used in the No Free Lunch theorem).
Are my deductions correct? Or did I make wrong assumptions? Are there actually any prior beliefs in a neural network that I am missing?
A detailed explanation would be nice.

I think your deduction is mostly correct.

Neural networks of depth are universal function approximators. This means that in principal, for any function of the form you describe, there's a NN that approximates it.

However, a particular NN architecture of fixed width and depth, with fixed connections is not a universal approximator for all functions. Only an infinitely wide NN is, and that's a theoretical construct, not something you can make in practice.

Typically, a practitioner using a NN injects their prior beliefs about the problem  by selecting an architecture. For example, it is common to use the ImageNet or ResNet architectures for image processing tasks. Those architectures are less effective on other types of tasks. For instance, it should be clear that they are ineffective on say, a time-series analysis task.


What is the difference between meta-learning and zero-shot learning? What is the difference between meta-learning and zero-shot learning? Are they synonymous?

I have seen articles where they seem to imply that they are at least very similar concepts.

First see the definition of meta-learning:

  Meta-learning is a subfield of machine learning where automatic learning algorithms are applied to metadata about machine learning experiments. As of 2017 the term had not found a standard interpretation, however the main goal is to use such metadata to understand how automatic learning can become flexible in solving learning problems, hence to improve the performance of existing learning algorithms or to learn (induce) the learning algorithm itself, hence the alternative term learning to learn.

and zero-shot learning:

  Zero-shot learning is being able to solve a task despite not having received any training examples of that task. For a concrete example, imagine recognizing a category of object in photos without ever having seen a photo of that kind of object before. If you've read a very detailed description of a cat, you might be able to tell what a cat is in a photograph the first time you see it.

As you can see, these are different but meta-learning can be used in zero-shot learning to work better. For example see this article, as an instance.


Why do neural nets and machine learning tend to work well with MCTS, but not with regular Minimax game-playing AI? I've often heard MCTS grouped together with neural nets and machine learning. From what I gather, MCTS uses a refined intuition (from maching learning) to evaluate positions. This allows it to better guess which moves are worth playing out more.

But I've almost never heard of using machine learning for Minimax+alpha-beta engines. Couldn't machine learning be used for the Engine to better guess which move is best, and then look at that move's subtree first? A major optimization of the minimax algorithm is move-ordering, and this seems like a good way to accomplish that.

It just does.
Take a look at this post explaining how MCTS works.

  In both Alpha Go Lee and Alpha Zero the tree traversal follows the nodes that maximize the following UCT variant:

  \begin{equation}
UCT(v_i,v) = \frac{Q(v_i)}{N(v_i)} + cP(v_i, v)\sqrt{\frac{N(v)}{1+N(v_i)}}
\end{equation}

  where P(vi,v)
  is prior probability of the move (transition from v to vi ), its value comes from the output of deep neural network called Policy Network . Policy Network is a function that consumes game state and produce probability distribution over possible moves.

As you can see the Policy Network(it is actually just one neural network for both the value and the policy) is used to guide the search tree. Not all possible moves are explored. Also during the learning phase the policy network uses the "visit count" of the MCTS Nodes to learn. Moves that were explored more are the better ones. The state-of-the-art chess engine Stockfish evaluates about 1000 times more positions per second than Alpha Zero. It relies on exploring "most of" the possible positions. Calculating the score using heuristics is much faster than using Alpha Zero's 19 layer residual network. If Google were to use minimax then they wouldn't be able to look very far ahead. Alpha Zero explores about 70K moves per second which would be only a couple ply deep. MCTS allows it to explore only the more promising moves and simulate those positions.


Knowledge required for understanding AlphaZero paper My goal is to understand AlphaZero paper published by deepmind. I'm beginning my journey trying to get the basic intuition of reinforcement learning from the book by Barto and Sutton.

As per my background, I'm familiar with MDPs, value iteration and policy iteration.

I wanted to ask till what chapter of Barto and Sutton's book is one required to read in order to fully comprehend AlphaZero paper. Monte-Carlo Tree Search is discussed in Chapter-8 of the book. Will it be enough till that? Or would I be needing more resources apart from this book? Thanks.

The more you read, the more deeply you can understand any paper, but given your stated background, reading the Monte-Carlo Tree Search chapter of Barto & Sutton, plus Gerald Tesauro's TD-Gammon paper (which is pretty accessible, and which is the basis for the other technique used in AlphaZero) should be enough to get a pretty good idea of what they did.


How do you manage negative rewards in policy gradients? This old question has no definitive answer yet, that's why I am asking it here again. I also asked this same question here.
If I'm doing policy gradient in Keras, using a loss of the form:
rewards*cross_entropy(action_pdf, selected_action_one_hot)

How do I manage negative rewards?
I've had success with this form in cases where the reward is always positive, but it does not train with negative rewards. The failure mode is for it to drive itself to very confident predictions all the time, which results in very large negative losses due to induced deviation for exploration. I can get it to train by clipping rewards at zero, but this throws a lot of valuable information on the table (only carrots, no sticks).

You don't need to manage negative rewards separately, if you implemented the algorithm correctly it will work regardless if the rewards are negative or not. You seem to be using rewards for the loss but you should be using the return which is the sum of the rewards for some state action pair from that point until the end of trajectory.

You also seem to be missing $-$ sign from the loss. The objective function for the vanilla policy gradient algorithm (REINFORCE) which we want to maximize is
\begin{equation}
J = \sum_a \pi(a|s) q_{\pi}(s, a)
\end{equation}
It can be shown that the gradient sample for this policy gradient method is
\begin{equation}
\nabla J = G_t \nabla \log (\pi(A_t|S_t))
\end{equation}
so in TensorFlow you should define your loss as
\begin{equation}
J = - G_t \pi(A_t|S_t)
\end{equation}
We need the $-$ because in TensorFlow you use minimizers, but we want to maximize this function so minimizing this loss is same as maximizing the objective function. In conclusion, the code similar to what you wrote, should be
-return * cross_entropy(action_pdf, selected_action_one_hot)

EDIT

As pointed out in the comment we don't actually need $-$ because it is already included in cross_entropy function.


How to handle class imbalancing when the actual data are that way My supervised learning training data are obtained from actual data; and in real cases, there's one class which happens less often than other classes, just around 5% of all cases.
To be precise, the first 2 classes are in 95% of training data and the last one is in 5%. Training while keeping the data ratio intact will make accuracy reach 50% at the right first step and reaches 90%+ immediately that doesn't make sense.
Should I exclude some data of class 1 and 2, to make the numbers of samples of 3 classes equal? But it's not real world ratio.

You can use stratified cross validation combined with an imbalanced learning technique applied to the training data. Stratification ensures that when you split your data into train and test, the ratio of frequencies between the classes will stay the same and therefore the test data will always be "realistic".
However, when training a model (using only the training data, of course), the imbalance may have a negative impact. Therefore, have a look at some of the imbalanced learning techniques that are out there to remedy this situation. For example you could try these:

random undersampling: discard random examples from the majority classes until the ratios of class frequencies are close to 1
random oversampling: make random duplicates of minority class examples until the ratios of class frequencies are close to 1
SMOTE: like random oversampling, except that synthetic examples are created instead of random duplicates
balanced bagging: performs random undersampling, but does so multiple times to create an ensemble of models trained on balanced subsets of the training data

etc
You should also take care about the metrics you use to assess predictive performance on the test data. Accuracy could be misleading here, so you may instead find metrics like sensitivity and specificity (calculated for each class individually) more informative.


How come that the addition of features can decrease the performance of a neural network? I have a Remaining Useful Life (RUL) prediction problem that I want to solve. When I added two or more features as inputs to my ANN, the accuracy of my ANN has been decreased. More precisely, I've added features like RMS or KURTOSIS (or both). I was expecting the system to improve, but it is getting worse.

Why might this be happening? What are the potential reasons for this degradation in performance?

I know that when we added more nodes in layers (like hidden layers), overfitting can happen. Would that be related to my problem: using more than two features?

Additional features can also cause overfitting if they have low or misleading information.

Consider the following problem:

$X = [1, 3, 3, 4, 5]$, $Y = [1, 3, 4, 4, 5]$.

Suppose that the real dataset was generated from the relationship:

$Y = X$, with a probability of 0.2 of adding or subtracting 1.

A reasonable model estimate is $Y = X$. Note that no model can fit this data perfectly, because the two 3 inputs map to different outputs.

Now, suppose we add a new feature: a random number between $0$ and $10$:
$W = [1 ,5, 2, 6, 3]$

It may not be obvious, but a sufficiently deep and broad neural network can learn a new function:

$g(W) = 1$ if $W = 2,4,7,8,9,$ or $0$.

$g(W) = 0$ otherwise.

and define a new prediction: $Y = X - g(W)$.

This happens to produce a perfect fit on the training data. However, it will perform extremely poorly on new data (like a test set), because it has learned a meaningless pattern out of random noise. Coincidentally, it will be wrong on about 50% of samples, while our first model will be wrong on only 20% of samples.


In MCTS, what to do if I do not want to simulate till the end of the game? I'm trying to implement MCTS with UCT for a board game and I'm kinda stuck. The state space is quite large (3e15), and I'd like to compute a good move in less than 2 seconds. I already have MCTS implemented in Java from here, and I noticed that it takes a long time to actually reach a terminal node in the simulation phase.
So, would it be possible to simulate games up until a specific depth?
Instead of returning the winner of the game after running until the max depth, I could return an evaluation of the board (the board game is simple enough to write an evaluation function), which then back propagates.
The issue I'm having is in handling the backpropagation. I'm not quite sure what to do here. Any help/resources/guidance is appreciated!

Famous example is AlphaZero. It doesn't do unrolls, but consults the value network for leaf node evaluation. The paper has the details on how the update is performed afterwards:

The leaf $s'$ position is expanded and evaluated only once by the network to gene-rate both prior probabilities and evaluation, $(P(s′ , \cdot),V(s ′ )) = f_\theta(s′ )$. Each edge $(s, a)$ traversed in the simulation is updated to increment its visit count $N(s, a)$, and to update its action value to the mean evaluation over these simulations, $Q(s,a) = \frac{1}{N(s,a)}\sum_{s,a\to s'}V(s')$  , where $s, a\to s′$ indicates that a simulation eventually reached s′ after taking move a from position s.


Are connections genes in a genome ever deleted or just disabled? When a new node is added, the previous connection is disabled and not removed.

Is there any situation in which a connection gene is removed? For example, in the above diagram connection gene with innovation number 2 is not present. It could be because some other genome used that innovation number for a different connection that isn't present in this genome. But are there cases where a connection gene has to be removed?

The original paper never goes further than disabling the link, but I have seen implementations on github that did have a probability of deleting the link.

python-neat is such an eaxample.


Will a neural network always predict the correct label if it sees the exact same input during training and testing? If I'm performing a text classification task using a model built in Keras, and, for example, I am attempting to predict the appropriate tag for a given Stack Overflow question:

How do I subtract 1 from an integer?

And the ground-truth tag for this question is:

objective-c

But my model is predicting:

c#

If I were to retrain my model, but this time add the above question and tag in both the training and testing data, would the model be guaranteed to predict the correct tag for this question in the test data?
I suppose the tl;dr is: Are neural networks deterministic if they encounter identical data during training and testing?
I'm aware it's not a good idea to use the same data in both training and testing, but I'm interested from a hypothetical perspective, and for gaining more insight into how neural networks actually learn. My intuition for this question is "no", but I'd really be interested in being pointed to some relevant literature that expands/explains that intuition.

No, Neural Networks do not have such a guarantee. In fact, I don't believe any kind of classifier in the entire field of Machine Learning has such a guarantee, though some may be slipping my mind...

For an easy counterexample, consider what happens if you have two instances with precisely identical inputs, but different output labels. If your classifier is deterministic (in the sense that there is no stochasticity in the procedure going from input to output after training), which a Neural Network is (unless, for example, you mess up a Dropout implementation and accidentally also apply dropout after training), it cannot possibly generate the correct output for both of those instances, even if they were presented as examples thousands of times during training.

Of course the above is an extreme example, but similar intuition applies to more realistic cases. There can be cases where getting the correct prediction on one instance would reduce the quality of predictions on many other instances if they have somewhat similar inputs. Normally, the training procedure would then prefer getting better predictions on the larger number of instances, and settle for failure on another instance.


Tensorflow-gpu cannot use Nvidia GPU with CUDA I'm working on a Python Keras/Tensorflow image recognition script (on Ubuntu 18.04) which works ok, but it will only train on CPU (which is slow) and I want to be using my GPU (i have a Nvidia Geforce GTX1080).

I have installed:

tensorflow-gpu 2.0.0-beta1 (via pip; note i also tried 1.4 but same issue)
Cuda10.2 (via deb files from Nvidia website)
cuDNN 7.6.5 (via Nvidia dev website)
CUPTI (via apt)
Nvidia drivers 440.33.01

$ nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    On   | 00000000:B3:00.0  On |                  N/A |
| 38%   42C    P0    38W / 180W |    213MiB /  8118MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2109      G   /usr/lib/xorg/Xorg                           210MiB |
+-----------------------------------------------------------------------------+

However when I run my script, tensorflow is unable to use the GPU, and gives these errors (and falls back to running on CPU):

I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:b3:00.0
I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
W tensorflow/core/common_runtime/gpu/gpu_device.cc:1663] Cannot dlopen some GPU libraries. Skipping registering GPU devices...

I seem to be missing these libraries: libcudart.so.10.0, libcublas.so.10.0, libcufft.so.10.0, libcurand.so.10.0, libcusolver.so.10.0, libcusparse.so.10.0, but I'm not sure how to get them installed, apt does not appear to have version 10 available for these packages.

Seems like you have a CUDA version conflict. Remove the existing CUDA 10.2 and install CUDA 10.0 (going by your missing libraries, it requires a v10.0).

You can find the archived releases here:
https://developer.nvidia.com/cuda-toolkit-archive


What AI technique should I use to assign a person to a task? I'm trying to learn AI and thinking to apply it to our system. We have an application for the translation industry. What we are doing now is the coordinator $C$ assigns a file to a translator $T$. The coordinator usually considers these criteria (but not limited to):

the deadline of the file and availability of a translator
the language pair that the translator can translate
is the translator already reached his target? (maybe we can give the file to other translators to reach their target)
the difficulty level of the file for the translator (basic translation, medical field, IT field)
accuracy of translator
speed of translator

Given the following, is it possible to make a recommendation to the coordinator, to whom she can assign a particular file?
What are the methods/topics that I need to research?
(I'm considering javascript as the primary tool, and maybe python if javascript will be more of a hindrance in implementation.)
In addition to suggesting a translator, we are also looking into suggesting the "deadline of the translator". Basically, we have "deadline of the customer" and "deadline of the translator"
The reason for this is that, if the translators are occupied throughout the day, it makes sense to suggest it to a busy translator but allow him to finish it until next day.

What you have could be well described as a Task Allocation problem, which is studied as part of the planning subfield of AI. Chapters 10 & 11 of Russell & Norvig provide a good overview of this area, although I think they don't talk too much about Task Allocation in particular.
There are two basic approaches to this problem: centralized approaches, and decentralized approaches.
In centralized approaches, the properties of each task (or sub-task) and the skills of each processing entity are recorded in a central database. The task is phrased as an optimization problem. For example, given the skills of the processors and the tasks' types, find the schedule that minimizes average processing time (or cost, or usage of rare-resource types, or whatever you're interested in). Common approaches include phrasing the optimization task as a linear-programming problem; phrasing the problem as a graph and using something like the graphplan algorithm; or phrasing the problem as a constraint satisfaction problem and using some kind of heuristic-guided local search.
There are all kinds of other more modern techniques too. I'm not aware of a survey paper for translation tasks in particular, but there are lots of examples in robotics and distributed computing.
Although good AI techniques exist for scheduling the task, they are predicated on being able to quantify the tasks' properties and the abilities of the agents, and on the translators accepting the decisions of the system. If you want an interactive system, you may need to look at techniques from Natural Language Processing. The work on Mixed Initiative Scheduling Systems might also be relevant if you have to go that route.


Is there a way of converting a neural network to another one that represents the same function? I have read the paper Neural Turing Machines and the paper On the Computational Power of Neural Nets about the computational power of neural networks. However, it isn't still clear to me one thing.

Is there a way of converting a neural network to another one that represents the same function? For example, given a trained recurrent neural network that represents a certain function, is it possible to convert this RNN to e.g. an MLP that represents the same function?

If an answer to the above question exists, can we show the equivalence between neural networks, e.g., all problems solved by a multilayer perceptron can be solved by a recurrent neural network, but the opposite is not true, i.e., MLP is subset of RNN (I do not know if this is true, it is just an example). So, if we obtain this relationship between all neural networks, we can get a neural network $X$ that is more powerful than others, so, we can throw away all other neural networks because $X$ can solve any problem that other NN can. Is this reasoning correct?

To answer this, it's helpful to consider the notion of a neural network architecture – in this context, we can think of the architecture as being the network depth (i.e. number of layers), width (i.e. number of nodes in a layer), and some other structural aspects, such as recurrent layers, convolution layers, pool layers, etc.

Theory

In terms of the theory, there are a few ideas that provide insight into your question. First is the commonly cited universal approximation theorem, which states that a single hidden layer, feed forward neural network with a sigmoid activation and finite width (i.e. the number of nodes in single the hidden layer) can approximate any n-dimensional real-valued function. Superficially, this tells us that we can do a whole lot with one of the most simple architectures and could (in theory) approximate more complex architectures such as LSTMs, CNNs, etc.

Yet, while this seems to indicate that reductions are always possible, research on the asymmetry in representational power between depth and width hints that things may not be so simple (for example, see this paper and this paper). Specifically, network depth provides much greater representational power relative to width. Intuitively, this makes sense – depth allows us to build hierarchies and stack multiple non-linearities. Indeed, this is where we begin to see the outlines of what the answer looks like in practice – depth provides a clear, easy way to bake structure into our network.

So, the upshot from the theoretical perspective is that the universal approximation theorem answers in the affirmative; yes, you could reduce a complex network architecture to a single hidden layer network, but the basis for doing this is a bit naïve – it doesn't consider the efficiency of the network relative to the number of parameters.The two papers linked above imply this.

Practice

In practice, there are issues with performing such a reduction. To date, the most successful models have deep architectures. More importantly though, many of these models are specialized – we have CNNs for computer vision, RNNs for sequence-based data, etc. These architectures were conceived as a way to integrate knowledge about the structure of the problem into the structure of the network to improve learning of the desired representation, and this too requires depth.

In fact, last year a paper took this idea to the extreme. Weight agnostic neural networks are specifically constructed such that their architecture is in essence the representation with a single parameter (weight) that can be varied to generate multiple distinct models with different performance characteristics. The WANNs that were developed in this research weren't especially deep, but they also weren't single layered.

The upshot here is that, for complex problems, depth is necessary for efficient learning and gradient computation and that it can be used to encode problem structure into a network. While it may be possible to reduce a learned representation to a single hidden layer network, it's not inconceivable that you may end up with more parameters condensed into a single hidden layer with very high dimensional gradients. In terms of implementation, this isn't desirable.


Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0.
However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) of more than 30. My question is, is it not possible to get successful completion of Cartpole without using sophisticated algorithms such as Deep learning etc.?
I am glad to share my code, but I am sure no one would have time to check it.

PS. I know there are many implementations out there, but I have learned from them but I want to implement my own code for learning purpose and do not just want to copy-paste.
PSS (Edit): I have added the code in the answer to this question for reference.

The code to my question is as below, for reference:
import gym
import numpy as np
import matplotlib.pyplot as plt

# Discretize the contiuous space
DISCRETE_POINTS = 50
X_position = np.linspace(-2.4, 2.4, DISCRETE_POINTS)
Velocity = np.linspace(-5, 5, DISCRETE_POINTS)
Angle = np.linspace(-0.7295476, 0.7295476, DISCRETE_POINTS)
Angular_vel = np.linspace(-5,5,DISCRETE_POINTS)

# Fit the instantnous state into any discrete box
def get_state(_state):

    X, X_bar, Angle_, Angle_bar = _state

    X = int(np.digitize(X, X_position))
    X_bar = int(np.digitize(X_bar, Velocity))
    Angle_ = int(np.digitize(Angle_, Angle))
    Angle_bar = int(np.digitize(Angle_bar, Angular_vel))

    return (X, X_bar, Angle_, Angle_bar)

def epsilon_greedy_action(s, epsilon):

    '''   Input argument: state 's' tuple in the form (4,0,1,0)    '''
    # if np.random.uniform() < epsilon:
    if np.random.random() > epsilon:
        a = env.action_space.sample()
    else:
        _,a = find_maxQ_value(s)

    return a

def find_maxQ_value(state):
    '''
    Input argument:
    state: should be a tuple of form (0,0,0,0) or (1,0,0,1) etc.

    Output argument:
    best_value: best q-value of the current state-action pair
    choosen_action: best action corresponding to current state. It depends on the best q-value
    '''

    for act_ in range(env.action_space.n):
        A = [Q[state,0], Q[state, 1]]
        best_value = np.max(A)
        choosen_action = np.argmax(A)

    return best_value, choosen_action

def plotRunningAverage(totalrewards, N, n_avg):

    running_avg = np.empty(N)
    for t in range(N):
        running_avg[t] = np.mean(totalrewards[max(0, t-N):(t+1)])
    return running_avg

if __name__ == '__main__':

    env = gym.make('CartPole-v0');

    EPISODES = 1000;
    no_actions = env.action_space.n

    # Hyper parameters
    alpha = 0.001 # Learning rate
    gamma = 0.99 #Discount Factor
    epsilon = 1 # For Epsilon-Greedy algorithm
    epsilon_decay_factor = 0.99;
    min_epsilon = 0.1;

    states = []
    for i in range(len(X_position)):
        for j in range(len(Velocity)):
            for k in range(len(Angle)):
                for l in range(len(Angular_vel)):
                    states.append((i,j,k,l))

    #Initialize Q-table :
    # 1. We make the Q-table in form of a dictionary
    # 2. We initialize Q-table values as zero in this
    Q = {}
    for s in states:
        for n_a in range(no_actions):
            Q[s, n_a] = 0

    Running_reward = [];
    l_action_cnt = 0;
    r_action_cnt = 0;
    wrong_action =0;

    #Q-Learning agent episodes
    for e in range(EPISODES):

        cn_state = env.reset()
        ds_state = get_state(cn_state)

        done = False
        ep_reward = 0
        ep_len = 0

        while not done:

            action = epsilon_greedy_action(ds_state, epsilon)
            if action == 0:
                l_action_cnt+=1
            elif action == 1:
                r_action_cnt+=1;
            else:
                wrong_action+=1;

            cn_next_state, reward , done , ep_len = env.step(action)
            ep_reward += reward

            ds_next_state = get_state(cn_next_state)

            # Update the Q-table based on the action
            Val_Q_bar, _ = find_maxQ_value(ds_next_state);

            Q[ds_state, action] = (1-alpha)*Q[ds_state, action] + alpha*(reward + gamma*Val_Q_bar)

            ds_state = ds_next_state;

        Running_reward.append(ep_reward)

        if e%100 == 0:
            print('Episode : {}, Episode reward: {}, Epsilon: {}'.format(e, ep_reward, epsilon))

        if epsilon >= min_epsilon:
            epsilon*=epsilon_decay_factor;

plt.plot(Running_reward);
plt.xlabel('episodes')
plt.ylabel('episodic reward')
plt.grid('ON')
running_avg = plotRunningAverage(Running_reward, EPISODES, 50)
plt.plot(running_avg);
plt.legend(['Episodic rewards', '50-Episode moving-average reward'])

print('The ratio of left to right action is : {}'.format(l_action_cnt/r_action_cnt))


How powerful are the computers that power the most advanced artificial intelligence nowdays How powerful is the machine that beat the poker player champion recently?

From the Deep Stack paper:

This seems to be for training:

  For the turn network, ten million poker turn situations (from after the turn card is dealt) were generated and solved with 6,144 CPU cores of the Calcul Quebec MP2 research cluster, using over 175 core years of computation time. For the flop network, one million poker flop situations (from after the flop cards are dealt) were generated and solved. These situations were solved using DeepStack’s depth limited solver with the turn network used for the counterfactual values at public states immediately after the turn card. We used a cluster of 20 GPUS and one-half of a GPU year of computation time. For the auxiliary network, ten million situations were generated and the target values were obtained by enumerating all 22,100 possible flops and averaging the counterfactual values from the flop network’s output.

And this for actual play:

  The re-solving computation and neural network evaluations are both implemented in Torch7 (53) and run on a single NVIDIA GeForce GTX 1080 graphics card.

For comparison: The distributed version of AlphaGo took 1.920 CPUs and 280 GPUs to run.


Why can't we apply value iteration when we do not know the reward and transition functions, and how does Q-learning solve this issue? I don't understand why we can't apply value iteration when don't know the reward and transition probabilities. In this lecture, the lecturer says it has to do with not being able to take max with samples, but what does this mean?
Why does Q-learning not need to know the reward and transition functions? In Q-learning, we also have a max, so I am not understanding.

For normal value iteration, you need to have the model, i.e. the transition probability, denoted by $P(s' \mid s,a)$.
With Q-learning, you use the current reward and the already stored Q value:

The relation between the value function $V(s)$ and the $Q$ function $Q(s, a)$ is that the $V(s)$ function is simply the value of the action $a$, such that $Q(s, a)$ is the highest, that is, $V(s) = \max_a Q(s, a)$.


What is the difference between artificial intelligence and swarm intelligence? Artificial intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions. The term may also be applied to any machine that exhibits traits associated with a human mind such as learning and problem-solving.
According to the Wikipedia article on swarm intelligence

Swarm intelligence (SI) is the collective behavior of decentralized, self-organized systems, natural or artificial. The concept is employed in work on artificial intelligence.
The application of swarm principles to robots is called swarm robotics, while 'swarm intelligence' refers to the more general set of algorithms.
SI systems consist typically of a population of simple agents or boids interacting locally with one another and with their environment. The inspiration often comes from nature, especially biological systems.

These two terms seem to be related, especially in their application in computer science and software engineering. Is one a subset of another? Is one tool (SI) is used to build a system for the other(AI)? What are their differences and why are they significant?

Swarm intelligence (SI) is a sub-field of or an approach to artificial intelligence (AI), where you have multiple individuals (for example, artificial ants), which collectively can produce what we (or most of us) would intuitively call intelligent behaviour.
SI is sometimes categorized as a sub-field of evolutionary computation (which also includes evolutionary algorithms, such as genetic algorithms, genetic programming, evolution strategies, and so on), which is often considered a sub-field of AI or techniques to produce artificial intelligence, because all these techniques are often based on the use of multiple individuals/solutions (that either compete or collaborate with each other).
One of the most commonly used SI techniques is ant colony optimization algorithms (proposed by M. Dorigo and further developed by other people like Luca M. Gambardella), which have been successfully applied to solve the non-decision version of the NP-complete problem (in simple words, it's a combinatorial problem that may require exponential time to be solved in the usual case) known as the travelling salesman problem. There are other SI techniques, which are somehow similar to ACO algorithms, such as particle swarm optimization or the artificial bee colony algorithm.
Occasionally, SI may also be categorised as a sub-field of computational intelligence, which often refers to specific techniques to create artificially intelligent systems (i.e. programs that exhibit what we would call intelligence) that are more based on or inspired by the biology, such as neural networks, genetic algorithms, or, in fact, SI algorithms, such as ACO algorithms. However, CI can also be considered a sub-field of AI, given that it studies techniques to produce artificial intelligence, so, in the end, as I said above, SI is an approach to AI, which includes other approaches, such as evolutionary algorithms, rule-based systems, deep learning or other machine learning techniques.


Facial recognition and classifying unknowns with neural networks As far as I understand, neural networks aren't good at classifying 'unknowns', i.e. objects that do not belong to a learned class. But how do face detection/recognition approaches usually determine that no face is detected/recognised in a region? Is the predicted probability somehow thresholded?

I'm asking because my application will involve identifying unknown objects. In fact, most of the input objects are unknown and only a fraction is known.

Summary

It is true that neural networks are inherently not good at classifying 'unknowns' because they tend to overfit to the data that they have been trained on, if the underlying structure of the neural network is complex enough. However, there are multiple ways to go about reducing the affects of overfitting. For example, one technique that is used for this is called dropout. Another example can be batch normalization. Despite these techniques, the best way to reduce the affects of overfitting is to use more data.

For the facial recognition example that you have given above, it is common that the models that have been trained have 'seen' a huge amount of data. This means that there are very few 'unknowns' and even if there are, the neural network has learned how to tell if there are facial features present or not. This is because certain structures of neural networks are really good at telling if there is a pattern of features present in the input data. This helps the neural networks to learn if the image that is being input has certain features/patterns in it or not. If the these features are found then the input data is classified as face otherwise it is not.

What can you do in your case?

Let us assume that you are going to train your neural network to recognize if an input image is a cat or not. You will use a Convolutional Neural Network (CNN) and train it to recognize if the input is cat or not. The not part means that you have to include a lot of examples in your training data that are not cat. In the perfect case you will be able to show it everything that is not a cat and classify it as such. Also you show it multiple images of what a cat is. CNNs are really great for this application. You might want to research regarding this and see what kind of CNN best suits your application. If you don't have gazillion samples of what a cat is not then you can use regularization techniques like dropout and batch normalization.

PS: For more details please mention what strategies you have used up till now. Also it would be better if you can share what your desired task is.


How to evaluate the performance of an autoencoder trained on image data? I am training an autoencoder on (general) image data.

I use binary crossentropy loss function, but it is not very informative when I want to evaluate the performance of my autoencoder.

An obvious performance metric would be pixel-wise MSE, but it has its own downsides, shown on some toy examples in an image from paper from Pihlgren et al.

In the same paper, the authors suggest using perceptual loss, but it seems complicated and not well-studied.

I found some other instances of this question, but there doesn't seem to be a concensus.

I understand that it depends on the application, but I want to know if there are some general guidelines as to which performance metric to use when training autoencoders on image data.

I will answer my own question to try and provide some insights.
My research supervisor suggested that I should use the SSIM metric or some other well-known image processing metric (see the book "Modern Image Quality Assessment" by Wang and Bovik) for assessing the visual similarity of an images.
Another way I evaluate the performance of an autoencoder is by simply visually comparing the input and output images taken from the test set. This is by no means very scientific, but it gives a good idea whether an autoencoder is able to reconstruct the input images. One thing I would add here is that even if an autoencoder can reconstruct images perfectly, it doesn't mean that the encoding it learned is useful. For example, when I wanted similar images to be mapped to similar encodings, the autoencoder that was able to do that better was outputting more blurred reconstructed images in comparison to the autoencoder that wasn't achieving this similarity preservation (but was outputting better reconstructions).


Is the playout started from a leaf or child of leaf in Monte Carlo Tree Search? On Wikipedia, the MCTS algorithm is described

Selection: start from root $R$ and select successive child nodes until a leaf node $L$ is reached. A leaf is any node from which no simulation (playout) has yet been initiated.
Expansion: create one (or more) child nodes and choose node $C$ from one of them. Child nodes are any valid moves from the game position defined by $L$.
Simulation: complete one random playout from node $C$.

Why is the playout started from a child of the first leaf, not the leaf itself? And aren't leaves then permanently stuck as leaves, since playouts always start from their children, not them? Or does the leaf get attributed as having had a "playout initialised" from it, even though it started at its child?

NOTE: their description of the selection phase actually does not match the "standard implementation". They state to traverse the tree according to the selection strategy until a leaf node is reached. I disagree with this. The standard implementation is to traverse the tree until a node is reached in which there still are legal actions for which no child node has been created (a node that is not yet "fully expanded").

These two are only equivalent in the case where the expansion strategy is to immediately fully expand, to immediately create new nodes for all legal actions in $L$. With such an expansion strategy, $L$ would immediately turn from a leaf node into a fully expanded node. But that is not the "standard implementation". The most common expansion strategy is to only add a single new node to the tree per iteration.

Let $L$ denote a node that is reached at the end of the selection phase. The standard implementation is indeed to first expand in some way, which results in a new child $C$ of $L$, and then start the playout from $L$ onwards.

  And aren't leaves then permanently stuck as leaves, since playouts always start from their children, not them?

No, as soon as you add the new child $C$ to $L$, $L$ is by definition no longer a leaf node; a leaf node is defined as a node with no children. The next time that MCTS decides to traverse the same part of the tree, $L$ will be just like any other internal node, $C$ will be a leaf node, and we may add yet another node below that in a future expansion phase. Or, of course, it is also possible that in a future iteration, our selection phase still ends in $L$ despite it not (anymore) being a leaf, if it still has unexpanded nodes other than $C$ (to-be-siblings of $C$).

  Why is the playout started from a child of the first leaf, not the leaf itself?

This is because we typically require a slightly different mechanism for choosing a node to expand than the playout mechanism. The most common (or, at least, most simple and straightforward) playout mechanism is to select actions uniformly at random from all legal actions. The most common (or, at least, most simple and straightforward) mechanism for selecting a new node $C$ to add to $L$ is to select uniformly at random only from those legal actions for which children have not already been added to $L$. Due to these mechanisms being different, we cannot initiliase the playout at $L$ and declare the first action of the playout to be the one that generates the new node $C$; we have to use a (slightly) different implementation.


Reinforcement learning with industrial continuous process I am new to RL and wish to realize a RL control for an industrial process. The goal is to control the temperature and humidity in a vegetal food production chamber.

States: External temperature and humidity, internal temperature and humidity, percentage of the proportional valves controlling heater, cooler and steam for humidity. The goal is to keep temperature and humidity in the chamber (measures) as close as possible to the desired values (the setpoints).

Agent actions: Increase/decrease the percentage of the proportional valves controlling the actuators.

Rewards: Deviation between measure and setpoints (small deviation => high reward, high deviation => low reward).

I have data available, the history of states and actions from a real system. The actions are made by several PID controllers (some of them in cascade). So far I have about 3 month every minutes (with some stops sometimes when a chamber is for example cleaned). The data are continuously logged and every month I get more data. The data includes bad/unwanted states.

For training the RL agent, I am planning to simulate the environment using a supervised learning model (with the predict function), probably XGboost. Is it feasible, are there pitfalls to avoid in this case?

RL is a generic technique that can be applied to any MDP system. From the looks of it you have data to produce a state-space model of your system (system identification, excluding your existing control loops) and then you can use that to drive a simulated exploration of your process and discover a control policy. As this is a continuous process, some discretization will be required

so yes, your quest is feasible, but not trivial!

I experimented sometime ago with the cartpole balancing system, which is a toy, but I found that a good old PID was much better than AI :)


What are some resources with exercises related to neural networks? I am asking for a book (or any other online resource) where we can solve exercises related to neural networks, similar to the books or online resources dedicated to mathematics where we can solve mathematical exercises.

The book Grokking Deep Learning, by Andrew Trask (a PhD student at Oxford University and a research scientist at DeepMind), a wonderful, clean, and plain-English discussion of the basic mechanics that go on under the hood of neural networks - from data flow to updating of weights.  It is written without a slant on normally-wonky math, the concepts are presented and then advanced at a digestible pace for anyone.

Here are a few more possibly useful resources.

Neural Networks: Playground Exercises
Getting Started With Deep Learning: Convolutional Neural Networks
Deep learning focuses on practical aspects of deep learning
First lab assignment in Deep learning
Deeplearning stanford


Detect data in tables of roughly the same structure I would like to train a model that serializes a table of nutrition facts into it's values.
The tables can vary in form and colour, but always contain the same set of keys (e.g. carbs, fats).
Examples for these tables can be found here.
The end goal is to be able to take a picture of such a table and have it's values added to a database.
My initial idea was to train a model on finding subpictures of the individual key/value pairs and then using OCR to find out which value it actually is.
As I am relatively new to ML, I would love to have some ideas about how one could try to build this, so I can do further research on it.
Thanks

Assuming all of the tables will be oriented in similar ways (label and value running horizontally) and that all writing will be printed rather than handwritten, one solution method would be to use an image segmentation method such as edge detection to segregate these horizontal (label, value) pairs and then use a library like Tesseract for OCR.
There are many types of image segmentation methods that may all have value, but if my assumption holds regarding the neat, structured nature of the tables, then I think simple edge detection methods could be sufficient.


Why is my model accuracy high in train-test split but actually worse than chance in validation set? I have trained a XGboost model to predict survival for the Kaggle Titanic ML competition.

As with all Kaggle competitions there is a train dataset with the target variable included and a test dataset without the target variable which is used by Kaggle to compute the final accuracy score that determines your leaderboard ranking.

My problem:

I have build a fairly simple ensemble classifier (based on XGboost) and evaluated it via standard train-test-splits of the train data. The accuracy I get from this validation is ~80% which is good but not amazing by public leaderboard standards (excluding the 100% cheaters).

The results and all the KPIs I looked at of this standard model do not indicate severe overfitting, etc. to me.

However when I submit my predictions for the test set my public score is ~35% which is way below even a random chance model. It is sooo bad I even improved my score by simply reversing all predictions from the model.

Why is my model so much worse on the test?

I know that Kaggle computes their scores a bit differently than I do locally, additionally there is probably some differences between the datasets. Most who join the competition notices at least some difference between their local test scores and the public scores.

However my difference is really drastic and indeed reversing the predictions improves my score. This does not make sense to me because reversing the predictions on my local validations leads to garbage predictions, so this is not a simple problem of generally reversed predictions.

So can you help me understand how those two issues happen at the same time:

Drastic difference between local accuracy and public score
Reversing actually leads to the better public score.

Here is my notebook for the code (please ignore the errors, they are simply because the code does not work on kaggle kernels only locally):

https://www.kaggle.com/fnguyen/titanicrising-test

Looking at your code, one set of data transformations were applied to the train data and a different set of transformations were applied to the test data. Different data transformations could account for different evaluation metric performance.

It is best practices to put all data transformations in a function so they can be applied to all data in a similar way.

Since you are using scikit-learn, sklearn.compose.ColumnTransformer is designed for this purpose. Example code for the Titanic dataset is here.


What are mono-variable and multi-variable neural networks? In this document, the terms "Redes Neuronales estáticas monovariables" and "Redes Neuronales estáticas multivariables" are mentioned.

What are mono-variable and multi-variable neural networks? Is it the same as a multi-layer or uni-layer NN?

I have searched about multivariable/mono-variable static/dynamic neural networks in some books, but at least in those books there's no information about these topics.

I have the idea it refers to the inputs/outputs, but I'm not sure.

What are mono-variable and multi-variable neural networks?

I am not sure about this, because most (if not all useful) neural networks are multivariable neural networks (i.e. they contain multiple parameters). Even the perceptron usually contains more than one parameter, so that terminology isn't clear even to me. Maybe they are referring to the number of inputs (sometimes called variables), but I don't see why this distinction in this context would make sense.

What are static and dynamic neural networks?

To answer this question, I will first quote an excerpt from this document (written in Spanish) to provide some context to Spanish speakers (I am not a Spanish speaker, but I understand 99% of it).

Un primer intento de clasificación puede separ estáticos y dinámicos o recurrentes (fig.2.1)
Los modelos estáticos realizan un mapeo entre entrada y salida. Despreciando el tiempo de procesamiento interno, la salida se obtiene en forma inmediata en función de la entrada, no existe memoria ni dinámica de estados en el sistema neuronal.
Por el contrario los sistemas recurrentes si la poseen, son sistemas realimentados que ante un estimulo de entrada evolucionan hasta converger a una salida estable.
Casos tipicos de ambos sistemas son el Perceptrón (Rosemblatt, 1960a) (de una o múltiples capas) y la memoria asociativa de Hopfield, respectivamente
(Tank, 1987).

So, in this document, the word "dynamic" and "recurrent" are being used interchangeably. An example of a static (i.e. non-recurrent) neural network is the perceptron. An example of recurrent (or dynamic) neural network is the Hopfield network.
Anyway, I recommend you contact the author of that article to ask for clarification (especially, about the mono-variable NNs)!