Successful methods for optical character recognition? Alright, I want to write a mobile app that lets you take a photo of your equation, detects the equation, transforms it from pixels to text and then solves if it's possible.

Right now, I am doing the part where I receive a photo (2D array of pixels) and what to output a resized part of this photo (the part has a rectangular shape) that just contains the equation and some pixels around it.

Basically, I need a model that takes a 2D array and returns a resized version of it.

Since I am new to AI, I don't know what techniques have been successful in optical character recognition.

Any suggestions?

This is not meant to be a direct answer, as OCR is not an area I have experience in, but I want to present a deeper problem, that of mathematical notation.

Mathematics has "exploded" in the last few centuries, and notation is not standard across various sub-fields.  Check out this question on notational change with integrals. In some cases, individual sub-fields have alternate forms of notation.

My suspicion is that this task will be enormously difficult without a great deal of human guidance, so I'd point you towards AI methods that utilize "human-assisted machine learning".


FastDownward PDDL Planner Limitations I recently had a look at automated planners and experimented a little bit with FastDownward. As I wanted to start a toy project, I created a PDDL model for the ordinary 3D Rubik's Cube (of course using a planner may not be the most efficient approach).

Although my model may not necessary be "totally correct" yet, so far it consists of 24 predicates, 12 actions for the respective moves (each with 8 typed parameters, 4 "edge subcubes" and 4 "corner subcubes"). For each of the movable subcubes I have a domain object whose position is basically determined by the respective predicate; overall, at first glance this seemed to me as a model of quite moderate size.

It was indeed not a very complex task to come up with this model and although it currently does not consider orientations of subcubes yet, I simply wanted to give it a try with an instance where only a single move (so application of one action) has to be conducted — I assumed that such a plangraph should level off quite soon, so basically after the first layer where a goal state can be reached.

However, as I started the planner, it soon run out of memory and started to page.

Previously, I only read something on PDDL and the respective sections of Russell & Norvig. I took a closer look at the planner itself and found that it transform the PDDL description into some intermediate description.

I tried to only execute the transformation and after cutting off a third of my Rubik's Cube it — at least — terminated. I investigated the transformed model file and found that the planner/solver actually instantiates (flattens) the actions with its parameters.

Since the Rubik's Cube has a quite unrestricted domain and each action has apparently 8 parameters (4 corners, 4 edges), this inherently results in a huge number of flattened actions. Even more, although I added precondition constraints to ensure distinctness of the parameters (so the very same subcube cannot be both edgecube $e_1$ and $e_2$), the flattened version still contains these invalid actions.

I have the following questions:

are state-of-the-art planners even suitable for such a problem or are they designed for problems where flattening the actions is of great advantage (e.g. few parameters, moderate number of objects per class, etc.). IMHO this would be a major limitation for their applicability in contrast to e.g. CP solvers.
can anyone recommend another planner that is more appropriate and that maybe does not perform the model transformation, which seems to be quite expensive for my PDDL spec (e.g. from this list, https://ipc2018-classical.bitbucket.io, i have chosen FastDownward since it seemed to be among the best...)
does PDDL or even FastDownward allow to specify that the parameters have to be distinct (i just found this: https://www.ida.liu.se/~TDDC17/info/labs/planning/writing.shtml — search for distinct — but this is more than vague), cause this may already lead to a significant reduction of flattened actions.
I'd also be happy for any other recommendations or remarks

You have stumbled upon a common drawback of the vast majority of modern planning technology. The "flattening" you refer to is actually called "grounding" in the community. Indeed, grounding is the first step of almost every planner out there. Planners that don't do this grounding phase are typically referred to as "lifted planners", but their availability is far more limited (i.e., you'd likely need to reach out to an author of a paper to get access to their code).

That said, many of the planners, Fast Downward included, will try to be smart about the grounding phase. This includes simple things such as making sure that typing for the parameters (and the objects that instantiate them) is adhered to, but goes further with certain checks on reachability. The process isn't flawless, but it usually does a good job when the true number of ground actions is relatively small. If your model really does have an astronomical number of ground actions, then you're kind of out of luck here.

To your questions specifically:

Yes, they are designed for problems where you can ground things. But the mark of suitability is the number of reachable ground actions. If this is small enough, then either (1) the planners should be able to handle it or (2) it presents a research opportunity for grounding things better.
This is the most recent work that comes to mind, as well as the papers that cite it:

https://www.aaai.org/ocs/index.php/ICAPS/ICAPS14/paper/view/7948/8033
https://scholar.google.com/scholar?cites=2377320520977506929&as_sdt=2005&sciodt=0,5&hl=en

They aren't assumed to be distinct. There are situations when you wouldn't want them to be, and so it is up to the designer to add the preconditions that ensure they are distinct (note that this is one excellent way to equip the grounder with extra information about what it should generate).
Generally, having actions with many parameters is a recipe for problems that have too many ground actions. Either manually remodeling things yourself, or applying a technique like operator splitting is the approach most often taken.

Finally, if you'd like to have a more directed discussion, feel free to join us over on the slack channel. There may be more eyes on the questions you pose there.


Classification with deeplearning : clean start vs continue training I trained some weights to identify apples and oranges (using YOLOv3).

If I want to be able to identify peaches, which approach is usually recommended:

Start clean and train the 3 classes.
Train the peaches over the already-trained weights (with apples and oranges)

Only train with peaches images
Use all available training data (including apples and oranges)

This is what I have found:

If I start clean, it will take longer until I can get a good result, but the detection is usually better.
Every time I add a new class (using 2.2), the detection get worse for the already learned objects, but it takes less time until I can get a good result (however I suspect that apples and oranges become over-fitted?).
I haven't tested 2.1, as I think that it won't be able to re-adjust the weights for the apples and the oranges.

Is the above expected? What is the recommended course of action?

If the task involves only apples, orange and peaches, you should use method 1. As the number of classes is small, the network cannot generalize well to all classes. As a side note, you should start with the pretrained weights of YOLO v3 as some classes of YOLO v3 may be fruits, which can help your model converge faster.

If the number of classes is large, for example a hundred different fruits, you should use method 2.2 . The model should be able to generalize to all fruits and can converage faster as many fruits look the same. This is the case of transfer learning. In original YOLO v3 training, image net weights for dark net is used for the backbone network. It accelerates the training of YOLO v3.

For 2.1, it will not work as gradient descent will not consider the trained weight. The trained weights will be over written by the weights for peaches.

For a recommended method, it depends on your class size. If the model will continue to add more classes, you should perhaps use 2.2 but only start using 2.2 when you have a considerable amount of classes. Hope I can help you.


How do deepfakes work and how they might be dangerous? Deepfakes (a portmanteau of "deep learning" and "fake") are synthetic media in which a person in an existing image or video is replaced with someone else's likeness.

Nowadays most of the news circulating in the news and social media are fake/gossip/rumors which may false-positives or false-negatives except  WikiLeaks

I know there has been a Deepfake Detection Challenge Kaggle competition for a whooping sum $1,000,000 prize money.

I would like to know how deepfakes work and how they might be dangerous?

In general, deepfakes rely on advanced context-aware digital signal manipulations - usually image, video or audio - that allow for very natural looking modifications of content that previously have been costly or near impossible to produce in high quality.

The AI models, often based on generative adversarial networks (GANs), style transfer, pose estimation and similar technologies, are capable of tasks such as transferring facial features from subject A to replace those of subject B in a still image or video, whilst copying subject B's pose, expression, and matching the scene's lighting. Similar technologies exist for voices.

A good example of this might be these Star Wars edits, where actors faces have been changed. It is not perfect, you can in a few shots see a little instability if you study the frames - but the quality is still pretty good, and it was done with a relatively inexpensive setup. The work was achieved using freely-available software, such as DeepFaceLab on Github.

The technology is not limited to simple replacements - other forms of puppet-like control over output are possible, where an actor can directly control the face of a target in real time using no more than a PC and webcam.

Essentially, with the aid of deepfakes, it becomes possible to back up slanderous or libelous commentary with convincing media, at a low price point. Or the reverse, to re-word or re-enact an event that would otherwise be negative publicity for someone, in order to make it seem very different yet still naturally captured.

The danger of this technology is it puts tools for misinformation into a lot of people's hands. This leads to potential problems including:

Attacks on integrity of public figures, backed by realistic-looking "evidence". Even with the knowledge that this fakery is possible (and perhaps likely given a particular context), then damage can still be done especially towards feeding people with already-polarised opinions with manufactured events, relying on confirmation bias.
Erosion of belief in any presented media as proof of anything. With deepfakes out in the wild, someone confronted with media evidence that went against any narrative can claim "fake" that much more easily.

Neither of these issues are new in the domains of reporting, political bias, propaganda etc. However, it adds another powerful tool for people willing to spread misinformation to support any agenda, alongside things such as selective statistics, quoting out of context, lies in media that is text-only or crudely photoshopped etc.

A search for papers studying impact of deep fakes should find academic research such as Deepfakes and Disinformation: Exploring the Impact of Synthetic Political Video on Deception, Uncertainty, and Trust in News.

Opinion: Video content, presented feasibly as a live capture or report, is especially compelling, as unlike text and still image media, it directly interfaces to two key senses that humans use to understand and navigate the world in real time. In short, it is more believable by default at an unconscious and emotional level compared to a newspaper article or even a photo. And that applies despite any academic knowledge of how it is produced that you might possess as a viewer.


Evolved networks fail to solve XOR My implementation of NEAT consistently fails to solve XOR completely. The species converge on different sub-optimal networks which map all input examples but one correctly (most commonly (1,1,0)). Do you have any ideas as to why that is?
Some information which might be relevant:

I use a plain logistic activation function in each non-input node 1/(1 + exp(-x)).
Some of the weights seem to grow quite large in magnitude after a large number of epochs.
I use the sum squared error as the fitness function.
Anything over 0.5 is considered a 1 (for comparing the output with the expected)

Here is one example of an evolved network. Node 0 is a bias node, the other red node is the output, the green are inputs and the blue "hidden". Disregard the labels on the connections.

EDIT: following the XOR suggestions on the NEAT users page of steepening the gain of the sigmoid function, a network that solved XOR was found for the first time after ca 50 epochs. But it still fails most of the time. Here is the network which successfully solved XOR:

The problem was due to the following issues in my implementation:

The offspring generated in the crossover was not mutated (!)
The mutations did not occur with the expected frequencies (too few links and weight mutations)
The sigmoid activation had to be steepened

Another thing that previously caused issues was the network.activate function. Make sure that you wait for the network to stabilize when doing classification tasks, so all signals have time to propagate through the network.


Can genetic algorithms be used to learn to play multiple games of the same type? Is it possible for a genetic algorithm + Neural Network that is used to learn to play one game such as a platform game able to be applied to another different game of the same genre.

So for example, could an AI that learns to play Mario also learn to play another similar platform game.

Also, if anyone could point me in the direction of material i should familiarise myself with in order to complete my project.

Genetic algorithms and Neural Networks both are "general" methods, in the sense that they are not "domain-specific", they do not rely specifically on any domain knowledge of the game of Mario. So yes, if they can be used to successfully learn how to play Mario, it is likely that they can also be applied with similar success to other Platformers (or even completely different games). Of course, some games may be more complex than others. Learning Tic Tac Toe will likely be easier than Mario, and learning Mario will likely be easier than StarCraft. But in principle the techniques should be similarly applicable.

If you only want to learn in one environment (e.g., Mario), and then immediately play a different game without separately training again, that's much more complicated. For research in that area you'll want to look for Transfer Learning and/or Multi-Task learning. There has definitely been research there, with the latest developments that I'm aware of having been published yesterday (this is Deep Reinforcement Learning though, no GAs I think).

The most "famous" recent work on training Neural Networks to play games using Genetic Algorithms that I'm aware of is this work by Uber (blog post links to multiple papers). I'm not 100% sure if that really is the state of the art anymore, if it's the best work, etc... I didn't follow all the work on GAs in sufficient detail to tell for sure. It'll be relevant at least though.

I know there's also been quite a lot of work on AI in general for Mario / other platformers (for instance in venues such as the IEEE Conference on Computational Intelligence and Games, and the TCIAIG journal).


How can I avoid overfitting when doing parameter tuning? I very often applied a grid search to tune the parameters of my supervised model. I have the feeling that parameter tuning will eventually (very often) lead to overfitting? Is this crazy to say?

Is there a way that we can apply grid search in such a way that it will not overfit?

Yes. Usually you would use cross validation to avoid overfitting during parameter tuning. If your dataset is large enough, and you don't try too many parameter combinations, this will work well, because to "get lucky" and overfit, a parameter combination will need to work very well on many variations of the problem, which is less likely than working well on just one set of data.


How to define or represent evil in logic Is there any well defined method to define or represent evil in abstract logic, binary or AI form?
Video games method of representing evil is relative to the player context (thus subjective, and not pure abstract evil in an objective sense).
What I am asking is there any data defined as well-known evil?
Example:
var x=666;

if (isEvil(x)) {
    //do something.
}

Remark:
Evil Number descried in http://mathworld.wolfram.com/EvilNumber.html doesn't qualify as well-known evil data.
Following Up:
One of the main objectives of the question is to understand scientifically the limits of evil in AI
According to my understanding of: https://en.wikipedia.org/wiki/Evil I think it's mandatory to explore "evil" in religion context in order to come up with valid model for evil. But I don't want go into (religion) debates or any divergence at this stage. hence below points Sums up my understanding:

The only well-known Evil source is the devil (our creator declared the devil as the first common enemy for ALL humans).
Whispering is devil method of attack, If human followed the whisper it will lead to evil. and gradually human Evil grow...
There are other points but I don't see its related to AI in any means.

Based on the above, I asked myself: since AI is human creation, where the evil in AI will come from??! my answer is: directly from us and indirectly by following the devil. So all crimes committed by Evil AI bounded to AI architect/designer/unethical hacker.
The next stage in getting closer to model evil, is to define and classify the evil acts:
Definitions:

Define evil in AI context (draft ver. 0.1): committing crimes against nature, civilizations or humans. And reprogramming, modifying or attacking tech devices/machines to perform malicious agenda.
Crime is broad and relative to the party: example: breaking one government regulations based on the orders of other government. I mean as long each group of humans makes its own laws and regulations unified justice can't be applied on Evil AI.

If my assumption of bounding evil to crime is valid then evil classification inherits crime classification which seems well-defined:
https://en.wikipedia.org/wiki/Crime#Classification_and_categorisation
Next step is to pick an easy to model crime class, prepare training data, ... Do you agree with the follow up? Do you agree that Boolean logic can't determine evil without AI?

I think you're going to have to be reconciled to the subjective nature of reality.  Objectivity is only possible in very special cases such as a Q.E.D. in mathematics, or a solved gamed.  Rationality is bounded, and any intractable problem results in a state of subjectivity/indeterminacy.  Additionally, pure values do not carry moral implications, despite popular associations, although it would be possible to create a game where certain values have negative effects, and the harm they result in could be understood as evil.  (i.e. 666/616 has numerological associations, and numerology can be understood as a proto form of number theory.)

A simple way to define evil would be through behavioral models in Game Theory.
In Game Theory, there is a concept knowns as the superrational strategy.
Superrationality may be understood as the logical/mathematical expression of the Golden Rule: "do unto others as you would have them do unto you."
The Golden Rule forms the basis for most religions, in the context of empathy/compassion and altruism.
If the agent is evil, it will always betray, even when the competitor has shown a willingness to cooperate.

Thus evil is defined as the opposite of the Golden Rule.  (Possibly we would call this the Brimstone Rule;)


Current research on indoor localization and navigation in changing environment? I'm trying to get up to speed on the latest research regarding indoor localization, scene classification, navigation in changing environment, etc.

Any advice would be appreciated, but I'm especially interested in recent research papers from vetted sources.

I would start with:

PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization
VidLoc: A Deep Spatio-Temporal Model for 6-DoF Video-Clip Relocalization
Image-based localization using LSTMs for structured feature correlation
Image-based Localization using Hourglass Networks
UAN: Unified Attention Network for Convolutional Neural Networks
Scene Coordinate and Correspondence Learning for Image-Based
Localization

You can also search for articles using the 7-scenes dataset (Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images) which is pretty standard


What are the conditions of convergence of temporal-difference learning? In reinforcement learning, temporal difference seem to update the value function in each new iteration of experience absorbed from the environment.

What would be the conditions for temporal-difference learning to converge in the end? How is it guaranteed to converge?

Any intuitive understanding of those conditions that lead to the convergence?

There are different TD algorithms, e.g. Q-learning and SARSA, whose convergence properties have been studied separately (in many cases).

In some convergence proofs, e.g. in the paper Convergence of Q-learning: A Simple Proof (by Francisco S. Melo), the required conditions for Q-learning to converge (in probability) are the Robbins-Monro conditions

$\sum_{t} \alpha_t(s, a) = \infty$
$\sum_{t} \alpha_t^2(s, a) < \infty,$

where $\alpha_t(s, a)$ is the learning rate at time step $t$ (that can depend on the state $s$ and action $a$), and that each state is visited infinitely often.

(The Robbins-Monro conditions (1 and 2) are due to Herbert Robbins and Sutton Monro, who started the field of stochastic approximation in the 1950s, with the paper A Stochastic Approximation Method. The fields of RL and stochastic approximation are related. See this answer for more details.)

However, note again that the specific required conditions for TD methods to converge may vary depending on the proof and the specific TD algorithm. For example, the Robbins-Monro conditions are not assumed in Learning to Predict by the Methods of Temporal Differences by Richard S. Sutton (because this is not a proof of convergence in probability but in expectation).

Moreover, note that the proofs mentioned above are only applicable to the tabular versions of Q-learning. If you use function approximation, Q-learning (and other TD algorithms) may not converge. Nevertheless, there are cases when Q-learning combined with function approximation converges. See An Analysis of Reinforcement Learning with Function Approximation by Francisco S. Melo et al. and SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation by Bo Dai et al.


Solving a Multi-Armed, "Multi-Bandit" Problem This is the problem: I have 66 slot-machines and for each of them I have 7 possible actions/arms to choose from. At each trial, I have to choose one of 7 actions for each and every one of the 66 slots. The reward depends on the combination of these actions but the slots are not equal, that is, pulling the same arm for different slots gives different results. I do not care about an initial state or feature-vector as the problem always starts from the same setting (it is not contextual). My reward depends on how I pull one of the 7 arms of all of the 66 bandits simultaneously, where, as said, each slots has its own unique properties towards the calculation of the total reward. Basically, the action space is a one-hot encoded 66x7 matrix.

My solution: I ignored the fact that I do not care about a feature vector or state and I treated the problem using a deep NN with a basic policy-gradient algorithm, where I increase directly the probability of each action depending on the reward I get. The state simply does not change, so the NN receive always the same input. This solution does work effectively in finding an approximately optimal strategy, however, it is very computationally expensive and something tells me I am overkilling the problem.

However, I do not see how I could apply standard solutions to MAB such as epsilon-greedy and so on. I need simultaneity between the different "slot-machines" and if I just take each possible permutation as a different action, in order to explore them with greedy methods, I get way too many actions (in the order of 10^12). I have not found in the literature something similar to this multi-armed multi-bandit problem and I am clueless if anything like that has ever been considered - perhaps I am overthinking it and this can be somehow reduced to a normal MAB?

Although you can frame your problem as a bandit problem or RL, it has other workable interpretations. Critical information from your comments is that:

Total reward is not a simple sum of all the results from 66 different machines. There are interactions between machines.
Total reward is deterministic.

This looks like a problem in combinatorial optimisation. There are many possible techniques you can throw at this. Which ones work best will depend on how nonlinearities and dependence between choices on different machines affect the end results.

Best Case

With deterministic results, if changes between machines were completely isolated, you could search each machine in turn, because you can treat all other 65 components as a constant if you don't change their settings. That would be very simple to code and take $7 \times 66 = 462$ steps to find the optimimum result.

Worst Case

In the worst case, the dependencies are so strong and chaotic that there is essentially no predictable difference between changing a single machine's setting and all of them. Pseudo-random number generators and secure hashing functions have this property, as do many quite simple physical systems with feedback loops.

In the worst case, there will be a "magic setting" with best results, and only a brute force search through all combinations of levers will find it.

In order to apply any more efficient search method, you have to assume that the response to combinations of levers is not quite so chaotic.

How to Search?

It seems likely from your description, that the best search algorithm is going to be somewhere between simple machine-by-machine optimisation and an exhaustive global search. However, it is hard to tell just where on that spectrum it lies.

There are a few different ways to frame it as reinforcement learning. For instance, you could use current switch combination as state, and run 66 switch changes as an "episode".

I would suggest that genetic algorithms are a good match for this search task, assuming there is at least some local-only effect that means combining two good solutions is likley to result in a third good solution. Genetic algorithms don't need calculations for gradients, and fit nicely with discrete combinations. Your genome can simply be the 66 different switch positions, and the fitness rating your black box score for those positions.

Plenty of other combinatorial search algorithms are available. Enough to fill a book or two. One place you could look for inspiration is Clever Algorithms: Nature-Inspired Programming Recipes which is a free PDF.


Can Convolutional Neural Networks be applied in domains other than image recognition? I'm new in this argument, my question is:

Can convolution be applied in other contexts different from image recognition?
Is there a good source to learn from?

Yes, Convolutional Neural Networks(CNNs) can and have been applied to non-image problems. Arguably, any problem in which the location of a feature(s) is relevant can be attempted via CNNs. CNNs works under the assumption that points close to each other in the data share some correlations/relationship whist points further apart don't share as much information. So, theoretically, if you can phrase your problem so that it meets this requirement(s), it can be attempted by a convolutional neural network. Here are few applications of CNNss that are dont involve images:

Speech recognition
Character-level Convolutional Networks for Text Classification
Text classification using CNNs
Natural Language Processing
Time Series Forecasting
Sentence modeling
I've used 1-D CNNs on spectral data. Here are some examples of CNNs applied to spectral data :

Spectral Convolutional Neural Network for music classification
Convolutional neural network based classification for hyperspectral data


What is "temporal depth"? I need some explanation about the following paragraph (page 3) from the paper A Novel Approach for Robust Multi Human Action Detection and Recognition based on 3-Dimentional Convolutional Neural Networks.

  We introduce a 3D convolution neural network with the following notations: $I(x, y, d)$ as an input video with a size of $x y$ and $d$ the temporal depth

What is "temporal depth"? Is it the number of frames?

Temporal Depth is a third parameter of a time series data. For example if you have a video clip of length 25 frames and on training a model you are giving first five frames with respect to time. Your temporal depth will be 5.


How could an AI interpret the requests of a human without the human needing to specify every detail? Let the request be the thing that we asked the AI to do, not necessarily using commands.

Nowadays, we have our personal AI in our devices: Siri by Apple, Cortana by Microsoft, and so on. Most times, when we ask them to do certain tasks, they do the tasks for us. However, their action is based on the list of commands. When they don't clearly recognize the commands in our request, they suggest us to use certain commands. It is clear that there are limits to our choices (requests).

So, let's suppose that we have an AI that can interpret requests. There may not be commands in our request. AI is fully able to do anything in order to do what it is asked for. Basically, I am talking about an independent AI.

Now, consider the scenario where an AI is asked to clean the room. AI is allowed to throw away garbage, and move unnecessary (or unused) stuff into the storage.

This is the list of things that were in the room at the moment:

A stained blanket
Various decorations
A dead clock on the wall
Various unused items in the desk drawer
A lost Airpod under the bed
A sleeping cat in the bed

In this condition

Is washing a stained blanket a part of cleaning?
How can AI tell if anything is in use? Are decorations in use?
Would dead clock that only needs battery replacement considered garbage?
Would items in the desk drawer be included in AI's to-be-cleared list?
Would AI be able to recognize the difference between unused and lost?
What would happen to the poor cat?

Since there are many holes in the scenario and questions, I would like to know how the answers are derived.

This is basically the problem of commonsense knowledge. It is AI-complete.

If we knew how to solve it, Siri and Cortana wouldn't be as limited as they are.


What does equation in the "related work" section of the GAN paper mean? I was going through the paper on GAN by Ian Goodfellow. Under the related work section, there is an equation. I cannot decipher the equation. Can anyone help me understand the meaning of the equation?

$$\lim_{\sigma \to 0} \nabla_{\mathbf  x} \mathbb E_{\epsilon \sim \mathcal N(0, \sigma^2 \mathbf I)} f(\mathbf x+\epsilon) = \nabla_x f(\mathbf x)$$

Also, any guide to understanding mathematical notation for reading research paper is highly appreciated.

In full:

  The limit, as standard deviation $\sigma$ tends towards zero, of the gradient with respect to vector $\mathbf{x}$, of the expectation - where perturbation $\epsilon$ follows the normal distribution with mean 0 and variance $\sigma^2$ times identity vector $[1,1,1,1...]$ * - of any function $f$ of $\mathbf{x}$ plus $\epsilon$ is equal to the gradient with respect to $x$ of the same function of $\mathbf{x}$.

If we break that down:

$$\lim\limits_{\sigma \rightarrow 0}$$

  The limit, as standard deviation $\sigma$ tends towards zero of

$$\nabla_\mathbf{x}$$

  the gradient with respect to vector $\mathbf{x}$ of

$$\mathbb{E}$$

  the expectation ...

$$\mathbb{E}_{\epsilon \sim \mathcal{N}(0, \sigma^2\mathbf{I})}$$

  [the expectation] - where perturbation $\epsilon$ follows the normal distribution with mean 0 and variance $\sigma^2$ times identity vector $[1,1,1,1...]$ * - of

$$f(\mathbf{x} +\epsilon)$$

  any function $f$ of $x$ plus $\epsilon$

$$ = \nabla_x f(\mathbf{x})$$

  is equal to the gradient with respect to $\mathbf{x}$ of the same function $f(\mathbf{x})$.

Basically it says that taking small perturbations of a vector input to a function and measuring the gradient at those varied points can be used to give you a valid estimate of the true gradient at the point you are making variations of.

In terms of understanding the equations, read introductory texts to the research area, and if like me your maths has been unused for many years before attempting this, expect to spend time and effort. Re-read the equations, memorise and write out the basic ones from the field, apply them to simple problems that might be presented in text books. Reading maths equations is not much different to reading music or reading another language - it takes concentration, practice, time and effort to become fluent enough to read an equation and comprehend it. The different research fields can be quite different too, some might be as similar enough to get by with what you know already, others may require learning all over again.

* I am not 100% certain of the interpretation of $\mathbf{I}$ as an identity vector - a matrix may be more appropriate, which depends on the form of $\mathcal{N}(\mu, \sigma^2)$ when handling vector distributions. A matrix form for the second argument would be more general and allow for covariance, although the use of $\mathbf{I}$ would then explicitly remove covariance and make each component of $\epsilon$ independent, which is required for this result.


Adding BERT embeddings in LSTM embedding layer I am planning to use BERT embeddings in the LSTM embedding layer instead of the usual Word2vec/Glove Embeddings. What are the possible ways to do that?

Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead.

# Sample code
# Model architecture

# Custom BERT layer
bert_output = BertLayer(n_fine_tune_layers=10)(bert_inputs)

# Build the rest of the classifier
dense = tf.keras.layers.Dense(256, activation='relu')(bert_output)
pred = tf.keras.layers.Dense(1, activation='sigmoid')(dense)

model = tf.keras.models.Model(inputs=bert_inputs, outputs=pred)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(...)

This article will walk you through the entire process of creating the custom BERT layer along with example code. Give it a read.


Programming a bandit to optimize donations I'm developing a multi-armed bandit which learns the best information to display to persuade someone to donate to charity.

Suppose I have treatments A, B, C, D (which are each one paragraph of text). The bandit selects one treatment to show to a person. The person is given $1 and has to decide how much (if any) to donate, in increments of one cent. The donation decision is recorded and fed to the multi-armed bandit, who will then re-optimize before another person is shown a treatment selected by the bandit.

How should I program the bandit if my objective is to maximize total donations? For example, can I use Thompson sampling, and if a participant donates $0.80, I count that as 80 successes and 20 failures?

It does not matter to the bandit algorithm that rewards are quantised or fractional, or that they can vary. This is true for pretty much all bandit optimisation algorithms.

So just treat the $0.80 donation as a real valued reward of 0.8, that occurs on a single timestep.

Treating a single reward on a single timestep as if it were multiple rewards across multiple timesteps might cause problems, depending on which algorithm you were using, even if the reward averaged to the same value. For instance it may skew the maths away from theoretical assumptions used in upper confidence bound action selection or gradient-based solutions.

It occurs to me that your $1 experiment is supposed to be some kind of proxy for the real deployment, where donations will be sparse and may vary more. Whilst this may help a little, essentially you are just setting some kind of prior based on results from a test. The real bandit algorithm starts once the system is deployed into production, and could well have very different results when the goal is to collect real and larger donations.


Goal oriented Action Planning with multiple Agents I'm a little bit stuck:

I implemented an AI with GOAP (Goal oriented Action Planning, http://alumni.media.mit.edu/~jorkin/gdc2006_orkin_jeff_fear.pdf) for a simulation game. That works fine.

Now I want that the agents can cooperate (e.g. doing actions together). What is in this case the best AI-Design that the GoapActions keep loose couplet?

Should they plan together? (what is in this case the "worldstate"?)Or Should they share their plans?

Example
Agent1:
Worldstate Agent 1: isLonely= true
Goal Agent1: isLonely = false

Plan Agent1: AskAgent2ToTalk -> TalkToAgent2

Agent2
Worldstate Agent 2: hasWood = false
Goal hasWood = true

Plan Agent2: GetAxe -> ChopWood -> BringWoodToSupply

How I get this constellation?

Agent1 Plan: TalkToAgent2
Agent2 Plan: TalkToAgent1 -> GetAxe -> ChopWood -> BringWoodToSupply

Or if they are talking and one of the agents is interrupted (e.g. by an attacking enemy) the other agent must know that his TalktoAgent2 Action has ended.

This evening I got inspired by this paper: http://www.dphrygian.com/bin/David-Pittman-GOAP-Masters-Thesis.doc

(GOAP paired with the Command and Control Pattern)

What do you think about this solution?

Each goal has a relevance (that depends on the agent needs)

When agent1 working on the "AskAgent2ToTalk" Action, it only sends a goal recommendation to agent2. Explicit means this: The agent only sends a list of modifiers for each goal relevance (in this case it could be a bonus for the "SocialInteraction" Goal. The value depends on the relationship between the agents) and the recommend goal.

if agent2's desired goal is the recommend goal, the agent replans.


Medical diagnosis systems based on artificial neural networks Are there any medical diagnosis systems that are already used somewhere that are based on artificial neural networks?

Yes, there are many, actually. A Google search turned this paper Artificial Neural Networks in Medical Diagnosis (2011) by Al-Shayea up.

Not only are they used in disease diagnosis, but even with things like prescribing medicines. In fact, the top project for a hackathon at my school analysed thousands of research articles, and took a patient's medication history as input, to best recommend them specific medicines. Check it out.


How to implement RAM versions of Atari games I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in short, I used double deep Q-learning, and trained it like it was CartPole or lunar-lander environment. In CartPole, the observation was a vector of 4 components. In that case, my double deep Q-learning agent solved the environment, but in the breakout-ram version whose observation was a vector of 128 elements, it was not even close.

Did I miss something?

To my knowledge from reading about model-based and model-free reinforcement learning,

DQN and Double DQN are model-free reinforcement learning methods. (Why am I mentioning this, see below):

https://link.springer.com/referenceworkentry/10.1007%2F978-1-4614-7320-6_674-1

You should see under the heading "Definition" (you may need to scroll down in the web-page) in the web-page (can be accessed by the above link), it states that: "model-free techniques require extensive experience."

Now extensive experience, depending of how fast you can go through states, can take a few days even weeks...model-free methods require a lot of samples to learn.

Also there are a lot of states: there are (256^128) states. That is a really big number (I'm just emphasizing that training may take a long while)

Due implementation specifics not being supplied in question, I am assuming your implementation is correct, although...you are using RAM. DQN used image data, I very quickly skimmed the Double DQN paper: https://arxiv.org/pdf/1509.06461.pdf . I am assuming they used image data as well because Double DQN was compared to DQN.


Is it possible to classify data using a genetic algorithm? Is it possible to classify data using a genetic algorithm? For example, would it be possible to sort this database?

Any example in Matlab?

It is possible, but is a pretty terrible idea.

There are a few options. One is to not use the GA as a direct classifier, but instead use a GA to learn the parameters of another classification model like a neural network. The basic idea of a GA is that it (very roughly speaking) forms a black-box method for searching an arbitrary space for solutions that minimize or maximize some function.

Here, you would be searching the space of possible neural network topologies and/or weights to find one that minimizes the misclassification rate.

Another approach is that taken by what are sometimes called Learning Classifier Systems (LCS) or Genetics Based Machine Learning (GBML). This approach is to use evolutionary mechanics to evolve rule sets of the form "if X condition is true, then do/classify Y". That's a more direct method of solving this sort of problem. You define some features on your dataset, and the algorithm tries to learn rules based on those features.

The problem with any of these approaches is just that there are so many better ways to solve the problem. Remember, a GA is basically a black-box that's supposed to work acceptably well for a huge range of unknown problems. But I'm not solving a huge range of unknown problems. I'm trying to separate ham from spam on one dataset. I can come up with methods that simply do that job better and more quickly than a GA has any real hope of doing.


Do I need classification or regression to predict the availability of a user given some features? While studying data mining methods I have come to understand that there are two main categories:

Predictive methods:

Classification
Regression

Descriptive methods:

Clustering
Association rules

Since I want to predict the user availability (output) based on location, activity, battery level (input for the training model), I think it's obvious that I would choose "Predictive methods", but now I can't seem to choose between classification and regression. From what I understand this far, classification can solve my problem, because the output is "available" or "not available".

Can classification provide me with the probability (or likelihood) of the user being available or not available?

As in the output wouldn't just be 0 (not available) or 1 (for available), but it's be something like:

$80\%$ available
$20\%$ not available

Can this problem also be solved using regression?

I get that regression is used for continuous output (not just 0 or 1 outputs), but can't the output be the continuous value of the user availability (like the output being $80$ meaning user is $80\%$ available, implicitly the user is $20\%$ unavailable).

Yes. For instance, the popular softmax regression gives you probability distribution for each class.
Yes. Softmax is a regression over a set of discrete classes.

We can use regression for classification, the most common strategy is to grab the most likely class for the prediction.


How to analyze data before going for machine learning training? For example, I have the following csv: training.csv
I want to know how I can determine which column will be the best feature for getting the output prediction before I go for machine training.
Please do share your responses

Though there is no universal method which can be blindly used for all datasets, but here is what i usually do;

Fill missing values using interpolation or mean, if missing values
are less than 10-15 percent of number of rows else drop the column.
Encode categorical data using some kind of encoding, e.g. one hot, etc.
Then normalize/rescale columns.
Now look at the variance in each feature. Usually, features with more variance are more important.
Next, see the correlation among columns. If two columns are highly
correlated, you only need to keep only one.


What's an appropriate algorithm for classification with categorical features? My input data consists of a series of 8 integers. Each integer is a discrete token, rather than a relative numeric value (i.e. '1' and '2' are as distinct as are '1' and '100').  The output is a single binary value indicating success or fail.  For example:

fail,12,35,60,82,98,111,142,161
success,23,46,59,87,102,121,145,161
fail,13,35,65,83,100,102,122,161

I have say 500,000 of these entries.

Success or failure is determined by the combination of the eight tokens that go to make up the input. I am certain that no single token will dictate success or failure, but there may be particular tokens or combinations of tokens which are significant in determining success or failure, I don't know, but would like to know.

My question is, what kind of machine learning algorithm should I implement to answer the question of which tokens and combinations of tokens are most likely to lead to success?

In case it's relevant or useful, a few more notes on the input data:

There is a limited range of tokens (and thus integers) in each slot. So with this data input:

success,A,B,C,D,E,F,G,H

A is always say one of 1, 2, 3, 4 or 5.  B is always one of 6, 7 or 8.  C is always one of 9, 10, 11 or 12.  So in the general case, possible values for A are never possible values for the other slots and there are between 2 and 12 values for each slot.  No idea if that makes a different to the answer but wanted to include it for completeness.

What you have is called a classification problem with categorical features. That is, the features can be represented numerically, but the numbers have no relative meaning.

Algorithms that rely on smooth function approximation will probably not work well here. These would include classic approaches to regression, and also function approximation via a neural network. That's because the data are anything but smooth!

In contrast, classic classification algorithms like Quinlan's C4.5 decision tree learner, (implemented in the Weka Toolkit as J48, and possibly in SciKitLearn as DecisionTreeClassifier, though the documentation is less clear), are ideal for this: they actually work by splitting up numeric values into discrete categories anyway, so there's no issue at all for them. Most versions also support a way to pre-tag features as categorical, and the algorithms rely on the cross-entropy of each feature's categories, without making assumptions of smoothness.


How to evaluate the goodness of images generated by GANs? As we all know, there has been tons of GAN variants featuring different aspects of the image generation task such as stability, resolution or the ability to manipulate images. However, it is still confusing to me that how do we determine that images generated by one network are more plausible than images generated by another?

PS: could someone with higher reputation create more tags like image generation?

I understand that you want to know about methods that we can use to evaluate GANs (Generative Adversarial Networks).

How can GANs be evaluated?

One Discriminator on Separate GANs

We can train a Discriminator beforehand and then we can use this Discriminator on various Generators to see what does this Discriminator say about the images generated from each Generator. The average output of the Discriminator for images generated from one Generator can be compared with that of the images generated from another Generator. This means that according to this Discriminator, one Generator is better than another.

Comparing Probabilistic Models

This is the class of evaluation metrics that is attracting research attention recently. Basically, all generative models are probabilistic in nature, even GANs. When we ask a generative model to generate something like an image, we are simply sampling from a probability distribution. This means that if we can compare the probability distributions of our Generator and that of our original data then we will have an evaluation metric. We can come up with a kernel function to define these probability distributions more accurately. Then we can take our generated samples from the Generator and see the probability of that sample being taken from the original distribution. Researchers have also utilized KL-divergence for comparing samples generated from two distributions.

Note: KL-divergence can not be directly applied in this case because corresponding ground truth is not available. However, there are some modifications proposed in the research work mentioned.

Recommended Readings

A NOTE ON THE EVALUATION OF GENERATIVE MODELS
UNLEARNING FOR BETTER MIXING
A QUANTITATIVE MEASURE OF GENERATIVE ADVERSARIAL NETWORK DISTRIBUTIONS


How can a DQN backpropagate its loss? I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I understood the math and intuition behind backpropagation. Now, I'm stuck with deep q-learning. I've tried to get an agent to learn in various environments. But somehow nothing works out. So there has to be something I'm getting wrong. And it seems that I do not understand the critical part right at least that's what I'm thinking.

The screenshot is from this video.

What I'm trying to draw here is my understanding of the very basic process of a simple DQN. Assuming this is right, how is the loss backpropagated?
Since only the selected $Q(s, a)$ values (5 and 7) are further processed in the loss function, how is the impact from the other neurons calculated so their weights can be adjusted to better predict the real q-values?

It is quite common in DQN to instead of having the neural network represent function $f(s,a) = \hat{q}(s,a,\theta)$ directly, it actually represents $f(s)= [\hat{q}(s,1,\theta), \hat{q}(s,2,\theta), \hat{q}(s,3,\theta) . . . \hat{q}(s,N_a,\theta)]$ where $N_a$ is the maximum action, and the input the current state. That is what is going on here. It is usually done for a performance gain, since calculating all values at once is faster than individually.

However, in a Q learning update, you cannot adjust this vector of output values for  actions that you did not take. You can do one of two things:

Figure out the gradient due to the one item with a TD error, and propagate that backwards. This involves inserting a known gradient into the normal training update step in a specific place and working from there. This works best if you are implementing your own backpropagation with low-level tools, otherwise it can be a bit fiddly figuring out how to do it in a framework like Keras.
Force the gradients of all other items to be zero by setting the target outputs to be whatever the learning network is currently generating.

If you are using something like Keras, the second approach is the way to go. A concrete example where you have two networks n_learn and n_target that output arrays of Q values might be like this:

For each sample (s, a, r, next_s, done) in your minibatch*

Calculate array of action values from your learning network qvals = n_learn.predict(s)
Calulate TD target for $(s,a)$ e.g. td_target = r + max(n_target.predict(next_s)) (discount factor and how to handle terminal states not shown)
Alter the one array item that you know about from this sample qvals[a] = td_target
Append s to your train_X data and qvals to your train_Y data

Fit the minibatch n_learn.fit(train_X, train_Y)

* It is possible to vectorise these calculations for efficiency. I show it as a for loop as it is simpler to describe that way


If deep Q-learning starts to choose only one action, is this a sign that the algorithm diverged? I'm working on a deep q-learning model in an infinite horizon problem, with a continous state space and 3 possible actions. I'm using a neural network to approximate the action-value function. Sometimes it happens that, after a few steps, the algorithm starts choosing only one between the possible actions (apart from a few steps where I suppose it explores, given the epsilon-greedy policy it follows), leading to bad results in terms of cumulative rewards. Is this a sign that the algorithm diverged?

Is this a sign that the algorithm diverged?

It is a common sign of a problem with learning process. That includes divergence due to poor hyper-parameters, even just bad luck. But it can also point to a design/architecture problem.

Other common causes of algorithm failing with a fixed action choice include:

Neural network inputs not scaled before use.
Large reward values causing large initial squared errors (either re-scale rewards or reduce learning rate to fix)
State representation too far from Markov property assumption
A bug in code (almost anywhere, unfortunately)
Catastrophic forgetting due to focusing too much on specific results and generalising from them incorrectly. Your agent might be suffering from this if it starts to learn correctly, reaching some level of competence at the task before failing.


Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention? In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix:

The reason why is stated that this constrains the distribution of the weights of the output to have a standard deviation of 1.
Quoted from Transformer model for language understanding | TensorFlow:

For example, consider that $Q$ and $K$ have a mean of 0 and variance of 1. Their matrix multiplication will have a mean of 0 and variance of $d_k$. Hence, square root of $d_k$ is used for scaling (and not any other number) because the matmul of $Q$ and $K$ should have a mean of 0 and variance of 1, and you get a gentler softmax.

Why does this multiplication have a variance of $d_k$?
If I understand this, I will then understand why dividing by $\sqrt({d_k})$ would normalize to 1.
Trying this experiment on 2x2 arrays I get an output of 1.6 variance:

In statistics, if $X$ and $Y$ are independent and randomly distributed variables:
$\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\
Var(X + Y) = Var(X) + Var(Y) \\
\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y] \\
Var(XY) = (Var(X) + \mathbb{E}[X]^2)(Var(Y) + \mathbb{E}[Y]^2) - \mathbb{E}[X]^2\mathbb{E}[Y]^2$
Let $Q$ and $K$ be random $d_k$ x $d_k$ matrices, where each entry is a some random distribution with $0$ mean and $1$ variance. Every entry is independent from each other.
Since each entry of $Q$ and $K$ have identical distribution, we can focus only on the top-left-most element of $QK$ without loss of generality. The same applies to every other element.
The top-left-most element of $QK$ is $\displaystyle \sum_{i=0}^{d_k} Q_{1,i} K_{i, 1}$.
Since $Q$ and $K$ are independent:
$\mathbb{E}[Q_{1, i} K_{i, 1}] = \mathbb{E}[Q_{1, i}] \mathbb{E}[K_{i, 1}] = 0 \\
Var(Q_{1, i} K_{i, 1}) = (Var(Q_{1, i}) + \mathbb{E}[Q_{1, i}]^2)(Var(K_{i, 1}) + \mathbb{E}[K_{i, 1}]^2) - \mathbb{E}[Q_{1, i}]^2\mathbb{E}[K_{i, 1}]^2 = 1$
And so summing up $d_k$ of them:
$\mathbb{E} \left[\displaystyle \sum_{i=0}^{d_k} Q_{1,i} K_{i, 1} \right] = \displaystyle \sum_{i=0}^{d_k} \mathbb{E} \left[ Q_{1,i} K_{i, 1}  \right] = 0 \\
Var\left(\displaystyle \sum_{i=0}^{d_k} Q_{1,i} K_{i, 1} \right) = \displaystyle \sum_{i=0}^{d_k} Var\left( Q_{1,i} K_{i, 1}  \right) = d_k$
For your code block, you are computing the dot product of matrices $a$ and $b$, when you should be doing a matrix multiplication (the attention function multiplies $Q$ by $K$ after all, which is the vectorized form of dot-product -- it doesn't actually do dot product). It should work out to unit variance.
edit: the last paragraph is incorrect, as dot is the same as matrix multiply in the above case


What is the relation between multi-agent learning and reinforcement learning? What is the relation between multi-agent learning and reinforcement learning?

Is one a sub-field of the other? For instance, would it make sense to state that your research interest are multi-agent learning and reinforcement learning, or would that be weird as one includes most of the topics of the other?

I think there is an intersection. There are problems that are in reinforcement learning and in learning in multi-agent systems. There are problems in reinforcement learning, but not exactly in multi-agent systems. And there is learning in multi-agent systems that is not through reinforcement learning. For sort you can say: multi-agent reinforcement learning. I recommend take a look at these references:

Bu, Lucian, Robert Babu, and Bart De Schutter. "A comprehensive survey of multiagent reinforcement learning." IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38.2 (2008): 156-172.
Busoniu, Lucian, Robert Babuska, and Bart De Schutter. "Multi-agent reinforcement learning: A survey." 2006 9th International Conference on Control, Automation, Robotics and Vision. IEEE, 2006.


What are the state-of-the-art Person-Detektion / Human-Segmentation? I would like to use a deep learning approach to detect people in videos. I have found some freely accessible implementations like Human Segementation with Pytorch or BodyPix / DeepLab / Pixellib with Tensorflow. They all work well, but with many it happens that, for example, half hand is not detected or if a person is sitting in the picture only the legs and the head are detected. Are there other approaches to detect people who are freely accessible or is that state-of-the-art?
I had imagined such problems have been solved, but I don't know so much about it. Thanks for your answers.

You can use any state of the art object detection such as DETR, EfficientDET, YOLO etc. since they have an object 'person' in them. But, if the density of people is high in the images, then, you should look at 'Finding Tiny Faces' which is good for that problem (this will help you detect faces only, think group photos). If you want to find the whole person even if they are occluded, then, it will be difficult (you can then think of segmentation techniques and then, try to find out the people but it will be more taxing).


What are the differences between stability and convergence in reinforcement learning? The terms are mentioned in the paper: “An Emphatic Approach to the Problem of off-Policy Temporal-Difference Learning.” (Sutton, Mahmood, White; 2016) and more, of course.

In which paper, they proposed the proof of "stability" but not convergence.

It seems that stability is guarantee if the "key matrix" is shown to be positive definite. However, convergence requires more than that.

I don't understand the exact difference between the two.

Sometimes when training, particularly in reinforcement learning, the model can become unstable due to the amount of variance that exists in the training data that the agent generates by interacting with the environment.  This is certainly a problem at the start of training as you can get huge outliers in the data because the agent is behaving randomly.  You can find that just one update to the policy could potentially make it collapse because it moves the policy into some obscure region, e.g. so the agent always take a particular action.  You can make training more stable by using larger batches and a smaller learning rate so it takes smaller steps at a time, but the downside to that is training is slower.  So you need to test different hyperparameters to find a good trade-off between the two.  You can also use an training architecture such as Proximal Policy Optimization (PPO) which clips the amount the policy can move in any given update to try and maintain some stability.

Convergence is a term used to describe when the model has found an optimal policy and isn't learning any further, usually demonstrated when the reward plateaus for a certain number of episodes.  Of course, it may have settled on a local optima, and other global optima may exist; the data you present and the way you train your model may yield better results - again, all part of testing and experimentation.


Why is a softmax used rather than dividing each activation by the sum? Just wondering why a softmax is typically used in practice on outputs of most neural nets rather than just summing the activations and dividing each activation by the sum. I know it's roughly the same thing but what is the mathematical reasoning behind a softmax over just a normal summation? Is it better in some way?

There are probably multiple different explanations and reasonings, but I can offer you one. If your output vector contains negative values, to get something that's related to probabilities (all components positive, summing to $1$) you cannot do what you suggested because you can possibly get a negative probability which doesn't make sense. Good property of exponential function used in softmax, in this case, is that it cannot give negative values, so regradless of your output vector, you will never get negative probability.

You could suggest adding some positive offset vector $\mathbf d$ to your output vector to get rid of negative values but there are couple of problems. First, you cannot know in advance your range of negative output values so that you can know what offset vector to use to cover all possible cases. Second, with such strategy in some cases you could get unrealistic results. For example let's assume output vector is $[-0.1, 0.2, 0.3]^T$ and offset vector is $[0.1, 0.1, 0.1]^T$. If you add those 2 you get $[0, 0.3, 0.4]^T$. Probability of first class would be $0$ since the numerator is $0$. That is very overconfident result and we probably wouldn't want to get the result of $0$ for this class. The result would change depending on the offset vector. Let's say that offset vector now is $[0.3, 0.3, 0.3]^T$. Addition of offset vector and output vector gives $[0.2, 0.5, 0.6]$. In this case probability of first class is now $0.2/(0.2 + 0.5 + 0.6) = 0.15$. We see that changing the offset vector changes the values of probabilities and as components of offset vector $\rightarrow \infty$ probabilities for all classes $\rightarrow 0.33$. We would likely want to get the same result regardless of the scale of the values of the vector, we only care about its relative relationships. Another good property of softmax is that it is shift invariant
\begin{align}
p_i &= \frac{e^{x_i + d}}{\sum_{j=1}^n e^{x_j + d}}\\
&= \frac{e^d \cdot e^{x_i}  }{ e^d \cdot \sum_{j=1}^n e^{x_j} }\\
&= \frac{e^{x_i}}{\sum_{j=1}^n e^{x_j}}
\end{align}
so we see that the probability of $i$-th component is independent of offset vector. Apparently softmax doesn't care about the scale of values, it manages to capture the relative relationships of the components.


Why do we need to provide false labels to the discriminator on purpose to train GANs? This is the tutorial that I used to learn about GANs. In this tutorial, it taught us to intentionally provide false labels to "fool" the discriminator, but does it make the discriminator actually inaccurate? I don't quite understand his explanation, can anyone help me?

in this tutorial, it taught us to intentionally provide false labels to "fool" the discriminator, does it make discriminator actually inaccurate?

When training GANs, the training steps for the generator and discriminator are separate:

There is a training stage for the discriminator, where it is presented with a mix of generated and real data, all correctly labelled. It is important at this stage to not update the generator (otherwise it will get worse by helping to make the fake data look more fake).

There is a training stage for the generator, where the discriminator is presented with only generated data, all labelled incorrectly as if it were real. It is important at this stage to not update the discriminator.

Typically you will alternate between the two stages frequently, making small updates to discriminator and generator separately. Some GANs use metrics to decide how much of each to do, because you don't want either the generator or discriminator to win outright and stop progress - at least at the start.
So yes there is a stage where you deliberately "fool" the discriminator, because being able to do so is the goal of the generator. However, one key detail of this stage is that the discriminator weights are not updated from that faked data. Instead the gradients from that stage are used only to update and improve the generator.
It may help if you don't think of the false labels as being "fool the discriminator", but instead they are "measure how well the generator is fooling the discriminator".


How can I test my trained network on the next unavailable hour? I have data of 695 hours. I use the first 694 hours to train the network and I use 695th hour to validate it. Now my goal is to predict the next hour.

How I can use my trained network to predict the next hour, that is, the 696th hour (which I do not have access to)?

You need to have access to the 696th hour (or successive hours), otherwise, you cannot test your model. An alternative would be, for example, to train your model on the first 693 hours, validate it on the 694th hour, and test it on the 695th hour.


Are PAC learnability and the No Free Lunch theorem contradictory? I am reading the Understanding Machine Learning book by Shalev-Shwartz and Ben-David and based on the definitions of PAC learnability and No Free Lunch Theorem, and my understanding of them it seems like they contradict themselves. I know this is not the case and I am wrong, but I just don't know what I am missing here.

So, a hypothesis class is (agnostic) PAC learnable if there exists a learner A and a function $m_{H}$ s.t. for every $\epsilon,\delta \in (0,1)$ and for every distribution $D$ over $X \times Y$, if $m \geq m_{H}$ the learner can return a hypothesis $h$, with a probability of at least $1 - \delta$
$$ L_{D}(h) \leq min_{h'\in H} L_{D}(h') + \epsilon $$

But, in layman's terms, the NFL theorem states that for prediction tasks, for every learner there exists a distribution on which the learner fails.

There needs to exists a learner that is successful (defined above) for every distribution $D$ over $X \times Y$ for a hypothesis to be PAC learnable, but according to NFL there exists a distribution for which the learner will fail, aren't these theorems contradicting themselves?

What am I missing or misinterpreting here?

There is no contradiction.
First, agnostic PAC learnable doesn't mean that the there is a good hypothesis in the hypothesis class; it just means that there is an algorithm that can probably approximately do as well as the best hypothesis in the hypothesis class.

Also, these NFL theorems have specific mathematical statements, and hypothesis classes for which they apply are often not the same as the hypothesis class for which PAC-learnability holds. For example, in Understanding Machine Learning  by Shalev-Shwartz and Ben-David, a hypothesis class is agnostic PAC learnable if and only if has finite VC dimension (Theorem 6.7). Here, the algorithm is ERM.
On the other hand, the application of the specific version of NFL that this book uses has Corollary 5.2, that the hypothesis class of all classifiers is not PAC learnable, and note that this hypothesis class has infinite VC dimension, so the Fundamental Theorem of PAC learning does not apply.

The main takeaway is that in order to learn, we need some sort of inductive bias (prior information). This can be seen in the form of measuring the complexity of the hypothesis class or using other tools in learning theory.


How should I encode a categorical input? Let's say you have an input which can take one of 10 different unique values. How would you encode it?

Have input length 10 and one-hot encode it.
Have 1 input but normalise the value between the input range.

Would the end result be the same?

As with many general questions about how to represent features, or best learn from them, the answer is "it depends".

If there is a natural sequence to the separate items, and that sequence is informative somehow for the prediction, then the classes may work best as a single feature which takes discrete values, scaled to the network. A good example might be predicting house prices where one of the features is property tax bands (in locations which have these such as UK Council Tax).
If the classes have no natural sequence to them relating to the problem, then one-hot-encoding is usually a better interpretation. An example of this might be predicting the price of a car based on the manufacturer.

In both cases, a neural network with enough layers and connections - and enough data - could resolve a more difficult representation, and make little effective difference between the representations in practice. However, if you know something useful about the feature, it is normal to choose the most "natural" representation based on that knowledge, and you will often see a small improvement by making the correct choice. In the first case, the neural network can benefit from requiring less parameters, and learn more efficiently.

If you are not sure, then choosing one-hot versus a single scaled input should be an experiment which you perform as part of your hyper parameter tuning, along with any other feature engineering that you are not sure of.


Can a character-level Seq2Seq setup learn to perfectly reconstruct structured data like name strings? If not perfect, how well can they do? For example, if I give the Seq2Seq setup a name it did not see in the training process, can it output the same name without error?

Example

name = "Will Smith"
output = DecoderRNN(EncoderRNN(name))
can_this_be_true = name == output

If you give it a name, it will probably be almost perfect model, as the number of hidden units are definitely enough to store the raw data of a name. However as a neural network it will still not be perfect. There may also be serious overfitting if you do this. A Seq2Seq model have way more parameters than necessary to just "remembers" all the possible names, as names variants are not a lot.


Have any AI systems yet been developed that can knowingly lie to / deceive a human? AI systems today are very capable machines, and recently the area of Natural Language Processing and Response has been exploding with innovation, as well as the fundamental algorithmic structure of AI machines.

I am asking if, given these recent breakthroughs, have any AI systems been developed that are able to (preferably with some measure of success) knowingly lie to humans about facts that it knows?

Note, what I'm asking goes beyond the canonical discussions of the Turing Test. I'm asking of machines that can 'understand' facts and then formulate a lie against this fact, perhaps using other facts to produce a believable 'cover-up' as part of the lie.

E.G.: CIA supercomputer is stolen by spies and they try to use the computer to do things, but the computer keeps saying it's missing dependencies though it really isn't or gives correct-looking but wrong answers knowingly. Or gives incorrect location of a person, knowing that the person frequents some place but isn't there at the moment. Doesn't have to be this sophisticated, of course.

The Saturday Papers: Would AI Lie To You? is a blog post summarizing a research paper called Toward Characters Who Observe, Tell, Misremember, and Lie. This research paper details some researchers' plans to implement "mental models" for NPCs in video games. NPCs will gather information about the world, and convey that knowledge to other people (including human players). However, they will also "misremember" that knowledge (either "mutating" that knowledge or just forgetting about it), or even lie:

As a subject of conversation gets brought up, a character may convey false information—more precisely, information that she herself does not believe—to her interlocutor. Currently, this happens probabilistically according to a character’s affinity toward the interlocutor, and the misinformation is randomly chosen.

Later on in the research paper, they detailed their future plans for lying:

Currently, lies are only stored in the knowledge of characters who receive them, but we plan to have characters who tell them also keep track of them so that they can reason about past lies when constructing subse- quent ones. While characters currently only lie about other characters, we plan to also implement self-centered lying (DePaulo 2004), e.g., characters lying about their job titles or relationships with other characters. Finally, we envision characters who discover they have been lied to revising their affinities toward the liars, or even confronting them.

The research paper also detailed how other video game developers attempted to create lying NPCs, with an emphasis on how their system differs:

TALE-SPIN characters may lie to one another (Meehan 1976, 183-84), though rather arbitrarily, as in our current system implementation. GOLEM implements a blocks world variant in which agents deceive others to achieve goals (Castelfranchi, Falcone, and De Rosis 1998), while Mouth of Truth uses a probabilistic representation of character belief to fuel agent deception in a variant of Turing’s imitation game (De Rosis et al. 2003). In Christian (2004), a deception planner injects inaccurate world state into the beliefs of a target agent so that she may unwittingly carry out actions that fulfill ulterior goals of a deceiving agent. Lastly, agents in Reis’s (2012) extension to FAtiMA employ multiple levels of theory of mind to deceive one another in the party game Werewolf. While all of the above systems showcase characters who perceive—and in some cases, deceive—other characters, none appear to support the following key components of our system: knowledge propagation and memory fallibility. ...
Like a few other systems noted above, Dwarf Fortress also features characters who autonomously lie. When a character commits a crime, she may falsely implicate someone else in a witness report to a sheriff, to protect herself or even to frame an enemy. These witness reports, however, are only seen by the player; characters don’t give false witness reports to each other. They may, however, lie about their opinions, for instance, out of fear of repercussions from criticizing a leader. Finally, Dwarf Fortress does not currently model issues of memory fallibility—Adams is wary that such phenomena would appear to arise from bugs if not artfully expressed to the player.


Why does Alpha Zero's Neural Network flip the board to be oriented towards the current player? While reading the AlphaZero paper in preparation to code my own RL algorithm to play Chess decently well, I saw that the

"The board is oriented to the perspective of the current player."

I was wondering why this is the case if there are two agents (black and white). Is it because there is only one central DCNN network used for board and move evaluation (i.e. there aren't two separate networks/policies used for the respective players - black and white) in the algorithm AlphaZero uses to generate moves?
If I were to implement a black move policy and a white move policy for the respective agents in my environment, would reflecting the board to match the perspective of the current player be necessary since theoretically the black agent should learn black's perspective of moves while the white agent should learn white's perspective of moves?

I am not an expert in RL. I have been playing Go for some years.
Let's quote from AlphaZero's paper first:

Aside from
komi, the rules of Go are also invariant to colour transposition; this knowledge is
exploited by representing the board from the perspective of the current player (see
Neural network architecture).

In the Game of Go, the difference between Black and White except the board representation is the komi (the amount of points that Black has to compensate White in the final count for playing first). Except the presence of komi, there should be no difference in strategy under the same position if colours exchanged. In other words, given a state $s$ of black stones and white stones on the board, if the optimal policy of Black playing first is $\pi$, then if colours of stones on the board exchanged and it is White's turn, the optimal policy for White should be the same as $\pi$.
With this in consideration, there are at least 2 advantages of using a network that represents the board in the perspective of Self/Opponent rather than Black/White.
The first is that it prevents the network from the possibility of giving inconsistent strategies under two representations of the same state. Consider a network $f_\theta$ that accepts the board representation in the order of $(B,W)$, and a state $s = (X_t,Y_t)$ in which $X_t$ is a feature map for black stones and $Y_t$ is a feature map for white stones and it is black's turn. Now consider a state $s' = (Y_t,X_t)$ (i.e. colours flipped) and it is white's turn. $s$ and $s'$ are essentially representation of the same state (except Komi which does not affect optimal policy). There could be a possibility that the network $f_\theta$ gives different policies for these two representations. However, if $f_\theta$ accepts the state as $(Self,Opponent)$, the input to the network would be the same (except the komi feature).
Therefore, this representation would significantly reduce number of states represented by the features vector $(X_t,Y_t)$, which would be the second advantage to training the neural network. If we consider that in Go, the same local position could appear in exchanged colour in another position, the network could by this implementation, recognize them as the same position. A decrease in the number of states could mean a significant drop in parameters and power needed from the network.
The same principle of making use of different representations of the same state is followed in AlphaGo's other training implementations as well, such as augmenting its training data to include rotations and reflections of the same board position.
However, in the game of Chess, this would be a different case. For a chess position, if the pieces' colours are exchanged and it becomes opponent's turn, it would be a different state because the positions of the KING and the QUEEN are not the same for the two colours.


Are mathematical models sufficient to create general artificial intelligence? Are mathematical models sufficient to create general artificial intelligence?

I am not sure if it is possible to represent e.g. emotions or intuition using mathematical model. Do we need a new approach in order to solve this problem?

Mathematical models are essentially highly formalised knowledge. When it comes to computer engineering, there is literally no other choice - anything you can write code for, or design a machine for, will have an associated mathematical model. That model may not be fully explored or comprehended analytically by theorists, it may be just too complex (and driven by calculations) or even mathematically intractable. However, that doesn't make it non-mathematical, just more driven by empirical results than theory.

We don't have complete models for how AGI definitely work, nor tight enough definitions of general intelligence to base maths from where we can say "if we implemented a framework based on this maths, we could build an AGI". Right now, exploration and experiments based on intuition of what might work are far ahead of such theory.

The theoretical work behind e.g. neural networks is chipping away at the problem, and there are more general over-arching theories about intelligent rational behaviour available e.g. the equations of AIXI. AIXI doesn't cover emotions and intuition directly, but does attempt to cover knowledge and how a rational agent would approach understanding the world in general from scratch. It is possible that an embodied system driven by a software implementing something like AIXI could exhibit intuition and emotions in an emergent fashion, but whether or not that would happen in practice is not at all clear from the theory. AIXI is just one theory/model out of many, and I am not qualified to analyse it in depth, but its creators have strong pedigree in AI research, so it is as good as any IMO if you are interested in starting to research AI from a theoretical perspective.

Despite all the unknowns, the success of deep learning and the loose analogies between artificial neural networks and biological ones, makes it look likely that neural networks or something like them will form a component of an AGI. The current state of the art for more narrow problem solving, learning from examples or experience through backpropagation of error gradients, might not be the most important or key component. However, whatever the structure, whether it is a small extension of existing systems, or involves some new  science, it will be describable mathematically.


Is there 1-dimensional reinforcement learning? From what I can find, reinforcement algorithms work on a grid or 2-dimensional environment.  How would I set up the problem for an approximate solution when I have a 1-dimensional signal from a light sensor.  The sensor sits some distance away from a lighthouse.  The intent would be to take the reading from the sensor to determine the orientation of the lighthouse beam.

The environment would be a lighthouse beam, the state would be the brightness seen at the sensor for a given orientation, and the agent would be the approximate brightness/orientation?  What would the reward be?  What reinforcement learning algorithm would I use to approximate the lighthouse orientation given sensor brightnesses?

From what I can find, reinforcement algorithms work on a grid or 2-dimensional environment.

A lot of teaching materials use a "grid world" presentation to demonstrate basic reinforcement learning (RL). However, the underlying Markov Decision Process (MDP) theory works on an arbitrary graph of connected states. This graph could be based on subdividing a metric space of any dimensions into a grid of the same dimensions (and using tiles of any shape that worked in that dimension). However, it is not limited to that, the state space does not need to be a metric that represents distances or physical properties.

In practice, the set of states can be arbitrary objects, connected via state transitions in any consistent way. Provided the transition probability function $p(s'|s,a)$ is consistent, the environment could be used in a RL problem.

A very common state description is that the state is a vector of numbers that capture all the variables relevant to the problem. The environment can then be measurements taken in the real world of those variables, or the same quantities provided by a simulation. That state vector can be of any size, and have arbitrary constraints on individual components. This is no different from numerical representations of other machine learning problems, such as the inputs allowed to a neural network.

  The environment would be a lighthouse beam, the state would be the brightness seen at the sensor for a given orientation, and the agent would be the approximate brightness/orientation?

Something not quite right about the description there. There does not seem to be any action that the agent takes.

  What would the reward be?

It would be whatever measure of reaching a goal or maintaining a "good" result that is appropriate for the problem. You do not give any information about the goal in your description.

If your goal is to light up a moving sensor with the highest brightness, then the brightness measured at the sensor, transformed into suitable units, would seem to be a good candidate for a reward function (you would also need the state to give information about the target - where it had been seen last for instance). Assuming the problem is continuous, you would also need a discount factor.

  What reinforcement learning algorithm would I use to approximate the lighthouse orientation given sensor brightnesses?

Generally RL algorithms estimate rewards, or generate policies. If the lighthouse orientation is the action you wanted to take, then pretty much all RL algorithms can do one or the other to allow you to do this. The differences are in things like complexity or speed of the algorithm, what approximations you are willing to take etc.

You don't give enough information about the problem to even nearly suggest a "best" algorithm. Before you start, you will need to determine a more thorough description of state, action and rewards, that will define the problem. Once you have a more formal description of the problem, that may suggest which algorithms would be good starting points.


Can I limit the possible choices for a computer vision framework to recognize? To risk giving away too much info, im building a piece of hardware with the job of  identifying the object in front of it.

If it can only be one of three different items, how can I tell the computer with simplecv?

Basically, I've found a way to limit the choices down to just a handful of potential objects, which should increase the probability of it recognizing the object correctly. Is there a way to limit the choices for the algorithm?

Me: hey raspberry pi - you see that thing in front of you?

Raspberry Pi: That thing? Ohh you mean that piece of food that might be a ham and cheese sandwich, but also kinda looks like a fish, with a slight twist of pe-

Me: - whoa okay, hold on! It's either a grilled cheese sandwich, or an apple

RPI: ohhh well that's easy! it's clearly (with 98% confidence) a grilled cheese sandwich

Any thought are appreciated!

I would recommend using a convolutional neural network. CNNs are the bleeding edge for image recognition currently. They should be able to solve your problem with a high accuracy, just maybe without the sort of dialogue that you provided in your question. In order to train a convolutional neural network, you would have to use a different computer than your desktop and "deploy" it to your raspberry pi. This means that your desktop computer would find the optimal wight configurations for your neural network and the images you want to detect, and then you would be able to send the neural network to your raspberry pi in order to be used. If you want to, you could train your entire convolutional neural network from scratch, using random weights to begin, or you could use a process known as transfer learning where you take an existing neural network, usually deep, and train it a little bit further according to your existing data set. In order to train your neural network from scratch, you will need a very large volume of sample images to train it. In order to do transfer learning, you will only need about 1000 images for each of your classes in order to achieve very accurate results for your specific task. For specific libraries that you can use, I would recommend using Matlab if you have access to it as it is extremely easy to train and run deep neural networks, and it is easy to deploy the model to your raspberry pi. You can implement transfer learning in Matlab with 10 lines of code. Next, if you do not have access to Matlab, I would recommend if you have some python experience, using tensorflow. Tensorflow supports a lot of deep learning algorithms and should be able to for your problem although it will not be as easy as using MATLAB. Finally, Microsoft has a toolkit called CNTK which is fairly similar to tensorflow, but instead you can use it in C# or other .net languages.


What is the main difference between additive rewards and discounted rewards? What is the difference between additive and discounted rewards?

I will disagree slightly with @mico. There is a usage of "additive rewards" that refers to decomposable reward functions (e.g. my reward in selling an item I do not want to own is composed of the reward of not having an unwanted item anymore, plus the monetary gain in selling the item). But, there is indeed a fundamental relation between additive and discounted rewards. Additive rewards are formulated simply as

$$
R([s_0, s_1, s_2, ...]) = R(s_0) + R(s_1) + R(s_2) + \cdots
$$

whereas discounted rewards include a discount factor $\gamma \in [0,1]$ such that

$$
R([s_0, s_1, s_2, ...]) = R(s_0) + \gamma R(s_1) + \gamma^2R(s_2) + \cdots
$$

Intuitively, the additive reward for a sequence of states is simply the sum of the rewards acquired at each state, while discounted rewards include a multiplicative discount factor that reduces the influence of rewards as time goes on. You will typically see additive rewards for finite-horizon problems, i.e. you have a discrete number of timesteps to optimize over, and discounted rewards are more relevant for infinite-horizon problems, i.e. you may need to optimize over an infinite number of timesteps (or at least a very large number). The discount factor governs the agent’s greediness in achieving immediate reward, where very small discount factors (closer to 0) encourage the agent to only seek rewards in proximal states, and very large discount factors (closer to 1) encourage the agent to think further into the future about what rewards it can expect to achieve in states it will visit later.

The most direct reference I found for this distinction is in these course slides, which are somewhat authoritative since they from Andrew Barto, a co-author of the de facto text on reinforcement learning.


Are linear approximators better suited to some tasks compared to complex neural net functions? Model based RL attempts to learn a function $f(s_{t+1}|s_t, a_t)$ representing the environment transitions, otherwise known as a model of the system. I see linear functions are still being used in model-based RL such as in robotic manipulation to learn system dynamics, and can work effectively well. (Here, I mean in learning the model, not as an optimization method for the controller selecting the best actions).
In model-based RL, are there situations where a learning a linear model such as using a Lyapunov function would be better suited than using a neural network, or are the examples of problems framed to use linear models when addressing them using model-based RL?

This is just a case of supervised learning. You are trying to predict $s_{t+1}$ given $s_t$ and $a_t$, so the answer to your question depends on how complex your state dynamics are.
For example, if the state space is really complex, e.g. if your state space is an image and you want to predict the next image given the current image and an action, then linear methods are unlikely to work well.


What reinforcement learning algorithm should I use in continuous states? I want to use reinforcement learning in an environment I made. The exact environment doesn't really matter, but it comes down to this: The amount of different states in the environment is infinite e.g. amount of ways you can put 4 cars at an intersection, but the amount of different actions is only 3 e.g. go forward, right or left. The state exists out of five numbers. My question is: what algorithm should I use or at least what kind of algorithm?

I would recommend looking at Deep Q-Learning.


Is it possible to use an RNN to predict a feature that is not an input feature? I came across RNN's a few minutes ago, which might solve a problem with sequenced data I've had for a while now.

Let's say I have a set of input features, generated every second. Corresponding with these input features, is an output feature (also available every second).
One set of input features does not carry enough data to correlate with the output feature, but a sequence of them most definitely does.

I read that RNN's can have node connections along sequences of inputs, which is exactly what I need, but almost all implementations/explanations show prediction of the next word or number in a text-sentence or in a sequence of numbers.

They predict what would be the next input value, the one that completes the sequence. However, in my case, the output feature will only be available during training. During inference, it will only have the input features available.

Is it possible to use RNN in this case? Can it also predict features that are not part of the input features?

Thanks in advance!

Is it possible to use RNN in this case? Can it also predict features that are not part of the input features?

Yes.

No changes are required to a RNN in order to do this. All you need is correctly labelled data mapping a sequence of $x$ to correct $y$ in order to train, and of course a RNN architecture which has input vectors matching shape of $x$ and output vectors matching shape of $y$. The case where $x$ and $y$ are the same data type is just a special case of RNN design, and not a requirement.

You may need to consider some details:

If the relationship between $x$ and $y$ is complex and non-linear even accounting for accumulated hidden state during the sequence, you may need to add deeper layers. The output of the LSTM can be some vector $h$ and you can add fully-connected layers to help with predicting $y$ from $h$. This, or adding more LSTM layers, is a choice of hyperparameter that you may want to experiment with. Start with a basic LSTM to see how that goes first.
If you wish to predict a sequence of output features that is either not the same length as the input feature sequence, or logically should come after the whole sequence (think language translation) then this may need a slight change in setup to get best results. For a predict-same-kind sequence you can feed your predicted output value into the next input, but if input and output have different data types, this will not work. Instead, you will need to have some dummy input or other setup for creating sequences of $y$.

In your specific case the second point does not seem to apply, as you want to predict a single $y$ immediately after a sequence of $x$.


What is the role of biology in AI? Biology is used in AI terminology. What are the reasons? What does biology have to do with AI? For instance, why is the genetic algorithm used in AI? Does it fully belong to biology?

Biological organisms (such as animals or plants) are the main examples of intelligent systems that we are aware of (excluding artificially intelligent systems, so as not to discuss whether current AI systems are really intelligent or not). Consequently, biological life is often an inspiration for AI researchers to develop AI systems.
There are numerous examples of AI systems that have been introduced  (at least, partially) based on or just inspired by the biology. Here are a few examples.

Reinforcement learning is based on a similar way that animals (such as dogs or pigeons) can learn. For more details, see Sutton & Barto's book (especially chapters 14 and 15).

Artificial neural networks are very approximative models of human neural networks.

Genetic algorithms are roughly based on Darwin's theory of evolution.

Ant colony optimization algorithms (and, in general, swarm intelligence) are based on the way real ants (and, respectively, biological swarms) behave. (There is even a rap song dedicated to ants).

There are probably other examples that don't come to my mind right now. See also this and this questions.
There are cases where AI discoveries have also helped the development of biology or related fields (such as neuroscience and psychology). For instance, Sutton & Barto (on page 4) write

Of all the forms of machine learning, reinforcement learning is the closest to the kind of learning that humans and other animals do, and many of the core algorithms of reinforcement learning were originally inspired by biological learning systems. Reinforcement learning has also given back, both through a psychological model of animal learning that better matches some of the empirical data, and through an influential model of parts of the brain's reward system.


Is there any disadvantage of the maximum number of fitness function call as a stop criterion? I'm studying different stop criteria in genetic algorithms and the advantages and disadvantages of each of them for evaluating different algorithms. One of these methods is the max number of fitness function calls (max NFFC), so that we define a value for max NFFC and, if the number of fitness function calls reached this value, the algorithm will stop. Fitness function is called for calculating the fitness of the initial population and whenever a crossover or mutation happens (if parents are chosen as offspring there is no need to compute fitness function).
I searched if there is a disadvantage or limitation about using this stop criterion, but I didn't find anything. So, I wanted to know if applying this stop criterion in my algorithm has any disadvantages or there is nothing wrong with using this criterion.

I imagine that using the MaxNFFC as a stop criterion only happens in very particular implementations. And this is its main disadvantage.
Normally, you'd evaluate each individual, each generation. So, NFFC will always be the same as the size of the population times the number of epochs (+1 to consider the initial population).
$$N*(E+1)$$
As the population size is generally constant, MaxNFFC looks a lot like an MaxEpoch stop.
$$ \frac{\text{NFFC}}{N} = E + 1 $$
So, it seems that it might be used in scenarios where:

the population size is not constant (in this case you want to make sure that at least K individuals have been evaluated before stopping).
not all individuals are evaluated each epoch. There might be a high abortive rate (individuals with invalid genomes that are kept in the population but not considered). In this case, as above, you want to perform a minimum amount of evaluations.

I wouldn't see the usage of MaxNFFC in other cases.