Understanding the update rule for the policy in the policy iteration algorithm Consider the grid world problem in RL. Formally, policy in RL is defined as $\pi(a|s)$. If we are solving grid world by policy iteration then the following pseudocode is used:

My question is related to the policy improvement step. Specifically, I am trying to understand the following update rule.

$$\pi(s) \leftarrow arg max_a \sum_{s'}p(s'|s,a)[r(s,a,s') + \gamma v(s')] $$

I can have 2 interpretations of this update rule.

In this step, we check which action (say, going right for a particular state) has the highest reward and assign going right a probability of 1 and rest actions a probability of 0. Thus, in PE step, we will always go right for all iterations for that state even if it might not be the most rewarding function after certain iterations.
We keep the policy improvement step in mind, and, while doing the PE step, we update the $v(s)$, based on the action giving highest reward (say, for 1st iteration $k=0$, going right gives the highest reward, we update based on that, while, for $k=1$, we see going left gives the highest reward, and update our value based on that likewise. Thus action changes depending on maximum reward).

For me, the second interpretation is very similar to value iteration. So, which one is the correct interpretation of a policy iteration?

A policy can be stochastic or deterministic. A deterministic policy is a function of the form $\pi_{\text{deterministic}}: S \rightarrow A$, that is, a function from the set of states to the set of actions. A stochastic policy is a map of the form $\pi_{\text{stochastic}} : S \rightarrow P(A)$, where $P(A)$ is a set of probability distributions ($P(A) = \{ p_{s_1}(A), p_{s_2}(A), \dots, p_{s_{|S|}}(A) \}$, where $p_{s_i}(A)$ is a probability distribution over the set of actions $A$ for the state $s_i$ and $|S|$ is the size of the set of states of the environment) over the set of actions $A$. A deterministic policy can be interpreted as a stochastic policy that gives the probability of $1$ to one of the available actions (and $0$ to the remaining actions), for each state.

In the case of value iteration (VI) and policy iteration (PI), the policy is deterministic, both in the policy evaluation (PE) or policy improvement (PI) steps.

In the PE step, for a specific state $s$ (inside the for loop), you use $\pi(s)$, that is, you assume that the action taken in the specific state $s$ is the greedy action, because of the $\text{argmax}$ in the PI, according to the current policy. However, in general, $\pi_k(s_i) \neq \pi_k(s_j)$, for $i \neq j$, where $k$ is the current iteration step (of PE and PI).

The update rule

$$\pi_{k+1}(s) \leftarrow arg max_a \sum_{s'}p(s' \mid s, a)[r(s, a, s') + \gamma v_k(s')]$$

computes the action that is expected to give the highest return, given the current and fixed value function $(v_k$). Have a look at the definition of expectation for discrete random variables: it is defined as a weighted sum (like in the update rule above, where the weights are $p(s' \mid s, a)$).

Value iteration is a shorter version of policy iteration. In VI, rather than performing a PI step for each state of the environment, when performing PE, the action that is expected to give the highest return is chosen. In VI, the policy is updated only once and at the end of policy evaluation. In PI, you alternate between PE and PI, and, at each PI, you update the policy.


Multi class text classification when having only one sample for classes I have a dataset of texts, each text was identified with an ID number. I would like to do a prediction by finding the best match ID number for upcoming new texts. To use multi text classification, I am not sure if this is the right approach since there is only one text for most of ID numbers. In this case, I wouldn't have any test set. Can up-sampling help? Or is there any other approach than classification for such a problem?
The data set looks like this:
id1 'text1', id2 'text2', id3 'text3', id3 'text4', id3 'text5', id4 'text6', . . id200 'text170'
I would appreciate any guidance to find the best approach for this problem.

Siamese networks may be useful in your case.
http://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf
https://en.wikipedia.org/wiki/Siamese_neural_network
https://link.springer.com/protocol/10.1007%2F978-1-0716-0826-5_3


Which hyperparameter does Neural Architecture search (NAS) use? i have read a lot about NAS but I still do not understand one concept:
When setting up a neural network, hyperparameters need to set up, like for example the learning rate, dropout rate, batch size, filter size... etc.

In NAS only the best architecture is decided, e.g. how many layers and neurons.. .
But what about the hyperparameters? Are they randomly decided?

Thank you for your help

Great question! It's not clearly stated (it's not stated at all on wikipedia), but after a bit of searching, I found an answer here about a third of the way down the page:

  The best performing architecture observed during the training of the controller is taken, and a grid search is performed over some basic hyperparameters such as learning rate and weight decay in order to achieve near STOTA (state of the art) performance.

So as a direct answer: The norm; A grid search.


How does a model based agent learn the model? I want to build model-based RL. I am wondering about the process of building the model.
If I already have data, from real experience:

$S_1, a \rightarrow R,S_2$
$S_2, a \rightarrow R,S_3$

Can I use this information, to build model-based RL? Or it is necessary that the agent directly interact with the environment (I mean the same above-mentioned data should be provided by the agent)?

If you already have some transition tuples then you can train a model to predict environment dynamics using these. However, you should be careful that your pre-gathered data is diverse enough to 'cover' enough of the state/action space so that your model remains accurate. For instance, when you start training your agent it will likely start to see more of the state space than it did at the start of training (imagine playing Atari, initially your agent will die quickly but as it gets better episodes will get longer) so you would need to make sure you have data for these states that appear late in episodes, otherwise your model will just be overfitting to the start of the episode and will give a poor performance on these other states, thus slowing down or even prohibiting learning of an optimal policy.


What happens if the opponent doesn't play optimally in minimax? I just read an article about the minimax algorithm. When you design the algorithm, you assume that your opponent is a perfect player, i.e. it plays optimally.

Let's consider the game of chess. What happens if the opponent plays irrationally or sub-optimally? Do you still have a guarantee that you are going to win?

What happens if the opponent plays irrationally or sub-optimally? Do you still have a guarantee that you are going to win?

If your search is deep enough to guarantee optimal play in all cases, then yes. Optimal play is such that the opponent's decision causes least impact to your agent. In fact, if the opponent makes a mistake, often that can make the search easier/faster, and the agent will win more convincingly.

What it may mean is that optimisations you may have taken - e.g. pruning game tree segments that lead to non-optimal decisions by either player - might not be as useful. This might impact decision time if you keep some partial game tree or cache branch evaluations between moves to help speed up the agent.

Actual optimal agents for games as complex as chess are not possible. In these games, you will not have a truly optimal agent, but approximately optimal. You will be relying on some heuristic to guide the minimax search when it cannot force an end game win. If the opponent manages to control play into a state where the heuristics are not accurate, they could cause minimax search to fail and misdirect your agent into making mistakes.

A combination of effects is also possible if you have implemented a tree caching mechanism for performance improvements and made the system playable by limiting planning time - e.g. you limit computer search time to 3 seconds max - an irrational opponent may cause your agent's performance to degrade to the point where it too starts to make mistakes. Whether or not this is enough for a smart opponent to take advantage of it and beat an agent which is capable of otherwise playing a "perfect" game depends on details of the game, and how narrow the agent's measure of "perfect" is.

An extreme case might be an agent that has memorised a single perfect game (by scoring high heuristics for any state on a single tarjectory through the game that has been pre-calculated for perfect moves by both players), and has poor heuristics otherwise - once state moves away from what it can evaluate directly, the agent will be limited by how well it can search for completed game wins, and can easily be manipulated by a more generally smart opponent into a losing position that is beyond the depth of its search.

In practice, well-coded agents will not suffer too much from this effect. If you make a mistake, or try a random probably bad move in an attempt to confuse the agent when playing against Stockfish, you will lose.


Is unsupervised learning a branch of AI? From Artificial Intelligence: A Modern Approach, a book by Stuart Russell and Peter Norvig, this is the definition of AI:

  We define AI as the study of agents that receive percepts from the environment and perform actions. Each such agent implements a function that maps percept sequences to actions, and we cover different ways to represent these functions, such as reactive agents, real-time planners, and decision-theoretic systems. We explain the role of learning as extending the reach of the designer into unknown environments, and we show how that role constrains agent design, favoring explicit knowledge representation and reasoning.

Given the definition of AI above, is unsupervised learning (e.g. clustering) a branch of AI? I think the definition above is more suitable for supervised or reinforcement learning.

There is a problem with confining Artificial Intelligence to a single definition, because it has become an umbrella term encompassing many fields of science. It has come a long way from the "thinking machines" of the 50s. Actually, the term was coined in a summer workshop in 1956, whose proposal was:

  The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.

So even from its very first introduction the field of AI included learning. Personally, I've never seen a definition of AI, that doesn't include learning, of which unsupervised learning is clearly a part. So by any means clustering algorithms, which are unsupervised learning algorithms, can be considered AI. This can be confirmed in multiple sources, e.g. wikipedia.


Is it possible to predict $x^2$, $\log(x)$, or variable function of $x$ using RNN? There were some posts that using RNN can predict the next point of the sine wave function with data history.
However, I wondered if it also works on all the functions of $x$, such as $x^2$, $x^3$, $\log(x)$, $\frac{1}{(x+1)}$ functions.

Yes, RNN can work on the functions you have mentioned. In fact, neural networks can approximate anything (Universal Approximation Theorem). This question also reminds me of Neural Turing Machine.
But, it would be a complete waste to use RNNs or NNs for such a task.


Isn't evolutionary theory the essence of intelligence after all? The theory of evolution seems to be intelligent as it creates life
The mechanism of evolutionary theory consists of mutation, recombination, and natural selection like a genetic algorithm.
Isn't this evolutionary mechanism itself the same as the essence of human intelligence?

The theory of evolution seems to be intelligent as it creates life

When you say "seems to be intelligent" that begs the question: How are you defining "intelligent"? Which of course is still one of the big issues in AI research.
I think there are some flaws with the argument that "creates life" = "intelligent":

Evolution does not create life. It operates on entities where there is a copy mechanism which is not 100% reliable, plus a selective environment that also impacts likelihood of further copies being made. Some form of proto-life (an initial Darwinian ancestor or Ida) needed to exist before evolution started.

The process of creating the first proto-life capable of undergoing evolution is generally thought to be a large semi-random search through chemical combinations. Random search is sometimes used in optimisation problems, and might be studied as part of AI search topics. However, it would normally be considered something of a baseline algorithm, and definitley not tick the boxes for all the general traits of intelligence.

Isn't this evolutionary mechanism itself the same as the essence of human intelligence?

The Wikipedia article on artificial intellegence lists challenges faced by researchers and developers in AI. The categories chosen there are:

Reasoning, problem solving; Knowledge representation; Planning; Learning; Natural language processing; Perception; Motion and manipulation; Social intelligence; General intelligence

Together, these are mainly traits of mammalian, avian and a few other multicellular species, with a few traits such as language heavily focused on humans.
I think it is important to separate out the mechanism whereby these traits arose naturally - which is generally agreed to be via an evolutionary process - from how those traits function. Artifical intelligence may use a little bit of reverse engineering from the natural traits in order to inspire design, but most AI systems do not use theory of evolution directly.
When used directly, evolutionary algorithms can be used to solve search and optimisation problems. Also they can be used to solve simplified problems in perception and motion/manipulation. However, we are not able to scale up such algorithms to solve all aspects of general intelligence. Instead, systems like machine learning are designed to work from analysis of the problem, inspired in part by working natural systems. These work far more efficiently than evolutionary algorithms. There are no competitive evolutionary variants of AlphaZero, Watson, GPT-3 or neural-networks used in image processing.
Evolutionary algorithms have their place in AI in practice and research. However, they do not define or encapsulate a form of general intelligence.


Integration of Sentiment analysis in CRM What is the process for integrating sentiment analysis in a CRM? What I am searching for is a system which analyzes the customer comments or reviews using the CRM and finds out the customer sentiment on the services provided by the system or company or a product.

I have done a sentiment analyzer which takes text and shows the sentiment of the text. Now I want to integrate the above-mentioned sentiment analyzer to a CRM, how can I do that?

I got one paper on the integration of opinion mining in CRM, which helps to get an idea about the process involved.


Ambulance dataset needed Could I get a dataset that can classify ambulances?
I have searched everywhere, but, couldn't seem to get hold of a set of annotated images for ambulances.

Answering my own question here.

Looked at the Open Image Dataset by Google @ https://storage.googleapis.com/openimages/web/index.html

They provide image-level labels, object bounding boxes, object segmentation masks, and visual relationships.


What if the more fit parent has fewer nodes compared to the other, will the disjoint and excess genes be discarded? In the paper Efficient Evolution of Neural Network Topologies (2002), the authors say

  Genes that do not match are inherited from the more fit parent

What if the more fit parent has fewer nodes compared to the other, will the disjoint/excess genes be discarded?

When crossover happens and one parent is fitter than the other, the nodes from the more fit parent are carried over to the child. This is the case as disjoint and excess genes are only carried over from the fittest parent.
Here's an example:

// Node Crossover
Parent 1 Nodes: {[0][1][2]} // more fit parent
Parent 2 Nodes: {[0][1][2][3]}

Child Nodes:    {[0][1][2]} // after crossover

Gene information is also passed to the child during crossover. Matching genes (those that have the same innovation number) are chosen at random and passed to the child. The disjoint and excess genes are choose from the more fit parent.

// Gene Crossover
Parent 1 Genes: [1][2][3]      [6]   [8][9][10] // more fit parent
Parent 2 Genes: [1][2][3][4][5]   [7]

Child Genes: [1][2][3][6][8][9][10] // after crossover

As you can see, the gene innovation numbers in the child match up with the innovation numbers of the fittest parent. However the gene information from matching genes (in the example genes 1, 2 and 3 match) have an equal chance of being carried over to the child. In the example, the child's first three genes could have come from either parent.


How can an AI make tactical manoeuvres? I'm working on a multi-player game that involves a board/map where there are walls, cover or open ground. The players then take turns moving units around and undertaking discrete actions, like move, fire etc.

I've been exploring adding an AI for single-player games, or to enhance multi-player games. I've looked at:

Behaviour Trees
Utility-based decisions
GOAP

The problem I see with all of these is that in order for the AI to use specific multi-unit tactics, like a pincer attack, or leap-frogging units; that behaviour needs to be explicitly added separate to the above.

So, how can a Behaviour Tree tell an AI how to identify it should use a Flanking manoeuvre, i.e. move a unit around to hit from the side? How does the AI identify that it's a good plan, that is action across multiple "moves"? How can the AI identify where to move the individual units in order to pull this off (how does it identify the flank)?

Reinforcement learning is the best fit for problems like these. To get started explore how DeepMind solved Atari games. The key is self play. RL works through trial and error and eventually figures out the best strategy. You also need to tweak so that it does not play at God level making it impossible for users to win.


In the multi-linear regression, how is the value of weight $b_2$ calculated? In multivariate linear regression (linear regression with more than one variable) the model is $yi = b_0 + b_1x_{1i} + b_2x_{2i} + ...$ , and so on. But how is the $b_n$ value calculated iteratively? Can it be calculated non-iteratively? What is the intuition behind using that method to calculate $b_2$?

It is calculated the same way $b_1$ is calculated.

Nearly following your notation, say your multiple linear regression function is

$H(X_i) =  b_0 + b_1x_{1,i} + ...+ b_nx_{n, i}$

for data instance $X_i=x_{1,i},...,x_{n, i}$
and weights $b_0,...,b_n$.

And say your error function is $E(X,Y) = \sum_i(H(X_i)-Y_i)^2$

where $X$ is the collection of all data points $X_i, Y_i$.

From your error function $E$, for whatever weights you have (with a gradient based method), calculate the partial derivative
$\partial E /\partial b_i$ and use this to update all of your weights at once in each iteration of your optimization routine.


Is it possible to use k-nearest neighbour for classification with more than two attributes? If I were to have a dataset of 9 attributes of different types that describe current weather, such as temperature, humidity, etc., and want to classify the current weather by use of a k-NN algorithm, is this possible?
From what I understand, k-NN has two different attributes that are plotted, and, wherever a point is drawn, its nearest neighbors will classify it.
Could I do the same thing but each data point is placed based on its 9 attributes?

The number of features is not important to use K-NN algotihm. You have to decide distance measure to detect neighbors. I share with you some links that you can check to see which kinds of distance measures that you can use. Just decide the meause and use your feature vectors in the measure.
https://www.kdnuggets.com/2020/11/most-popular-distance-metrics-knn.html
https://medium.com/@luigi.fiori.lf0303/distance-metrics-and-k-nearest-neighbor-knn-1b840969c0f4


Strategic planning and multi dimensional knapsack problem I'm trying to find a planning approach to solve a problem that attempts to model learning of new material. We assume that we only have one resource such as Wikipedia, which contains a list of articles represented as a vector of knowledge it contains and an effort to read that article.

Knowledge vector and effort

Before we start, we set a size for the vector, depending on the number of different types of knowledge. For example, we can define the items in the vector to be (algebra, geometry, dark ages), and then 'measure' all the articles from this point of view. So, a math article will probably be (5,7,0), since it will talk a lot about algebra and geometry but not about the dark ages. It will also have an effort to read it, which is simply an integer.

Problem

Given all the articles (represented as knowledge vectors with an effort), we want to find the optimal set of articles that help us to reach a knowledge goal (also represented as a vector).

So, a knowledge goal can be (4,4,0), and it's enough to read an article (2,1,0) and (2,3,0), since, when added, it adds up to the knowledge goal. We want to do this with minimal effort.

Question

I've tried to some heuristics to find an approximation, but I was wondering if there is any state of the art strategic planning method that can be used instead?

Here is a speculative cast of the problem to a travelling salesman problem, which would lead to shortest-path algorithms.

Please note this idea suggests different constraints to explore.

Given the knowledge vectors and efforts, build a acyclic directed graph (acyclic, as we are not supposed to unlearn). A vertex is an article, represented by its knowledge vector. An edge links two articles, weighted by the effort to "move" to the target article/vertex (i.e. acquire the knowledge of that article).
Assign a zero vector to a new participant. That is the starting point on the graph is vertex V0 = (0, ..., 0).
Define a learning objective as a vector V.
Use a shortest-path algorithm to find a (V0, V) plan.

This procedure is insufficient, as there are many ways to build the graph (in other words, the above is completely pointless as is). Extra constraints are necessary to make it practical. For example, we can order the vertices by ordering them along each dimension. Such setting would lead learners to start with "easy" articles (V[i] is low), and move step by step toward more complex topics ((V[i] gets higher).

The graph construction depends on the data available. For example, are knowledge vectors "absolute", or can they be relative? Relative can help in creating a path, as moving from V to W requires an effort that depends on your learner's initial conditions (V0 may not be 0 everywhere, afterall).

Is it an AI question? Definitely.


Oscillating around the saddle point in gradient descent? I was reading a blog post that talked about the problem of the saddle point in training.

  In the post, it says if the loss function is flatter in the direction of x (local minima here) compared to y at the saddle point, gradient descent will oscillate to and from the y direction. This gives an illusion of converging to a minima. Why is this?

  Wouldn’t it continue down in the y direction and hence escape the saddle point?

Link to post:
https://blog.paperspace.com/intro-to-optimization-in-deep-learning-gradient-descent/

Please go to Challenges with Gradient Descent #2: Saddle Points.

It's important to note that in a pragmatic instance of ML on a "real dataset", you likely wouldn't have a "strict" saddle point with precisely zero gradient. Your error surface won't be "smooth", so even though you would have something that resembles a saddle point, what you would obtain would in fact be a small region with near-zero gradient.

So let's say you are in a region with near-zero gradient. Assuming that the gradient is this area is normalized at 0 with small Gaussian distributed noise (thus gradient =  small Gaussian noise). You can then see that the algorithm can't quite escape the region (or at least, will spend a lot of time here) since 1. Gaussian random walks will more-or-less stay in place (unless for a long time) 2. small gradients means there is no obvious direction to leave the region.

In any case, SGD more or less solves this issue, and its usage is standard practice for reasons beyond this problem.


Is there any good app for iOS or Android to create, train and run neural networks? Is there any good app for iOS or Android to create, train and run neural networks? I know there is NeuralMesh for the web, but I want something similar offline.

You can use Neuroph to develop and train your network and add in app via NetBeans.

check this link

Creating Android image recognition application using NetBeans and Neuroph.


Why is depth-first search an artificial intelligence algorithm? I'm new to the artificial intelligence field. In our first chapters, there is one topic called "problem-solving by searching". After searching for it on the internet, I found the depth-first search algorithm. The algorithm is easy to understand, but no one explains why this algorithm is included in the artificial intelligence study.
Where do we use it? What makes it an artificial intelligence algorithm? Is every search algorithm is an AI algorithm?

This is a fundamentally a philosophical question. What makes AI AI? But first things, why would DFS be considered an AI algorithm?
In its most basic form, DFS is a very general algorithm that is applied to wildly different categories of problems: topological sorting, finding all the connected components in a graph, etc. It may be also used for searching. For instance, you could use DFS for finding a path in a 2D maze (although not necessarily the shortest one). Or you could use it to navigate through more abstract state spaces (e.g. between configuration of chess or in the towers of Hanoi). And this is where the connection to AI arises. DFS can be used on its own for navigating such spaces, or as a basic subroutine for more complex algorithms. I believe that in the book Artificial Intelligence: A Modern Approach (which you may be reading at the moment) they introduce DFS and Breadth-First Search this way, as a first milestone before reaching more complex algorithms like A*.
Now, you may be wondering why such search algorithms should be considered AI. Here, I'm speculating, but maybe the source of the confusion comes from the fact that DFS does not learn anything. This is a common misconception among new AI practitioners. Not every AI technique has to revolve around learning. In other words, AI != Machine Learning. ML is one of the many subfields within AI. In fact, early AI (around the 50s-60s) was more about logic reasoning than it was about learning.
AI is about making an artificial system behave "intelligently" in a given setting, whatever it takes to reach that intelligent behavior. If what it takes is applying well-known algorithms from computer science like DFS, then so be it. Now, what is it that intelligent means? This is where we enter more philosophical grounds. My interpretation is that "intelligence" is a broad term to define the large set of techniques that we use to approach the immense complexity that reality and certain puzzle-like problems have to offer. Often, "intelligent behavior" revolves around heuristics and proxy methods away from the perfect, provable algorithms that work elsewhere in computer science. While certain algorithms (like DFS or A*) may be proven to give optimal answers if infinitely many resources can be devoted to the task at hand, only in sufficiently constrained settings would such techniques be affordable. Fortunately, we can make them work in many situations (like A* for chess or for robot navigation, or Monte Carlo Tree Search for Go), but only if reasonable assumptions and constraints over the state space are imposed. For all the rest is where learning techniques (like Markov Random Fields for image segmentation, or Neural Nets paired with Reinforcement Learning for situated agents) may come handy.
Funny enough, even if intelligence is often regarded as a good thing, my interpretation can be summed up as imperfect modes of behavior to address immensely complex problems for which no known perfect solution exists (with rare exceptions in sufficiently bounded problems). If we had a huge table that, for each chess position, gives the best possible move you can make, and put that table inside a program, would this program be intelligent? Maybe you'd think so, but in any case it seems more arguable than a program that makes real-time reasoning and spits a decision after some reasonable time, even if it's not the best one. Similarly, do you consider sorting algorithms intelligent? Again, the answer is arguable, but the fact is that algorithms exist with optimal time and memory complexities, we know that we can't do better than what those algorithms do, and we do not have to resort to any heuristic or any learning to do better (disclaimer: I haven't actually checked if there's some madman out in the wild applying learning to solve sorting with better average times).


What is the time complexity of the value iteration algorithm? Recently, I have come across the information (lecture 8 and 9 about MDPs of this UC Berkeley AI course) that the time complexity for each iteration of the value iteration algorithm is $\mathcal{O}(|S|^{2}|A|)$, where $|S|$ is the number of states and $|A|$ the number of actions.

Here is the equation for each iteration:

$$
V_{k+1}(s) \gets \max_a \sum_{s'} T(s, a, s') [R(s, a, s') + \gamma V_k(s')]
$$

I could't understand why the time complexity is $\mathcal{O}(|S|^{2}|A|)$. I searched the internet, but I didn't find any good explanation.

The update equation for value iteration that you show is time complexity $O(|\mathcal{S}\times\mathcal{A}|)$ for each update to a single $V(s)$ estimate, because it iterates over all actions to perform $\text{max}_a$ and over all next states for $\sum_{s'}$.

The sources you have found are probably counting an entire sweep through the state space as an "iteration" i.e. $\forall \space s \in \mathcal{S}: V_{k+1}(s) \leftarrow \text{max}_a \sum_{s'} T...$ That adds another factor of $|\mathcal{S}|$ making the overall complexity $O(|\mathcal{S}\times\mathcal{S}\times\mathcal{A}|)$ or $O(|\mathcal{S}|^2|\mathcal{A}|)$

This definition of iteration makes sense, as the basic value iteration algorithm is required to sweep through the whole state space in order to converge. This also matches the standard test for convergence, which is made after each full sweep, and checks what the largest absolute update was at the end of the sweep - if it is below some target value for accuracy then the process is declared complete.


What kind of deep learning model does latest version of AlphaFold use for protein folding problem? I understand there are multiple versions used in AlphaFold. What kind of deep learning model does the more advanced version use? CNN, RNN, or something else?
(Additionally, is there an open-source reference model for the protein folding problem?)

The accurate answer is 'we don't know for sure'. But that said, the Deepmind article about AlphaFold 2, which won the CASP 14 competition, says this:

A folded protein can be thought of as a “spatial graph”, where residues are the nodes and edges connect the residues in close proximity. This graph is important for understanding the physical interactions within proteins, as well as their evolutionary history. For the latest version of AlphaFold, used at CASP14, we created an attention-based neural network system, trained end-to-end, that attempts to interpret the structure of this graph, while reasoning over the implicit graph that it’s building. It uses evolutionarily related sequences, multiple sequence alignment (MSA), and a representation of amino acid residue pairs to refine this graph.

A reasonable assumption that can be formed from this paragraph is that it is using some kind of a Transformer-based architecture. Yannic Kilcher's review video on AlphaFold discusses this assumption, along with a detailed explanation of the previous version of AlphaFold.
The source code for this previous version of AlphaFold, the one that was used in CASP 13, seems to have been open-sourced by Deepmind in their Github repository:
https://github.com/deepmind/deepmind-research/tree/master/alphafold_casp13


What difference does it make whether Actor and Critic share the same network or not? I'm learning about Actor-Critic reinforcement learning algorithms. One source I encountered mentioned that Actor and Critic can either share one network (but use different output layers) or they can use two completely separate networks. In this video he mentions that using two separate networks works for simpler problems, such as Mountain Car. However, more complex problems like Lunar Lander works better with a shared network. Why is that? Could you explain what difference that choosing one design over another would that make?

One can expect the optimal high-level features required to choose the next action and to evaluate a state to be quite similar. Because of that, it is a reasonable idea to share the same network for both policy and value function – you are essentially parameter sharing the feature-extraction part of your neural network, and fine tuning the different heads of your network on the two different tasks: action choice and value prediction.
Using two vs one networks is mostly a question of sample efficiency: theoretically, in both case your AC algorithm should work. In practice however, it will usually be useful to have parameter sharing as the representations encouraged by one of the tasks might be highly useful for the other and vice-versa, enabling one task to cause the other task to get unstuck from local optima. Another reason why this might work better is simply because you do not have to learn the same (or at least similar) representations from scratch twice – leading to a more sample efficient training.


How should I weight the factors that affect the choice of an action in a strategy board game with multiple actions? I have written an AI that plays a strategy board game. There are lots of different types of moves (e.g. attack, defend, help ally colony, etc.).

I calculate the best moves to do depending on a variety of factors, such as the value of nearby enemy colonies, the number of armies the colony currently has, etc (each of these has separate weightings). I'm trying to find the optimal weighting for each of the different factors.

Currently, I decide the best configuration of parameters in a King of the Hill style tournament. I choose random values between a suitable range for each of the different parameters and then play two of these AI against each other 20 times. I have a total of 100 AI that play against the king, and then take the final king as the best AI.

The problem is that this is quite slow and I feel it's very inefficient, as a lot of the AI don't play well at all (probably due to the randomness of parameter values).

I'm wondering if there's a more efficient way to determine the optimal value of parameters?

You could use a genetic algorithm to optimise the parameter settings. Here you don't choose random parameters all the time, but only at the beginning. Each AI (which is a vector of parameter settings) plays each other one for a ranking (you can probably reduce the number of total games by using a ladder-style ranking where only neighbours play against each other). Also, you don't need 100 to start with, as you might come across better combinations than those present initially throughout the processing.

Then you discard the worst AIs (ie parameter vectors), and recombine the best ones, adding some random mutations into the mix. In theory this should converge faster, as you preserve good parameter settings (depending on how interdependent they are) and remove bad ones from the 'pool'.

You can also have a higher rate of mutations initially, which slowly goes down as you progress through the generations.


How to create and train (with mutation and selection) a neural network to predict the next state of a board? I'm aiming to create a neural network that can learn to predict the next state of a board using the rules of Conway's Game of Life.
Technically, I have three questions, but I felt that they needed to be together to get the full picture.
My network will look at each cell individually (to reduce the computing power needed and to increase learning speed) and its surrounding cells. 9 input nodes for the network will go into one hidden layer. The output layer will be one node for the state of that cell in the next state of the game.
Nodes have two states (alive and dead) and connections between nodes can either transfer that value or invert it.
For the learning part, I was going to make use of mutation and natural selection. The starting network will have the input and output layers with no hidden layer and no connections. My idea was then to introduce mutation by randomly generating a number of new networks by adding nodes and randomly connecting them to inputs and to the output. The number of nodes in the middle layer will be limited to 512, since there are 512 possible inputs; however, I may reduce this if it is too slow.
Should I also have it randomly delete nodes and connections, in case they also make improvements?
Each network will be tested on the same board state, and their accuracy will be calculated by comparing their output to a correct output generated by a computer program. The most accurate network will then be used for the next generation.
My issue is that I don't know how to program the nodes. Should the nodes in the hidden layer perform a logical AND on all of their inputs or an OR?
I know that the network won't learn the rules within the first few turns, but how do I know if it will ever get above 90% accuracy, or even just 50%?

Re: Should I also have it randomly delete nodes and connections, in case they also make improvements?
You may read about "dropout". In this case it's not preferred since overfitting is actually a benefit.

Re: Should the nodes in the hidden layer perform a logical AND on all of their inputs or an OR?
There are no logical operations in Neural Networks. You need to multiply the input by the weights. At the end you will arrive to a fraction number that you need to convert back to 1 or 0, here I suggest to look into "softmax" that is an activation function exactly for this.

Re:how do I know if it will ever get above 90% accuracy, or even just 50%?
As Jaden suggested you will need a benchmark. You may simply take my results from your other post. (i.e 100% in 20 epochs)

I agree with Jaden, you have a lot on your table. Perhaps it would make sense to first develop a DNN to learn the GOL, you will need the activations, the loss calculation and at least one optimizer. Once this is tested and working you could start with the second model for the genetic algorithm that will need a different loss function and also the genetic operations, population selection etc. Nice big project, good luck!


What are "function terms" in the context of an ontology? I was going through the Wikipedia page on ontology components and noticed something that I had been hoping to find, for a long time. In the components' overview it mentioned:

  Function terms: complex structures formed from certain relations that can be used in place of an individual term in a statement

I tried digging deeper into function terms but couldn't find any other references. Is this a standard component of an ontology?

References would be nice. Ultimately, I'm looking for something like this in the DBPedia ontology. Would have been nice if I could add ontology and dbpedia as tags to the question, but I don't have the requisite reputation.

If you are looking for the functional verbs used in VA/VE (value analysis / value engineering) look at:

https://www.valueanalysis.ca/functionanalysis.php
http://www.brcommunity.com/articles.php?id=b679
https://www.washoeschools.net/cms/lib08/NV01912265/Centricity/Domain/253/Social%20Studies/Marker%20Verbs.pdf


Using GAN's to generate dataset for CNN training I'm doing bachaleor thesis on traffic sign detection using single shot detector called YOLO. These single shot detectors can perform detection of objects in image and so they have specific way of training, ie. training on full images. Thats quite problem for me, because the biggest real dataset with full traffic sign images is Belgian one with 9000 images in 210 classes, which is unfortunately not enough to train good detector.

To overcome this problem, I've created DatasetGenerator, which does quite good job in generating synthetic datasets, you can see in the results directory.

Recently I came across GAN's which can (besides others) generate or extend existing dataset and I would like to use these networks to compare with my dataset generator. I've tried this introduction to GANs succesfully.

The problem is it's unsupervised learning and so there are no annotations. It means it's able to extend my dataset of traffic signs, but the generated dataset won't be annotated at all, which is problem.

So my question is: Is there any way how to use GAN's to extend my dataset of full traffic sign images with annotations of traffic sign class and position? Actually the class is not important, because I can do it separately for each class, but what matters is the position of traffic sign in generated image.

I think you'll enjoy this work from Apple on improving the realism of synthetic images.  Essentially what you need to do is generate a synthetic image then have your GAN modify the synthetic image so that a 1) a discriminator thinks it is real while also 2) not changing the gross structure of the image very much (so the traffic sign doesn't move) - yes, this loss function is going to take a little work!

Making synthetic data realistic enough to allow models to generalize successfully in the real world is a very active and exciting area of research, not least with respect to robotics, and so the work you are doing now should make you very attractive indeed to the right employer.


How many hidden layers are needed for this training data set I'm trying to separate classes in 3D space, the data are as in the sketch below:

There are 3 classes: 0,1,2; and with the look into the sketch, it seems that I need 3 planes to separate the classes, thus how many hidden layers should be in the DNN? Any roughly how many neurons in each layer?

Some say the number of hidden layers is roughly the number of separation times, so I put 3 hidden layers and it worked! But any reasons behind that simple measure?

You need to perform Hyperparameter Tuning to identify -

Number of hidden layers.
Number of neurons in each of the hidden layers.
Dropout
The activation function you use in each of your hidden layers.

There parameters are only related to how you build your model. There are others that relate to training like batch size, number of epochs and so on. Your model's performance ultimately depends on how well you tune your hyperparameters.

Also note that hyperparameter tuning is a trial and error task because it depends on several factors that may not be obvious to us. With experience, experts do build certain thumb rules about what may be the right choice, but there is no way to generalize it. "Some say the number of hidden layers is roughly the number of separation times" - is just another thumb rule. You simply need to find out what best suits your scenario.


Can we apply transfer learning between any two different CNN architectures? There are many types of CNN architectures: LeNet, AlexNet, VGG, GoogLeNet, ResNet, etc. Can we apply transfer learning between any two different CNN architectures? For instance, can we apply transfer learning from AlexNet to GoogLeNet, etc.? Or even just from a "conventional" CNN to one of these other architectures, or the other way around? Is this possible in general?
EDIT: My understanding is that all machine learning models have the ability to perform transfer learning. If this is true, then I guess the question is, as I said, whether we can transfer between two different CNN architectures – for instance, what was learned by a conventional CNN to a different CNN architecture.

No, transfer learning cannot be applied "between" different architectures, as transfer learning is the practice of taking a neural network that has already been trained on one task and retraining it on another task with the same input modality, which means that only the weights (and other trainable parameters) of the network change during transfer learning but not the architecture.
In my understanding, transfer learning is also only really effective in deep learning, but I could be wrong, considering that this Google search seems to yield some results.
You might otherwise be thinking of knowledge distillation, which is a related but different concept, where an already trained network acts as a teacher and teaches another network (a student network) with possibly a different architecture (or a machine learning model not based on neural networks at all) the correct outputs for a bunch of input examples.


How can we derive a Convolution Neural Network from a more generic Graph Neural Network? Convolution Neural Network (CNNs) operate over strict grid-like structures ($M \times N \times C$ images), whereas Graph Neural Networks (GNNs) can operate over all-flexible graphs, with an undefined number of neighbors and edges.
On the face of it, GNNs appear to be neural architectures that can subsume CNNs. Are GNNs really generalized architectures that can operate arbitrary functions over arbitrary graph structures?
An obvious follow-up - How can we derive a CNN out of a GNN?
Since non-spectral GNNs are based on message-passing that employ permutation-invariant functions, is it possible to derive a CNN from a base-architecture of GNN?

Spectral Graph Convolution
We use the Convolution Theorem to define convolution for graphs. The Convolution Theorem states that the Fourier transform of the convolution of two functions is the pointwise product of their Fourier transforms:
$$\mathcal{F}(w*h) = \mathcal{F}(w) \odot \mathcal{F}(h) \tag{1}\label{1} $$
$$ w * h = \mathcal{F}^{-1}(\mathcal{F}(w)\odot\mathcal{F}(h))  \tag{2}\label{2}$$
Here $w$ is the filter in spatial domain(time domain) and $h$ is the signal in spatial domain(time domain). For images this signal $h$ is a $2D$ matrix and for other cases this $h$ can be a $1D$ signal.
Assume we have $n$ number of node in a graph. In graph fourier transform the eigenvalues carry the notion of frequency. $\Lambda$ is the $ n \times n$ egenvalue matrix and it is a diagonal matrix. We can write equation 2 as:
$$w * h = \phi(\phi^{T}w \odot \phi^{T}h) = \phi\hat{w}(\Lambda)\phi^{T}h \tag{3}$$
Here $\phi$ is the eigenvector matrix of graph Laplacian $\in R^{n \times n}$,
$\hat{w}(\Lambda)$ is the filter in spectral domain(frequency domain) $\in R^{n \times n}$ a diagonal matrix, $h$ is the $1D$ graph signal $\in R^{n}$ in spatial domain and w is the filter in spatial domain $\in R^{n}$.
Vanilla Spectral GCN
We define the spatial convolutional layer such that given layer $h^{l}$ , the activation of the next layer is:
$$h^{l+1}=\sigma(w^l*h^l)  \tag{4}\label{4},$$
where $\sigma$ represents a nonlinear activation and $w^l$ is a spatial filter and $h$ is the graph signal.
We can perform the above equation in terms of spectral graph convolution operation as:
$$h^{l+1}=\sigma(\hat{w}^l*\hat{h}^l)  \tag{5}\label{5},$$
where $\hat{w}$ is the same filter but in the spectral domain(frequency domain). In case of vanilla GCN this equation yeild to:
$$ h^{l+1} = \sigma(\phi\hat{w}^{l}(\Lambda)\phi^{T}h)  \tag{6}\label{6}$$
Now, we will learn the $\hat{w}$ using backpropagation.
This vanilla GCN has several limitations, like larger time complexity and this does not guarantee localization in the spatial domain that we get from CNN's filter.
In next works, such as SplineGCNs, ChebNet, Klipf and Welling's GCN, and many other works address those issues, and try to solve them.
Note that we can think of ChebNet and Klipf and Welling's GCN as a message-passing system, but, in the background, they are computing spectral convolution and also they use some standard assumption that's why we do not need any eigenvector and we implement them in the spatial domain, but still they are spectral convolution.
There is also another branch in graph convolution called spatial graph convolution. I only talked about the spectral graph convolution.


Why is having low variance important in offline policy evaluation of reinforcement learning? Intuitively, I understand that having an unbiased estimate of a policy is important because being biased just means that our estimate is distant from the truth value.

However, I don't understand clearly why having lower variance is important. Is that because, in offline policy evaluation, we can have only 'one' estimate with a stream of data, and we don't know if it is because of variance or bias when our estimate is far from the truth value? Basically, variance acts like bias.

Also, if that is the case, why having variance is preferable to having a bias?

Having low variance is important in general as it reduces the number of samples needed to obtain accurate estimates. This is the case for all statistical machine learning, not just reinforcement learning.

In general, if you are estimating a mean or expected quantity by taking many samples, the variation in the error is proportional to $\frac{\sigma}{\sqrt{N}}$ for a direct arithemtic mean of all samples, and behaves similarly for other averaging approaches (such as recency-weighted means using a learning rate). The bounds on accuracy can be made better by either increasing $N$ i.e. taking more samples, or by decreasing the variance $\sigma^2$.

So anything you can do to reduce variance in your measurements has a direct consequence of reducing the number of samples required to achieve the same degree of accuracy.

In the case of off-policy reinforcement learning, there is added variance - compared to on-policy learning - due to different probabilities of taking an action in behaviour and target policies. This is due to the need to adjust reward signals using importance sampling - multiplying by the importance sampling ratio will make the reward signal vary more (in fact it can become unbounded). This is not really any more of a challenge than any other source of variance, but as it interferes with the goal of speedy learning, a lot of research effort has been put into methods that reduce the variance.


Why are lambda returns so rarely used in policy gradients? I've seen monte-carlo reward $G_{t}$ used in REINFORCE and TD($0$) reward $r_t + \gamma Q(s', a')$ used in vanilla actor-critic. I've never seen someone use lambda reward $G^{\lambda}_{t}$ in these situations, nor in any other algorithms.

Is there a specific reason for this? Could there be performance improvements if we used $G^{\lambda}_{t}$?

That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using $G_t^{\lambda}$ returns for the critic (value function estimator), but also for the actor's policy gradients.

Note that you do not explicitly see the $G_t^{\lambda}$ returns mentioned in the pseudocode. They are being used implicitly through eligibility traces, which allow for an efficient online implementation (the "backward view").

I do indeed have the impression that such uses are fairly rare in recent research though. I haven't personally played around with policy gradient methods to tell from personal experience why that would be. My guess would be that it is because policy gradient methods are almost always combined with Deep Neural Networks, and variance is already a big enough problem in training these things without starting to involve long-trajectory returns.

If you use large $\lambda$ with $\lambda$-returns, you get low bias, but high variance. For $\lambda = 1$, you basically get REINFORCE again, which isn't really used much in practice, and has very high variance. For $\lambda = 0$, you just get one-step returns again. Higher values for $\lambda$ (such as $\lambda = 0.8$) tend to work very well in my experience with tabular methods or linear function approximation, but I suspect the variance may simply be too much when using DNNs.

Note that it is quite popular to use $n$-step returns with a fixed, generally fairly small, $n$ in Deep RL approaches. For instance, I believe the original A3C paper used $5$-step returns, and Rainbow uses $3$-step returns. These often work better in practice than $1$-step returns, but still have reasonably low variance due to using small $n$.


In UCB, is the actual upper bound an upper bound of an one-sided or two-sided confidence interval? I'm a bit confused about the visualization of the upper bound (following the notation of (c.f. Sutton & Barto (2018))
$$Q_t(a)+C\sqrt{\frac{\mathrm{ln}(t)}{N_t(a)}}$$
In many blog posts about the UCB(1)-algorithm, such as visualized in the following image (c.f. Link ):

Isn't the upper (confidence) bound simply the upper bound of a one-sided confidence interval instead of a two-sided confidence interval as shown in the image above?
A lower bound of the interval is completely useless in this case, or am I wrong?

The upper bound used here is derived from Hoeffding's inequality, which provides a symmetric, two-sided confidence interval. A good pair of blog posts on how this bound used in UCB for bandits is derived can be found here:

First steps: Explore-then-Commit
The Upper Confidence Bound Algorithm

Indeed, in practice when using this UCB for bandits, we do not actually care about the lower bound. We only need the upper found for the exploration mechanism. But the lower bound does still exist, even if we don't use it.


Explanation of this L2 minimization equation I am trying to understand the last two lines of this math notation. How Var and double summation of Cov came to the equation. The first two lines I understood something like $(a-b)^2 = a^2 -2ab +b^2$.

As in the second line, the first two terms are $\mathbb{E}_{\hat{y}}$, it means the variable of the expectation is $\hat{y}_i$ and you can take out $y_i$s from $\mathbb{E}_{\hat{y}}$s. Now we can use $\mathbb{E}\{\hat{y}_i\} = y_i$ and rewrite the terms of the second line likes the following:
$$
\mathbb{E}_{\hat{y}}(\sum_i y_i)^2 = (\sum_i y_i)^2 \\
\mathbb{E}_{\hat{y}}\left[(\sum_i y_i)(\sum_i \hat{y}_i)\right] = (\sum_i y_i) \mathbb{E}_{\hat{y}}(\sum_i \hat{y}_i) = (\sum_i y_i)^2
$$
The second line is written based on linearity of the expectation and $\mathbb{E}\{\hat{y}_i\} = y_i$. Hence, we can rewrite the second line like the following as $\sum_i y_i = \mathbb{E}_{\hat{y}}(\sum_i \hat{y}_i)$:
$$
\frac{1}{N^2}\left[\mathbb{E}_{\hat{y}}(\sum_i \hat{y}_i)^2 - \left(\mathbb{E}_{\hat{y}}(\sum_i \hat{y}_i)\right)^2\right]
$$
And the final step is using this formula $Var(X) = E(X^2) - (E(X))^2$  and take $X = \sum_i \hat{y}_i$:
$$
\frac{1}{N^2}\left[\mathbb{E}_{\hat{y}}(\sum_i \hat{y}_i)^2 - \left(\mathbb{E}_{\hat{y}}(\sum_i \hat{y}_i)\right)^2\right] = \frac{1}{N^2} Var(\sum_i \hat{y}_i)
$$
From variance to covariance, you can use this formula: $$Var(\sum_{i=1}^nX_i) = \sum_{i=1}^n\sum_{j=1}^n cov(X_i, X_j)$$


How is the AI in 3d games implemented? A few days ago, I started looking a bit more into AI and learning about the way it works, and it is very interesting, but I can't find a clear answer on how the artificial intelligence is implemented in 3d shooter games, like COD or practically any 3d game.
I just don't understand how they teach the enemies such different things based on the game to fit its narratives. For example, is the enemy "AI" in 3d games just a bunch of if-else statements, or do they actually teach the enemies to think strategically? In big AAA games, you can clearly see that enemies hide from you in shootings and peek to shoot not just rush and get killed.
So, how is the AI in 3d games implemented? How do they code it? You don't need to explain in detail, but just give me the idea. Do they use algorithms?

Overlap between AI and "Game AI"
Nowadays, if you search for AI online, you will find a lot of material about machine learning, natural language processing, intelligent agents and neural networks. These are not the whole of AI by any means, expecially in a historical context, but they have recently been very successful, there is lots of published material about them.
Games, especially action games, tend not to use these new popular technologies because they have other priorities. However, under the broadest definitions, there is good overlap between AI in general and game AI, or more specifically enemy AI within a game:

Computer-controlled opponents within a game are effectively autonomous agents operating within an environment.

The game's purpose is to entertain the player, and this usually translates into requiring challenging and believable behaviour for actions taken by opponents (for some value of "believable").

A 3D map with a physics engine, other moving agents and possible hazards is a complex environment that needs some effort from a developer in order to define working behaviours.

Many games need to solve one or two classic AI problems such as pathfinding.

All the above means that the term "Game AI" is a good choice for describing how enemy agents are designed and implemented. However, there are large differences between goals of AI applied today in a 3D shooter and the goals of AI used in industry to control decision making.

Action games have a low CPU budget available per enemy for decision-making. Control systems in industry can have dedicated machinery, sometimes multiple machines, just to run the AI.

Action game priority is the game experience of the player, and this translates into strict targeting of a definition of "believable" for enemy behaviour that will entertain. In contrast, AI used in industry has accuracy as a high priority.

Typical game AI in 3D shooters
Developing enemy AI in games is a specialist skill, and only partly related to the purpose of this site. For detailed discussion you may want to look into Game Development Stack Exchange which has many questions and answers on topics like Enemy AI
Very roughly, the key traits of enemy AI in a game like Call of Duty could look like this (I have no inside knowledge of how COD does this, so may be inaccurate in places):

Enemies will have one or more modes of behaviour defined. This might be as simple as "Idle" and "Attack Player", or there could be many. For COD I would suspect there are many, and some may be in a hierarchy - e.g. there may be several sub-types of "Attack" behaviour depending on the enemy design.

Within a mode of behaviour, there may be a few different components defined, only some of which use AI routines. For instance, there will be specific animations related to walking or standing which are not AI, but there will also be some pathfinding AI if the agent is attempting to move around.

Decisions to switch between different modes of behaviour are often scripted with simple triggers, such as detecting player visibility using ray-casting between the location of player and enemy. When game AI fails to produce realistic results, it is often these high level triggers being brittle and not covering edge cases that causes it. Depending on complexity of the game and number of behaviours, there may be an algorithm managing the transitions between them, such as a finite state engine or behaviour trees.

Enemies will be presented with highly simplified observations of the game world from their perspective, in order to speed AI decisions.

AI systems will have CPU budget restricted. A 3D game spends significant resources rendering scenes, and will often try to render quickly, e.g. 100 times per second. There are multiple ways that AI budget can be allocated, but it is relatively common for AI calculations to be spread over muliple frames, and for search structures for tasks like path-finding to persist over time.

There will be a middle ground between scripted behaviour and AI-driven behaviour where analysis is done as part of game design. For instance, pathfinding routes may be pre-calculated to some degree. A system of way points is one example of this - it might be set by the game designer, it may be calculated by an AI component of the game asset-building pipeline, or it may be dynamically calculated and cached during a game session so that multiple enemy units can share it.

When complex AI is not needed to achieve a goal, when a simple caclulation or "puppet-like" behaviour would do just fine, then this could be chosen instead. For instance, an enemy "aiming" at a player can be a simple vector calculation, perhaps with a fudge factor of a miss chance depending on the range to make the enemy seem fallible and not as much like a machine.

I do not know how the behaviour of enemies using cover is implemented in COD. A simple variant would be to have each enemy hard-coded to use a specific piece of cover that usually works well against the player due to map design. However, it is definitley possible to have a search algorithm running (perhaps over multiple frames) that assesses nearby locations that the enemy could reach against the player's current position, and then pick those as variables to plug into the "take cover and fire on player" scripted behaviour. That assessment would use the same kind of visibilty detection between enemy and player that is used to trigger changes between "Idle" and "Attack" behaviours for the enemy.

As an aside, one related thing I find interesting is in using modern AI techniques to blend animations and make interactions between actors and the environment look more realistic. Although it is a lower-level feature than the question about enemy behaviour you are asking, it is an interesting cross-over between robotics and game playing that we will likely see in next-generation games, and probably applied first to the more detailed player model animations. Here is another example applied to a humanoid agent switching smoothly between different tasks, which any game player will recognise as something which game engines cannot do well at the moment - there are usually many jarring transitions caused by events in the game.


Can DQN perform better than Double DQN? I'm training both DQN and double DQN in the same environment, but DQN performs significantly better than double DQN. As I've seen in the double DQN paper, double DQN should perform better than DQN. Am I doing something wrong or is it possible?

There is no thorough proof, theoretical or experimental that Double DQN is better then vanilla DQN. There are a lot of different tasks, paper and later experiments only explore some of them. What practitioner can take out of it is that on some tasks DDQN is better. That's the essence of Deep Mind's "Rainbow" approach - drop a lot of different methods into bucket and take best results.


Is there a place where I can read or watch to get an accurate TensorFlow code wise explanation? I have a piece of code and I don't seem to really understand it but I'd love to get a source/link/material that would help me understand the basic functions in TensorFlow. Are there any recommended resources for learning the same?

The best resource for learning TensorFlow 1.9 and earlier is this course by Stanford. Also additional resources for the entire overview of TensorFlow and its comparisons with NumPy has been made in this video. For hands on models check these videos by Sentdex and also some high level tutorials by Hvass Labs.


Can TensorFlow minimize "symbolically" From https://stackoverflow.com/questions/36370129/does-tensorflow-use-automatic-or-symbolic-gradients, I understood TensorFlow requires all the operations in the Graph to be explicit formulas (instead of black-boxes, such as raw python functions) to do Automatic Differentiation. Then it will do some kind of Gradient Descent based on that to minimization.

I'm wondering, since it already know all the explicit formulas, can it directly find out the minimum by examining the equation itself? Like computing the points where gradient is zero or do not exist, then do some kind of processing to find out the minimum.

I found it is simple to do this "symbolic minimization" above with few variables such as minimizing Σ(a_i - v)^2 where v is the trainable variable an a_i are all the training samples. I'm not sure is there a general way though.

If by "symbolic" you mean finding an analytical solution, that is, an equation for each weight, then the answer is no. The example you chose results in a system linear equations, which can be solved analytically. However once you introduce non linearities (by using activation functions with more than one layer), most non trivial cases will have no analytical solution and will need to be solved numerically. This is not a problem specific to tensorflow, it is  a mathematical issue, it will not be possible on any language, current or future. Unless there is some revolution in math first.


For some reasons, a reward becomes a penalty if I am working to build a reinforcement agent with DQN. The agent would be able to place buy and sell orders for a day trading purpose. I am facing a little problem with that project. The question is "how to tell the agent to maximize the profit and avoid the transaction where the profit is less than 100$".

I want to maximize the profit inside a trading day and avoid to place the pair (limit buy order, limit sell order) if the profit on that transaction is less than 100$. The idea here is to avoid the little noisy movements. Instead, I prefer long beautiful profitable movements. Be aware that I thought using the "Profit & Loss" as the reward.

"I want the minimal profit per transaction to be 100$" ==> It seems this is not something that is enforceable. I can train the agent to maximize profit per transaction, but how that profit is cannot be ensured.

At the beginning, I wanted to tell the agent, if the profit of a transaction is 50 dollars, I will remove 100 dollars, then It becomes a penalty of 50 dollars for the agent. I thought it was a great way to tell the agent to not place a limit buy order if you are not sure it will give us a minimal profit of 100$. It seems that all I would be doing there is simply shifting the value of the reward. The agent only cares about maximizing the sum of rewards and not taking care of individual transactions.

How to tell the agent to maximize the profit and avoid the transaction where the profit is less than 100$? With that strategy, what guarantee that the agent will never make a buy/sell decision that results in less than 100 dollars profit? Does the sum of reward - # transaction * 100 can be a solution?

I want to maximize the profit inside a trading day and avoid to place the pair (limit buy order, limit sell order) if the profit on that transaction is less than 100$. Be aware that I thought using the "Profit & Loss" as the reward.

To me this implies that your profit per transaction is not the true reward function that you should be using. You don't say directly in the question, but presumably there is some per-transaction cost, tax or other issue which means that these low gain transactions are not desirable.

The answer is to find a more accurate reward function. You have suggested a quick fix of subtracting a fixed offset from the reward. This should have a desirable effect of limiting borderline buy/sell arrangements and meeting your constraint, so I would suggest trying it. The main issue I see with it is that you may have changed the reward function so that it doesn't truly reflect your goals.

A better approach is to look more carefully at the problem and your goals at a higher level. There must be a reason for wanting to avoid these smaller profit transactions. What is it, and can it be expressed itself as a reward? For instance, if there will be a transaction fee, include that. If the fee structure or reasoning is complicated or appears delayed or aggregated over many actions, then this makes optimisation harder, but RL is actually designed to cope with this. You could for instance only reward the agent with profit/loss after a group of actions covering a whole day's trade. You would then rely on the learning algorithm to figure out which states and actions combined to generate the observed reward.

If there is an unknown or random difference between predicted and actual financial gain then no amount of juggling the reward function gets around that. For RL, learning about and predicting expected gains is "built in", but that does not mean that this task becomes easier - in fact if prediction in the problem is hard, you may be better off focusing on that and forgetting RL at least initially. Your question is not clear on that, but if you want to avoid risk whilst learning, you should bear in mind that a simple hard rule would likely interfere with the ability of RL to work. At its heart RL is a trial-and-error learning system. Errors should be expected, and are required for the system to learn where best the balance point is between risk and reward. Of course that doesn't help you if the learning system would deliberately make you bankrupt exploring what happens when you sell things at a loss - there are likely to be ways to avoid that and still achieve your goals, but you will need to explain more about your system in a different question.


How can I formulate the map colouring problem as a hill climbing search problem? I have a map. I need to colour it with $k$ colours, such that two adjacent regions do not share a colour.

How can I formulate the map colouring problem as a hill climbing search problem?

First of all you need an initial solution. You will then improve this solution with hill climbing.

For your initial solution, you can color the map randomly using the K colors. This will most likely result in conflicts (adjacent regions of the same color).

Then the hill climbing part: Find a region which has conflicts and swap its color for another color, making sure that the new color does not incur more conflicts than the old one. With each iteration your solution should slowly improve.

Note that hill climbing is not perfect and that you may not find a feasible solution in the end.


How do we reach at the formula for UCB action-selection in multi-armed bandit problem? I came across the formula for Upper Confidence Bound Action Selection (while studying multi-armed bandit problem), which looks like:

$$
A_t \dot{=} \operatorname{argmax}_a \left[ Q_t(a) + c \sqrt{ \frac{\ln t}{N_t(a)} } \right]
$$

Although, I understand what the second term in the summation actually means but I am not able to figure out how and from where the exact expression came from, what is the log doing there? What effect does $c$ have? And, why a square root?

Here is an intuitive description/explanation.
$c$ is there for a trade-off between exploration and exploitation. If $c=0$ then you only consider $Q_t(a)$ (no exploration). If $c \rightarrow \infty$ then you only consider exploration term.
$\frac{\ln t}{N_t(a)}$ is there to balance out exploration term. If you consider a simple case where you only have one action (then it wouldn't make sense to explore you could always pick that action but let's pretend there is) then as $t \rightarrow \infty$, because $\ln t$ has sublinear growth,
\begin{equation}
\frac{\ln t}{N_t} \rightarrow 0
\end{equation}
So, after you picked an action infinitely many times, the exploration term will completely diminish, i.e. you already know a lot about what that action does. If you picked a numerator that doesn't have sublinear growth, then as $t \rightarrow \infty$ exploration term would not diminish, so you would always have a chance to explore and exploration term can "overpower" action value term if $Q_t$ is very small, even after you picked an action infinitely many times, which is not desired.
A similar thing is with multiple actions, $\ln t$ will make sure the exploration term $\rightarrow 0$ if you picked it many times, but it's still better than constant term $K/N_t(a)$, where $K$ is some constant, because it can diminish too fast.
With $\ln t$ you will also not stop exploring completely if you haven't picked some action in a long time, because $\ln t$ will keep growing and $N_t(a)$ will remain the same, so their fraction will increase with time, which is useful in non-stationary environments.
The square root is also there probably to balance out the magnitude of exploration term.
You can also see this answer. It has couple of links to some papers for a more mathematical description.


What are the real world uses for SAT solvers? Why somebody would use SAT solvers (Boolean satisfiability problem) to solve their real world problems?

Are there any examples of the real uses of this model?

Instead of talking about just SAT solvers, let me talk about optimization in general. Many economic problems can be cast as optimization problems: for example, FedEx may have a list of packages and the destinations for those packages, and must decide which packages to put on which trucks, and what order to deliver those packages in.

If you write out a mathematical description of this problem, there are a truly stunning number of possible solutions, and a well-defined way to evaluate which of two solutions is better. A solver is an algorithm that will evaluate a solution, come up with another solution, and then evaluate that one, and so on.

In small cases and simple problems, the solver can also terminate with a proof that it is actually the best solution possible. But typically instead the solver just reports "this is the best solution that I've seen," and that's used. An improvement in the solver means you can reliably get lower-cost solutions than you were seeing before.

For the SAT problem specifically, the Wikipedia page on SAT gives some examples:

  Since the SAT problem is NP-complete, only algorithms with exponential worst-case complexity are known for it. In spite of this, efficient and scalable algorithms for SAT were developed over the last decade[when?] and have contributed to dramatic advances in our ability to automatically solve problem instances involving tens of thousands of variables and millions of constraints (i.e. clauses).[1] Examples of such problems in electronic design automation (EDA) include formal equivalence checking, model checking, formal verification of pipelined microprocessors,[12] automatic test pattern generation, routing of FPGAs,[14] planning, and scheduling problems, and so on. A SAT-solving engine is now considered to be an essential component in the EDA toolbox.


Do all neurons in a layer have the same activation function? I'm new to machine learning (so excuse my nomenclature), and not being a python developer, I decided to jump in at the deep (no pun intended) end writing my own framework in C++.

In my current design I have given each neuron/cell the possiblity to have a different activation function. Is this a plausible design for a neural network? A lot of the examples I see use the same activation function for all neurons in a given layer.

Is there a model which may require this, or should all neurons in a layer use the same activation function? Would I be correct in using different activation functions for different layers in the same model, or would all layers have the same activation function within a model?

From here:

  Using other activation functions don’t provide significant improvement in performance and tweaking them doesn’t provide any big improvement. So as per simplicity we use same activation function for most of the case in Deep Neural Networks.


Unexpected results when comparing a greedy policy to a DQN policy I am trying to work on a variation of the Access-Control Queuing Task problem presented in Chapter 10 of Sutton’s reinforcement learning book [1].

Specific details of my setup are as follows:

I have different types of tasks that arrive to a system
(heavy/moderate/light with heavy tasks requiring more time to be
processed.). The specific task type is chosen uniformly at random. The task inter-arrival time is $0.1s$ on average.
I have different classes of servers that can process these tasks
(low-capacity; medium capacity; high capacity; with high capacity
servers having a faster processing time). When I select a specific
server from a given class, it becomes unavailable during the
processing time of the task assigned to it. Note that the set of servers (and as a result the number of servers of each class) is not fixed, it instead changes periodically, according to the dataset used to model the set of servers (so specific servers may disappear and new ones may appear, as opposed to the unavailability caused by the assignment). The maximum number of servers of each class is $10$.
My goal is to decide which class of server should process a given
task, in a way that minimizes the sum of the processing times over
all tasks.

The specific reinforcement learning formulation is as follows:

State: the type of task (heavy/moderate/light) ; the number of available low
capacity servers; the number of available medium capacity servers; the number
of available high capacity servers
Actions: (1) Assign the task to a low capacity server (2) assign the
task to a medium capacity server (3) assign the task to a high
capacity server (4) a dummy action that has a worse processing time
than the servers with low capacity. It is selected when there are no free servers.
Rewards: the opposite of the processing time, where the processing times are as follows (in seconds):

|               | Slow server | Medium Server | Fast server | "Dummy action" |
|---------------|-------------|---------------|-------------|----------------|
| Light task    | 0.5         | 0.25          | 0.166       | 0.625          |
| Moderate task | 1.5         | 0.75          | 0.5         | 1.875          |
| Heavy task    | 2.5         | 1.25          | 0.833       | 3.125          |

My intuition for formulating the problem as an RL problem is that 'Even though assigning Light tasks to High capacity servers (i.e. being greedy) might lead to a high reward in the short term, it may reduce the number of High capacity servers available when a Heavy task arrives. As a result, Heavy tasks will have to be processed by lower capacity servers which will reduce the accumulated rewards'.

However, when I implemented this (using a deep Q-network[2] specifically), and compared it to the greedy policy, I found that both approaches obtain the same rewards. In fact, the deep Q-network ends up learning the greedy policy.

I am wondering why such a behaviour occured, especially that I expected the DQN approach to learn a better policy than the greedy one. Could this be related to my RL problem formulation? Or there is no need for RL to address this problem?

[1]Sutton, R. S., & Barto, A. G. (1998). Introduction to reinforcement learning (Vol. 135). Cambridge: MIT press.

[2]Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., ... & Petersen, S. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529-533.f

Your problems appear to derive from a non-Markov state description. In brief, the agent has no way to ever learn that "heavy jobs will make a server unavailable for longer" and this is further compounded by arbitrary state transitions every 10 steps whilst the agent cannot track time.

Looking at the example from Sutton & Barto, you should note that they took care to model a valid MDP. Servers become available at random on each time step, and there is no hidden state. In your case you have three sources of hidden data that affects state evolution systematically. If an agent had some data about these then it might be able to use to optimise choices better:

Servers in use due to tasks assigned by the agent return deterministically n steps after the action that assigned them. The agents inability to track this is likely to be your largest problem - there is no data flow in RL that will allow the agent to associate the heavier jobs or slower servers with making a server unavailable for longer, so it never "sees" the clash of resource management that you want it to resolve.
Every 10 timesteps there is an arbitrary change to server availability. This is a problem because it is entirely predictable on which time steps it can occur, but the agent does not know the current timestep.
You are driving this from a dataset, not randomly, so the impact on this will vary depending on how the dataset evolves over time. A particularly bad case here would be if the dataset represented a "working day" so that there were clear timezones when availability followed different patterns. The agent has no data input in the state in order to associate patterns of use (such as the time of day), so the environment either becomes an online learning problem (agent must continuously learn and adapt to new patterns as they arise), or it must come up with a strategy that works in some average over all possible patterns.

In addition I can see a possible secondary effect, an "exploit" in the simulation, that may have a noticeable effect depending on how often specific servers drop out of the pool. It would appear if an agent assigns a task to a server, that you have no mechanism to unassign that task should the server drop out. So an aggressive agent that assigns greedily to the best available servers will get extra "free" time on the high capacity servers if it always claims them. As the agent doesn't know when the drop outs could happen, this effect competes with the average of leaving servers available just in case a heavy task arrives.

How to fix it? It depends on how much of your simulation is fixed by your requirements. You can either adjust the simulation to match your current state representation, or you can change the state representation to capture more system data from the simulation.

Probably the simplest state change would be to expand the list of servers to always include all servers, and for each unavailable server to have a "ticker" to show how many time steps before it becomes available. To make the problem smaller, you could maybe just track 1-10 ticks and have a combined state for (>10), as that will expose enough of the key state information that the agent can make decisions about pushing easy tasks into medium or low capacity servers. This does not need to be done to the level server id if the servers truly are interchangeable otherwise. So instead of an array of number of free servers by type, consider that it expands to a table with number of free servers by type at one end, then a column of number of servers by type with 0.1s remaining until they are free, then a column for servers that still have 0.2s remaining etc etc


What would be the implications of mistakenly adding bias after the activation function? I was looking at the source code for a personal project neural network implementation, and the bias for each node was mistakenly applied after the activation function. The output of each node was therefore $\sigma\big(\sum_{i=1}^n w_i x_i\big)+b$ instead of $\sigma\big(\sum_{i=1}^n w_i x_i + b\big)$. Assuming standard back-propagation algorithms are used for training, what (if any) impact would this have?

Let $L(\mathbf{w}, b) = \sigma \left(\sum_{i=1}^n w_i x_i \right)+b$, then the partial derivative of $L$ with respect to $b$, in Leibniz's notation, is $\frac{\partial L}{\partial b} = 1$. Let $L(\mathbf{w}, b) = \sigma \left(\sum_{i=1}^n w_i x_i + b \right)$, then the partial derivative of $L$ with respect to $b$ is $\frac{\partial L}{\partial b} = \frac{\partial L}{\partial \sigma} \frac{\partial \sigma}{\partial b}$.

So, in general, the partial derivatives with respect to $b$ would be different in these two cases, thus, in the gradient descent step, $b$ would be updated differently in both cases. In practice, it may or not affect the performance of the model, depending on the importance of the bias with respect to the given problem. Read this answer to understand the role of the bias.


Why can't we combine both training and validation data, given that both types of data are used for developing the model? Sorry if I sound confused. I read that data to be fed to a machine are divided into training, validation and test data. Both training and validation data are used for developing the model. Test data is used only for testing the model and no tuning of the model is done using test data.
Why is there a need to separate out training and validation data since both sets of data are for developing/tuning the model? Why not keep things simple and combine both data sets into a single one?

Usually we are splinting the training dataset because we want to fine tune, find the best hyper-parameters, for our model. If we combine the validation and training dataset given a network complex enough we could achieve perfect performance for the given task. But having very good performance on the training dataset does not mean our model is useful. It might be the case that our model is very good on the training dataset, but it fails to generalize, so when given samples outside of the training dataset the performance will significantly drop (this is called over-fitting). By splitting the dataset, we are testing how well our model is performing on new/unseen data.


When should I create a custom loss function? I'm using a neural network to solve a multi regression problem because I'm trying to predict continuous values. To be more specific, I'm making a tracking algorithm to track the position of an object, I'm trying to predict two values, the latitude and longitude of an object.
Now, to calculate the loss of the model, there are some common functions, like mean squared error or mean absolute error, etc., but I'm wondering if I can use some custom function, like this, to calculate the distance between the two longitude and latitude values, and then the loss would be the difference between the real distance (calculated from the real longitude and latitude) and the predicted distance (calculated from the predicted longitude and latitude). These are some thoughts from me, so I'm wondering if such an idea would make sense?
Would this work in my case better than using the mean squared error as a loss function?
I had another question in mind. In my case, I'm predicting two values (longitude and latitude), but is there a way to transform these two target values to only one value so that my neural network can learn better and faster? If yes, which method should I use? Should I calculate the summation of the two and make that as a new target? Does this make sense?

Using two value and using MSE is probably a better approach. I'd you combine the value to one value, like the case of summation, the network may fits to output 0 on one axis and the value on the other. The method you propose also have the same issue. There are many combination to the real distance, but only one is correct.
For a neural network to learn faster, one value will not help it learn faster. Instead, accuracy is often increased if the predicted value is a one hot vector of labels instead of a single value. Hope this can help you.


Can reinforcement learning be utilized for creating a simulation? According to the definition, the AI agent has to play a game by it's own. A typical domain is the blocksworld problem. The AI determines which action the robot in a game should execute and a possible strategy for determine the action sequence is reinforcement learning. Colloquial spoken, reinforcement learning leads to an AI agent who can play games.

Before a self-learning character can be realized, the simulation has to be programmed first. That is an environment which contains the rules for playing blocksworld or any other game. The environment is the house in which the AI character operates. Can the Q-learning algorithm be utilized to build the simulation itself?

Can the Q-learning algorithm be utilized to build the simulation itself?

Only in the presence of a meta-environment, or meta-simulation where the goals of creating the original simulation are encoded in the states, available actions and rewards.

A special case of this might be in model-learning planning algorithms where there exists a "real" environment to refer to, and the agent benefits from exploring it and constructing a statistical model that it can then use to create an approximate simulation of the outcomes of sequences of actions. The Dyna-Q algorithm, which is a simple extension of Q-learning, is an example of this kind of model building approach. The simulation is very basic - it simply replays previous relevant experience. But you could consider this as an example of the agent constructing a simulation.

Getting an agent to act like a researcher and actually design and/or code a simulation from scratch would require a different kind of meta-environment. It is theoretically possible, but likely very hard to implement in general way - even figuring out the reward scheme to express the goals of such an agent could be a challenge. I'm not aware of any examples, but entirely possible someone has attempted this kind of meta agent, because it is an interesting idea.

Possibly the simplest example would be a gridworld meta-environment where a "designer" agent could select the layout of objects in the maze, with the goal of making a second "explorer" agent's task progressivley more difficult. The designer would be "creating the simulation" only in a very abstract way though, by setting easy-to-manage parameters of the environment, not writing low level code.

There is not much difference between the approach above and having two opposing agents playing a game. It is different from a turn-based game like chess in that each agent would complete a full episode, and then be rewarded by the outcome at the end of the combined two episodes. There are some similarities to GANs for image generation.


Is there any research work on the usefulness of the concept of "personality" for an AGI? Humans have been endowed with personalities by nature, and it is not clear (to me at least) if this is a feature or a bug. This has been explored in science fiction by various notions of Borg-like entities. It is my belief that, for narrative reasons, such stories usually end with the humans with their flawed personalities winning in the end.
Are there experts who have analyzed, perhaps mathematically, design criteria for an AI agent with weakly enforced goals (e.g. to maximize reproduction in the human case) in an uncertain environment, and ended up with the answer that a notion of personality is useful?
If there are AI researchers, philosophers (or maybe even science fiction) writers who have examined this question in their work, I would be happy to know about those too.

'Personality' is something of a 'suitcase word' (Minsky) for quite a large collection of (presumably reasonably consistent) observable traits.

It seems clear that there is a certain collective advantage in having a consistent personality - specifically that it affords observers some learning gradient in an otherwise uncertain environment. This is of particular importance because those consistencies might have been arrived at using different learning mechanisms than the ones a given observer has.

Hence, in any non-trivial coevolutionary system, other organisms will inevitably make use of any such consistencies. Consider a simple robot, called Alice, say, that has the trait of 'quickly flashing red when it sees a blue robot'. It makes sense for all observers to exploit everything that they perceive as correlating with Alice's behavior, in particular, the prediction that a blue robot is likely to be present.

The best reference I can recommend on this (which shows that we tend to ascribe 'personality' to even very simple mechanisms) is 'Vehicles' by Valentino Braitenberg.


S-shaped nonlinearities in tanh neurons I have started reading Fundamentals of Deep Learning by Nikhil Buduma and I have a question regarding tanh neurons. In the book, it is stated:

  "When S-shaped nonlinearities are used, the tanh neuron is often preferred over the sigmoid neuron because it is zero-centered."

Can anyone explain me why exactly??

It should be mentioned that RELU is the current activation function standard. But to answer your question:
The importance here is that it is very common to have normalized your data (e.g. using batch normalization), then the data is centered around 0.

As @DuttaA commented, look at this answer from Cross-Validated:

  Since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that [output] values are in the range [0,1].

And

  The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1] Avoiding bias in the gradients. This is explained very well in the paper, and it is worth reading it to understand these issues.


How do I get multiple loss per sample in keras evaluate? Usually, when I evaluate() a model, I would get a single loss that is already averaged over all samples. How do I get the loss per each sample and return all of them?

E.g. if my dataset has 100 samples, I want to get 100 losses, for each of the sample instead of 1 averaged loss.

You cannot achieve that from a single model.evaluate(...) call. But you could always do evals = [model.eval(X[i:i+1], Y[i:i+1]) for i in range(len(X))].

Note if you get the tensors yourself, you can get the backend session from keras.backend.get_session() and get anything you want by making your own sess.run() calls


Does it make sense to use batch normalization in deep (stacked) or sparse auto-encoders? Does it make sense to use batch normalization in deep (stacked) or sparse auto-encoders?

I cannot find any resources for that. Is it safe to assume that, since it works for other DNNs, it will also make sense to use it and will offer benefits on training AEs?

The term Batch Normalization is both

A system reliability choice (in terms of convergence) and
an execution strategy.

Batching is generally the process of focusing on process P with source data S to produce result R under conditions that are favorable in terms of timing, data availability, and resource utilization, such as these.

P is requires nontrivial time and computing resource and there is a window of time when it can be done on hardware H with minimal impact on other operations
R is needed by multiple other processes P2, P3, ... and draws on a set of additional inputs I2, I3, ... so that completing P in larger groups is efficient

One cannot draw a general conclusion about the value of batching that will apply to all specific installations of hardware, networking, real time demands, functional intention, and ML topology.  Also, beware of equating deep with stacked.  Depth may have multiple meanings now, but was originally used for the dimension describing the number of layers in a multi-layer perceptron, with width being the number of artificial neurons in each layer.

The term stacking can be ambiguously related to depth.  Consider that process flows through artificial neural nets has often been represented graphically as signal propagation from left to right, as is the convention in electrical engineering and text in western languages.

We then have two directions for stacking that means two different things.

Stacking in the same direction as signal flow, which indicates a sequence of operations on a vector, matrix, cube, or hyper-cube
Stacking in the tangential direction with respect to signal flow, which indicates a connectivity between items in the stack that is related to supervision, collaboration, or competition

Whether Sparse Autoencoders or other elements of DNNs should be fronted with batch normalization has two forks to the answer.

Is normalization helpful in reducing either signal saturation in gradient calculations during propagation or overflow conditions?
If that answer is, "Yes," will batching the normalization have some value for reasons like the ones described above?


LSTM model on different time scales I am a newbie to machine learning. I have an LSTM model that predicts the next output n+1

time 1, params 1, output 1

time 2, params 2, output 2

time 3, params 3, output 3

.
.

time n, params n, , output n

time n+1 --> predicts output n+1

 Here the times are all in minutes, so I can predict the next output in the series which is going to be the next minute. My question is that what if I want to predict the next 5 minutes. One solution was to throw out all the data except in steps of 5 minutes so the next step is automatically would be 5 minutes. This is clearly a waste of all the data that I have gathered. Can you please recommend what I can do about the prediction on different time scales?

What you could do is just try and bypass the rest of the network after the LSTM if it isn't the 5'th minute.
Depending on your framework this can be easy or a painstakingly task compared to the alternative.
The alternative is just running and throwing away the output that isn't the next 5'th minute. While the last may seem inefficient it's rather easy to implement and takes just a bit more execution time. If execution time isn't a problem it's the easiest to get started with, if it doesn't work for your task you can always change it.