Any interesting ways to combine Monte Carlo tree search with the minimax algorithm? I've been working on a game-playing engine for about half a year now, and it uses the well known algorithms. These include minimax with alpha-beta pruning, iterative deepening, transposition tables, etc.

I'm now looking for a way to include Monte Carlo tree search, which is something I've wanted to do for a long time. I was thinking of just making a new engine from scratch, but if possible I'd like to somehow import MC tree search into the engine I've already built.

Are there any interesting strategies to import MC tree search into a standard game-playing AI?

There has indeed been some research towards combining MCTS and minimax-like algorithms. For example, the following two publications:

Monte-Carlo Tree Search and minimax hybrids
Monte-Carlo Tree Search and Minimax Hybrids with Heuristic Evaluation Functions

The basic intuition behind such combinations tends to be to use small minimax-like searches inside a larger MCTS search, and/or backing up proven wins/losses (proven in the sense that minimax-like search has established that a certain state is a certain win or loss when following perfect play) through the MCTS tree, in addition to the more standard value estimates.

  but if possible I'd like to somehow import MC search into the engine I've already built.

I'm not sure how easy that would be. As I described above, I think it is more common that small minimax searches are used inside a larger MCTS search. I think that would mean that you'd first want to build an MCTS, and then try to import your already implemented minimax into it. That's kind of the opposite / other way around of what you write that you'd like to do.

I don't think integrating small MCTS searches inside a "larger" minimax would work well. The value estimates of small MCTS searches would be too unreliable for subsequent use in an "outer" minimax-like algorithm.


Pseudocode for CNN with Bounding Box and Classifier I've been looking at various bounding box algorithms, like the three versions of RCNN, SSD and YOLO, and I have noticed that not even the original papers include pseudocode for their algorithms. I have built a CNN classifier and I am attempting to incorporate bounding box regression, though I am having difficulties in implementation. I was wondering if anyone can whip up some pseudocode for any bounding box classifier or a link to one (unsuccessful in my search) to aid my endeavor.

Note: I do know that there are many pre-built and pre-trained versions of these object classifiers that I can download from various sources, I am interested in building it myself.

The minimal algorithm for convolution in $\mathbb{R}^2$ is a four dimensional iteration.

for all vertical kernel positions
  for all horizontal kernel positions
    initialize the value at the output position to the bias
    for all vertical positions in the kernel
      for all horizontal positions in the kernel
        add the product of the input value to that of the output position

In $\mathbb{R}^n$ it is a $2n$ dimensional iteration following this pattern.

The minimal algorithm for regression of bounding boxes orthogonal with respect to the image grid (no tilting) is this.

until number of boxes reaches max
  make first guess of two coordinates
  until number of guesses reaches max or matching criteria is met
    evaluate guess
    remember guess and guess results
    improve on guess based on evaluation results and
          possibly injected randomness,
          excluding locations already covered
    if some intermediate criteria is met
      change the nature of the guessing, evaluation, and improving
            as is appropriate for the criteria match
            (this covers approaches that have multiple phases)
  if no guess matched criteria
    break

That's approaching concepts from the top down. When approaching from the other direction, reverse engineer the best code. In the case of RCNN, it is unadvisable to find implementations following the first paper expressing the approach. Reading the first paper may be helpful to get the gist of the approach, but reverse engineer the best one, which, in this case, may be Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, 2016. Study the implementation they pushed to git at https://github.com/rbgirshick/py-faster-rcnn/tree/master/. The algorithm is in lib/fast_rcnn.

The reason this algorithm isn't spelled out in their paper or any paper from the first on down through the lineage to their paper is simple.

The pseudo-code above is universal across all convolutions and all bounding box regressions, so that doesn't need to be restated with each approach.
The main features of an approach like RCNN, SSD, or YOLO are not algorithmic. They are algebraic expressions of the guess, the evaluation, the improvement upon the guess, and the test for the criteria.
The use of objects and functional programming makes the implementation more readable, so it can be easier to read the implementation than read a huge chunk of the above pseudo-code with all the algebra and test branches plugged in.
For the above reasons, it is rare that pseudo-code would be used prior to the implementation when the paper is written.
The return on investment of reverse engineering from code to pseudo-code is only sufficient motivation if one is going to improve the algorithm and write another paper, and on the way to finishing the prior paper's pseudo-code, the new paper and the new code gets finished first.

Since the author of this question seems interested in writing their own code, it may be reasonable to assume the same author may be interested in thinking their own thoughts, so I'll add this.

None of these algorithms are object recognition. Recognition has to do with cognition, and these approaches do not even touch upon cognitive processing, another branch of AI not related to convolution and probably not closely related to formal regression either. Additionally, bounding boxes are not the way animal vision systems work. Early gestalt experiments in vision indicate a complete independence of human vision from rectilinear formalities. In lay terms, humans and other organisms with vision systems don't have any conception of Cartesian coordinates. We can still read books if tilted slightly relative to the plane passing through our eyes. We don't zoom or tilt in Cartesian coordinates.

These facts may not be necessary to comprehend to create an automated vehicle driving system that produces a better safety record than average human drivers, but that is only because humans don't set that bar very high and because cars roll in the plane of the road. These facts are indeed necessary in aeronautic system used in military applications, where nothing is particularly Cartesian and the meaning of horizontal and vertical is ambiguous. For that reason, it is unlikely that bounding boxes will be the edge of vision technology for very long.

If one wishes to transcend current mediocrity, consider bounding circles with fuzzy boundaries, which would be more like the systems that evolved over millions of biological iterations. If the computer hardware is poorly fit to radial processing, design new hardware in which radial processing is native and in which Cartesian coordinates may be foreign and cumbersome.

Regarding the classifier, the classifier papers do generally include the algorithm, so those can be found by doing an academic search for the original paper describing the classifier being used.


Is there a way to calculate the closed-form expression of the function that a neural network computes? As stated in the universal approximation theorem, a neural network can approximate almost any function.
Is there a way to calculate the closed-form (or analytical) expression of the function that a neural network computes/approximates?
Or, alternatively, figure out if the function is linear or non-linear?

To check if a function is linear is easy: if you can train one fully connected layer, without activations, of the right dimensions (for a function $\mathbb{R}^n \rightarrow \mathbb{R}^m$ you need $nm$ weights aka the matrix corresponding to the linear application), with enough data, to 100% accuracy... then it is linear.

The estimated function is explicit: it is given by the architecture of the NN and its weights. I don't think you can hope for an answer like "$\sin(\pi x + \phi) +4$" if it's not already in the predictive capacity of the NN. Interpretability of NNs is a hot topic. In what cases symbolic reasoning can lead to a simplified expression?


Can deep learning be used to help mathematical research? I am currently learning about deep learning and artificial intelligence and exploring his possibilities, and, as a mathematician at heart, I am inquisitive about how it can be used to solve problems in mathematics.

Seeing how well recurrent neural networks can understand human language, I suppose that they could also be used to follow some simple mathematical statements and maybe even come up with some proofs. I know that computer-assisted proofs are more and more frequent and that some software can now understand simple mathematical language and verify proofs (e.g. Coq). Still, I've never heard of deep learning applied to mathematical research.

Can deep learning be used to help mathematical research? So, I am curious about whether systems like Coq could be combined with deep learning systems to help mathematical research. Are there some exciting results?

Mathematical equations are generally expressed in a sequential form known as 'infix notation'.  It is characterised by the placement of operators between operands. To make the order of the operations in the Infix notation unambiguous, a lot of parenthesis are needed. Infix notation is more difficult to parse by computers than prefix notation (e.g. + 2 2) or postfix notation (e.g. 2 2 +).

There is a deep learning approach to symbolic mathematics recommended in the research paper by Guillaume Lample and François Charton. They have found an interesting approach to use deep neural networks for symbolic integration and differentiation equations. This paper proposes a syntax for representing mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models.

Deep Learning for Symbolic Mathematics

This approach is essentially representing mathematical problems in prefix notation. First a symbolic syntax tree is constructed that captures the order and values of the operations in the expression. Second, the tree is traversed from top to bottom and from left to right. If the current node is a primitive value (a number), add it to the sequence string. If the current node is a binary operation, add the operations symbol to the sequence string. Then, add the representation of the left child node (could be recursive). Then, add the representation of the right child node. This procedure resulted in the following expression.

We can expect further more advances in this area with the emergence of better symbolic learning models leveraging attention based transformers and other neural symbolic learning models. Recent work by MIT, DeepMind and IBM has shown the power of combining connectionist techniques like deep neural networks with symbolic reasoning. Please find the details in the following article.

The Neuro-Symbolic Concept Learner


In the machine learning literature, what does it mean to say that something is "embedded" in some space? In the machine learning literature, I often see it said that something is "embedded" in some space. For instance, that something is "embedded" in feature space, or that our data are "embedded" in dot product space, etc. However, I've never actually seen an explanation of what this is supposed to mean. So what does it actually mean to say that something is "embedded" in some space?

Embedding is the process of representing data (from a source domain) in a new (or target) domain. Usually, the source domain is discrete, and the target domain is continuous. For example, embedding words into the continuous vector space can be done by the word2vec method.
The main reason behind using the embedding is doing meaningful mathematical computations in the target domain, which is not possible or straightforward in the source domain. For example, summing two words "brother" - "man" + "woman" not meaningful in the word and character levels. However, when using word2vec, embedding("brother") - embedding("man") + embedding("woman") can be meaningful and comparable with other embedded vectors; It should be near the embedded vector of "sister".


What is a unified neural network model? In many articles (for example, in the YOLO paper, this paper or this one), I see the term "unified" being used. I was wondering what the meaning of "unified" in this case is.

A unified neural network model consists of one neural network as opposed to other models that rely on two or more neural networks.
For example, from page two of the YOLO paper:

2. Unified Detection
We unify the separate components of object detection into a single neural network. Our network uses features from the entire image to predict each bounding box. It also predicts all bounding boxes across all classes for an image simultaneously. This means our network reasons globally about the full image and all the objects in the image. The YOLO design enables end-to-end training and realtime speeds while maintaining high average precision.

In the paper by Xu and Wang, they add a branch to handle tracking to the Faster R-CNN architecture, which was designed for object detection.  This is a 'unification' of two models.
In the paper by Ebrahimi et al., on pages 42-43, you can see they are fusing multiple neural networks into one unified model:

We propose a unified user geolocation method that relies on a fusion of neural networks, incorporating different types of available information: tweet message, users' social relationships, and metadata fields embedded in tweets and profiles.


Finding the seed of a random number I'm new to machine learning, and AI in general (but with 20+ years for programming). I'm wondering if machine learning is a good general approach to find the seed of a random number generator.

Suppose I have a list of 2000 numbers. Is there a machine learning algorithm to correctly guess the next number?

Just to be clear, as there are many random number generator algorithms. I'm taking about rand and srand from the stdlib.

Thanks,
Eden

Machine Learning is a bad fit to this problem.

Even simple PRNGs that are not suitable for use in simulators (such as rand()) are varied enough that it is very hard to reverse engineer them statistically using generic techniques - essentially what 90% of ML does is fit a generic model to data statistically by altering parameters. The remaining 10% might do things in specialist manner, such as saving all the data and picking best option.

In theory most ML approaches would eventually solve a PRNG, however that would typically involve iterating through the entire state space of the PRNG multiple times. The statistical relationship between internal state, next state and output of a PRNG is complex by design, so that this is the only "black box" statistical approach, and this is clearly not feasible for any real implementation of a random number generator, which is going to have at least $2^{31}$ states on modern machines. Perhaps older 16-bit PRNGs, with a single value for state might be tractable.

An AI advanced enough to attempt to reverse engineer the output logically based on purely the data and researching how RNGs work is too advanced for current ML techniques to consider.

That leaves approaches that might try to construct a similar RNG, such as Genetic Programming (where the genome is converted to executable code). The trouble with this approach is there is no heuristic for a RNG that measures how close its output is to a target. A single bit of state difference or any tiny but meaningful change in generated RNG design will produce output that has no similarities with the target output whatsoever. Without such a measure you have no fitness function, and no way to attempt a guided search using the many discrete optimisation tools from AI.

Instead the usual approach to "breaking" a PRNG is to analyse the algorithm. Knowing the algorithm of many non-cryptographic PRNGs can allow predicting the internal state of the generator, sometimes in very few steps (for really simple Linear Congruential Generators that might be just a single step!).


Can neural networks be used to prove conjectures? Imagine I have a list (in a computer-readable form) of all problems (or statements) and proofs that math relies on.

Could I train a neural network in such a way that, for example, I enter a problem and it generates a proof for it?

Of course, those proofs then needed to be checked manually, but maybe the network then creates proofs from combinations of older proofs for problems yet unsolved.

Is that possible?

Would it be possible, for example, to solve the Collatz-conjecture or the Riemann-conjecture with this type of network? Or, if not solve, but maybe rearrange patterns in a way that mathematicians are able to use a new "proof method" to make a real proof?

Your idea may be feasible in general, but a neural network is probably the wrong high level tool to use to explore this problem.

A neural network's strength is in finding internal representations that allow for a highly nonlinear solution when mapping inputs to outputs. When we train a neural network, those mappings are learned statistically through repetition of examples. This tends to produce models that interpolate well when given data similar to training set, but that extrapolate badly.

Neural network models also lack context, such that if you used a generative model (e.g. an RNN trained on sequences that create valid or interesting proof) then it can easily produce statistically pleasing but meaningless rubbish.

What you will need is some organising principle that allows you to explore and confirm proofs in a combinatorial fashion. In fact something like your idea has already been done more than once, but I am not able to find a reference currently.

None of this stops you using a neural network within an AI that searches for proofs. There may be places within a maths AI where you need a good heuristic to guide searches for instance - e.g. in context X is sub-proof Y likely to be interesting or relevant. Assessing a likelihood score is something that a neural network can do as part of a broader AI scheme. That's similar to how neural networks are combined with reinforcement learning.

It may be possible to build your idea entirety out of neural networks in principle. After all, there are good reasons to suspect human reasoning works similarly using biological neurons (not proven that artificial ones can match this either way). However, the architecture of such a system is beyond any modern NN design or training setup. It definitely will not be a matter of just adding enough layers then feeding in data.


Should I use single or double view for gender recognition? My project requires gender recognition of people shown on the given images, with more than one person per image. However, these people can be positioned in frontal or side view(passing by perpendicularly, no face visible). On the pictures there will be entire bodies shown, not only the faces. My idea is to firstly use object detection to point where people would be and next use CNNs to recognize gender of each person.

My question is: should I use one object detection algorithm for both frontal and side views of a person and then classify them with one CNN, or should I use object detection to separately find people positioned in frontal and side manner and use two different CNNs, one for classification of frontal views and one for side views?

I am asking this because I think it might be easier for one NN to classify only one view at a time, because side view might have different features than frontal, and mixing this features might be confusing for a network. However I am not really sure. If something is unclear, please let me know.

[EDIT] Since problem might be hard to understand only by reading, I made some illustrations. Basically I wonder if using second option can help in achieveing better accuracy for the subtle differences like those in gender recognition, especially when face is not visible:

Single detection and classification:

Two different classifiers:

Img Source

I think your fundamental question is: can convolutional neural networks generalize to objects independent of the view?
The answer is mostly yes (given the dataset contains multiple views of objects). This is evident by looking at the results of various challenges: e.g. COCO object detection challenge results on youtube. You can see that no matter what the view is on the car, pedestrians, the detector is not biased towards any specific viewpoint.

Therefore, one can assume that you can build only one network to perform object detection and another network to perform classification.

If you really want to go even further:
- you can make a small change to the architecture of your detector (I guess you might be using something like SSD, YOLO or Faster-RCNN), in which you make gender classification for every bounding box prediction. If you think about it, it is intuitive because the detector is already doing classification (there is softmax + cross-entropy loss usually), you can just add another term in its tensor output and modify the loss. That way you don't even need another network! It would be much faster and simpler.
- you can predict the pose estimation of the object (and corresponding normals) with respect to the camera to capture the best viewpoint to perform classification.


How does Tara AI work? Cisco and other companies are using Tara AI—a matching tool that connects IT projects with freelancers who have the exact skills required to complete them.

Looking for an explanation of how Tara AI works.  (I read about it on Forbes but cannot find info about the AI methods utilized via Google search.)

This is a start-up company, not a software. There are many clients cannot find right freelancers in a short time, sometimes they take more than a month to find the right freelancer.
They use artificial intelligence to help businesses find the ones they expect. Tara AI's software helps businesses automatically scope out and manage projects with freelancers, monitoring the pace and quality of their work, and making sure contractors are paid on-time for their code commits, for example.
There are now a lot of companies that use AI technology for the same purpose, for example, Amazon, Facebook, Getdone or Google.


How do capsule neural networks work? Geoffrey Hinton has been researching something he calls "capsules theory" in neural networks. What is it? How do capsule neural networks work?

It appears to not be published yet; the best available online are these slides for this talk. (Several people reference an earlier talk with this link, but sadly it's broken at time of writing this answer.)

My impression is that it's an attempt to formalize and abstract the creation of subnetworks inside a neural network. That is, if you look at a standard neural network, layers are fully connected (that is, every neuron in layer 1 has access to every neuron in layer 0, and is itself accessed by every neuron in layer 2). But this isn't obviously useful; one might instead have, say, n parallel stacks of layers (the 'capsules') that each specializes on some separate task (which may itself require more than one layer to complete successfully).

If I'm imagining its results correctly, this more sophisticated graph topology seems like something that could easily increase both the effectiveness and the interpretability of the resulting network.


What are some implications of Gödel's theorems on AI research? Note: My experience with Gödel's theorem is quite limited: I have read Gödel Escher Bach; skimmed the 1st half of Introduction to Godel's Theorem (by Peter Smith); and some random stuff here and there on the internet. That is, I only have a vague high level understanding of the theory.

In my humble opinion, Gödel's incompleteness theorem (and its many related Theorems, such as the Halting problem, and Löbs Theorem) are among the most important theoretical discoveries.

However, its a bit disappointing to observe that there aren't that many (at least to my knowledge) theoretical applications of the theorems, probably in part due to 1. the obtuse nature of the proof 2. the strong philosophical implications people aren't willing to easily commit towards.

Despite that, there are still some attempts to apply the theorems in a philosophy of mind / AI context. Off the top of my head:

The Lucas-Penrose Argument: which argues that the mind is not implemented on a formal system (as in computer). (Not a very rigour proof however)
Apparently, some of the research at MIRI uses Löbs Thereom, though the only example I know of is Löbian agent cooperation.

These are all really cool, but are there some more examples? Especially ones that are actually seriously considered by the academic community.

See also What are the philosophical implications of Gödel's First Incompleteness Theorem?

Definitely there are a lot of implications for AI, including:

Inference with first-order-logic is semi-decidable. This is a big disappointment for all the folks that wanted to use logic as a primary AI tool.
Basic equivalence of two first-order logic statements is undecidable, which has implications for knowledge-based systems and databases. For example, optimisation of database queries is an undecidable problem because of this.
Equivalence of two context-free grammars is undecidable, which is a problem for formal linguistic approach toward language processing
When doing planning in AI, just finding a feasible plan is undecidable for some planning languages that are needed in practice.
When doing automatic program generation - we are faced with a bunch of decidability results, since any reasonable programming language is as powerful as a Turing machine.
Finally, all non-trivial questions about an expressive computing paradigm, such as Perti nets or cellular automata, are undecidable.


How do we know that now artificial intelligence will really take off? A lot of people are claiming that we are at an inflection point, and machine learning/artificial intelligence will take off. This is in spite of the fact that for a long machine learning has stagnated.

What are the signals that indicate that machine learning is going to take off?

In general, how do you know that we are at an inflection point for a certain technology?

This has happened in the past where people were really excited and saying things like we will have AI in decade or so. This is happening again. Not sure why people don't learn from history of AI. In both the cases what's happening is this - You develop a technique to solve a particular problem, you apply that technique, the technique seems to be general enough, people start to apply that same technique to various problems, people get excited that this is the silver bullet they were looking for, the technique starts to show its limitations and doesn't work for many problems, hype gets shattered, start over.


How to create a Q-Learning agent when we have a matrix as an action space? I have a 2-dimentional matrix as an action space, the rows being a resource to be allocated, and the columns are the users that we will allocate the resources to. (I built my own RL environment)
The possible actions are 'Zero' or 'One'. One if the resource was allocated to the user, Zero if not.
I have a constraint related to the resource allocation, which states that each resource can be allocated to one user only, and the resource should only be allocated to users who have requested a resource to be allocated to them, and that would be the state space which is another matrix.
A penalty would be applied if the agent violates the constraints and the episode would end and the reward would equal the penalty. Otherwise, the reward would equal the sum of all the users that were satisfied with the allocation.
I am struggling with the implementation. The agent starts by exploring, then little by little it starts exploiting. When it gets to be more exploitative, I've noticed that the action matrix's values are all set to 'One', and the penalty always has the same value from episode to episode.

I was thinking this strategy may work.
So, Q-learning takes vector input as state representation let's say your vector has n dimensions i.e. [$n_0$, $n_1$, $n_2$,..., $n_{n-1}$]
Now, from my interpretation you want to populate a matrix with 0 and 1's given the state vector but action-space has a high complexity e.g. an 8*8 matrix has 64 cells i.e. $2^{64}$ possible actions if you want the action to be a matrix.
I suggest this:
Fill each cell, one at a time. i.e. Your agent has only two possible action 0 and 1.
To indicate to your agent that you are at a specific cell, concatenate the row and column number to the state vector before passing it as input to the Q-learning agent.
Example:
If you original state-vector is [55, 22, 100, 4] and you have to fill cell at position (10, 30) of the matrix, the state-vector should be modified as follows:
[55, 22, 100, 4, 10, 30].
I'm not sure of the sample-efficiency of this approach though.


Several questions regarding the NEAT algorithm I've recently read the paper Evolving Neural Networks through Augmenting Topologies which introduces NEAT. I am now trying to prototype it myself in JavaScript. However, I stumbled across a few questions I can't answer.

What is the definition of "structural innovation", and how do I store these so I can check if an innovation has already happened before?

  However, by keeping a list of the innovations that occurred in the current generation, it is possible to ensure that when the same structure arises more than once through independent mutations in the same generation, each identical mutation is assigned the same innovation number

Is there a reason for storing the type of a node (input, hidden, output)?
In the original paper, only connections have an innovation number, but in other sources, nodes do as well. Is this necessary for the crossover? (This has already been asked here.)
How could I limit the mutation functions to not add recurrent connections?

What is the definition of "structural innovation", and how do I store these so I can check if an innovation has already happened before?

Structural innovation is anything added that changes the topology of the network. So a structural innovation is any added connection or added node. I don't want to get too much into the implementation, but something similar to a dictionary global variable will work. When adding a connection (or node) between two nodes, we check if a connection (or node) has ever been placed between these two nodes before. If it has then we have stored an innovation number for what this connection (or node) is identical and we set its innovation number to that. We identify each node by its number.

  Is there a reason for storing the type of a node (input, hidden, output)?

Yes, there are many structural innovations that we do not want to happen. An innovation between two input node or an innovation between two output nodes are two examples of structural innovations we don't want.

  In the original paper, only connections have an innovation number, but in other sources, nodes do as well. Is this necessary for crossover? (This has already been asked here.)

Although I am unsure if it is explicitly stated, it is implied that nodes have an identifying number. You can see this in Figure 2 and 3 of the paper where each node is labeled with a number.

  How could I limit the mutation functions to not add backpropagation connections?

By backpropagation connections I assume you mean recurrent connections (correct me if I am wrong). You can prevent recurrent connections in many ways, but one way is to maintain a partially ordered set. When adding a connection between two nodes you check to see the order of those two nodes in the partially ordered set. The one that comes first will be the node from which the connection starts, while the second will be the node at which the connection ends.

As a note, we can prevent recurrent connections but often times allowing them provides opportunities to find a better solution (as @PaulG comments)


Will parameter sweeping on one split of data followed by cross validation discover the right hyperparameters? Let's call our dataset splits train/test/evaluate. We're in a situation where we require months of data. So we prefer to use the evaluation dataset as infrequently as possible to avoid polluting our results. Instead, we do 10 fold cross validation (CV) to estimate how well the model might generalize.
We're training deep learning models that take between 24-48 hours each, and the process of parameter sweeping is obviously very slow when performing 10-fold cross validation.
Does anyone have any experience or citations for how well parameter sweeping on one split of the data followed by cross validation (used to estimate how well it generalizes) works?
I suspect it's highly dependent on the distribution of data and local minima & maxima of the hyper parameters, but I wanted to ask.

Depending on the number of features you have you might want to try reducing features in order to reduce overfitting and speed up your model. I assume you are using proper regularization as well. PCA may be an option to help if time is the main issue, medium PCA. As the medium article states PCA can be used to reduce the dimensionality of the data while keeping retaining variance. If you run a random forest on the different sets and the feature importance ranking is at all different that is a big issue and the number of features should be reduced. Ideally, the features should be cropped till they have the same (structure/feature importance) but slightly different weights associated with those features. Here is a article demonstrating feature selection.
Here are some publications I believe are relevant to your problem. More in line with your desire to try to validate after one sweep. You may want to look up applications of LSTM or Yolo to your model I have a feeling these technologies will help direct your final decisions on what to do.
Subspace regularization method for the single-trial estimation of evoked potentials (1999) by P.A. Karjalainen, J.P. Kaipio, A.S. Koistinen, M. Vauhkonen
On Quadratic Penalties in Elastic Weight Consolidation (2017) by Ferenc Huszár
Parameter space exploration with Gaussian process trees (2014) by  Robert B. Gramacy   University of California, Santa Cruz, CA
Herbert K. H. Lee   University of California, Santa Cruz, CA
William G. Macready
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017) by Nils Reimers, Iryna Gurevych
Evaluating Hospital Case Cost Prediction Models Using Azure Machine Learning Studio (2018) by Alexei Botchkarev -evaluation of regression performance


Recognising common short forms of appliances used daily I'm working on a chatbot for homes and would need to understand the appliance the user is talking about.

Suppose the command is:
Turn on the air conditioner.
and another variant could be
Turn on the ac.

Here ac and air conditioner refers to the same appliance but how would the chatbot know that?

This is a pretty broad topic!

The question you're interested in has several parts.

The first part is Named Entity Recognition. This NLP task focuses on identifying the types of named objects. In your case, you'd want to be able to tell that something is an appliance (or perhaps even, that it is an air conditioner).

But that's not enough on its own. You also need to be able to tell which appliance they're talking about! This is usually studied as a separate problem, called Entity Recognition.

Both of these are considered pretty hard, and there isn't likely to be one "right" answer for your specific problem, but there are viable techniques. The wikipedia articles I linked to should be good starting places and suggest some out-of-the-box toolkits.


If the normal equation works, why do we need gradient descent? Recently, I followed the open course CS229,
http://cs229.stanford.edu/notes/cs229-notes1.pdf
This lecturer introduces an alternative approach to gradient descent that is called "Normal Equation" and the equation is as follows:
$$\theta=\left(X^{T} X\right)^{-1} X^{T} \vec{y}$$
The normal equation can directly compute the $\theta$.
If the normal equation works, why do we need gradient descent? What is the trade-off between these two methods?

That normal equation is sometimes called the closed-form solution.
The short answer to your question is that the closed-form solution may be impractical or unavailable in certain cases or the iterative numerical method (such as gradient descent) may be more efficient (in terms of resources).
This answer gives you more details and an example.


Is there a way to protect humanity against the impending singularity? We are careening into the future which may hold unpredictable dangers in relation to AI. I've haven't yet heard of Chappie or Robocop style police robots, but militarized drone tech is replacing many conventional weapons platforms.  I love the idea that I may one day be able to transfer my consciousness to a computer, and improve my capabilities and potential.  However, what constitutes morality can differ greatly among individual humans.

How do we move forward toward the singularity in a way that protects humans, as opposed to possibly lead to our extinction?

What you are talking about is known as the Control Problem. We have our own tag for this specific topic here, which you can use for this and similar questions.

How to address the control problem is heavily discussed and still considered unsolved. Two of the important approaches are motivation control and capability control.

Motivation control aims at creating suitable reward functions that only reward behavior that's beneficial to the human race. This topic alone is so complex that many different competing theories exist. Minor errors in the definition of an AI's goal might lead to catastrophe.

Capability control aims at limiting the possible scope of action for the AI. The main problem here is, that most restrictions make the AI less useful. And to ensure robust control a lot of restrictions must be put in place. Finding the right tradeoff is a challenge we need to solve.

In addition to those two mechanisms we must think about how we create a superintelligence. One of the major aspects here is our understanding of the underlying technologies. If we create a self-optimizing neural network from scratch, which we do not understand in depth, it is very likely that we will not understand the potential threats before it is to late. Creating a digital copy of a human brain as a starting point on the other hand is a little saver, because we can assume that it will be driven by similar factors as a real human being. If that's desirable is a totally different discussion of course. But the general idea is, that understanding the motivation of a superintelligence is key for steering it in the right direction.

For more details about the topics I scratched in my answer and the control problem in general, I recommend reading Superintelligence by Nick Bostrom. He spends many chapters on those problems and possible solutions.

If you have more in-depth questions concerning specific topics, please open a new question to get a more detailed answer.


Can machine learning be used to pass the Turing test? Can we say that the Turing test aims to develop machines or methods to reach human-level performance in all cognitive tasks and that machine learning is one of these methods that can pass the Turing test?

Essentially yes.

The Turing Test is essentially a benchmark or challenge problem. It is a task that AI researchers would like to be able to solve.

Machine learning is a technique. It is a tool developed by AI researchers to solve various problems. Some kinds of machine learning are applicable to the Turing Test, but others are not. Machine learning is also applicable to a wide range of other problems.


Are these statements about the performance of neural networks as a function of the number of hidden layers contradictory? In this note Justin Domke says that

  In practice, neural networks seem to usually find a reasonable solution when the number of layers is not too large, but find poor solutions when using more than, say, 2 hidden layers.

But in Bengio's remark, he says

  Very simple. Just keep adding layers until the test error does not improve anymore.

There seems to be a conflict. Can anyone explain why they suggest differently? Or am I missing something?

There are many problems requiring more than two hidden layers. Randomly select a recent Google journal paper on deep learning, you'll see their network could have something like 5 (or more) hidden layers.

Justin Domke wrote his notes for students, so he probably tried to make his points as simple as possible. For a "typical" machine learning problem that students would most likely work on, two hidden layers should be sufficient. But that doesn't add up for a real practical problem. "Deep" learning usually mean more than two hidden layers.

Number of hidden layers is network design that nobody knows for sure. Yoshua Bengio's suggestion is common and simple. It's not a mathmatic proof, but simply a guideline if you don't know what to  do. You just repeat and repeat, until you see the test error no longer improve.


Why do you have to register a bot with Azure? I am (planning on) using the Microsoft bot framework to build a chatbot I am working on.  However I simply cannot wrap my head around why the chatbot needs to be registered with Azure.

My line of expectation going into this was the same as any other .NET project.  I build it, I deploy it to my own server and I live happily ever after.

I have bought two books on the subject and yet after going through online videos, the books and articles I still cannot see a good reason for why one is forced to register the bot.   I feel like Lisa Simpson in that episode of The Simpsons where there is a puzzle that is super easy and everyone understands it but her.  I know I am being blinded by my own assumptions and expectations going into this.

If anyone can explain the 'why' of the registration that will really help.  I am also wondering if Azure went down like AWS did if that would stop the bot working?

My backup plan is just to write the bot using ASP.NET web API and then connect to it from a web container and facebook.  Neither of which needs the registration.

Thanks,

-Mike

These bots depend on heavy NLP services that are provided via Azure. Implementing/deploying your own production-quality versions of these is nontrivial, if not impossible (since you don't have all the training data, internal algorithms, etc. that Microsoft/Amazon/Google/IBM et al have), and is generally non-feasible on home-grade devices.

And yes, that makes the relevant Azure services a dependency that your chatbot will have, and if you use that service you'll need to be aware of their SLAs.

Plus, they'd like to charge you money. :-)


What causes a model to require a low learning rate? I've pondered this for a while without developing an intuition for the math behind the cause of this.

So what causes a model to need a low learning rate?

Gradient Descent is a method to find the optimum parameter of the hypothesis or minimize the cost function.

where alpha is learning rate

If the learning rate is high then it can overshoot the minimum and can fail to minimize the cost function.

hence result in a higher loss.

Since Gradient descent can only find local minimum so, the lower learning rate may result in bad performance. To do so, it is better to start with the random value of the hyperparameter can increase model training time but there are advanced methods such as adaptive gradient descent can manage the training time.

There are lots of optimizer for the same task but no optimizer is perfect. It depends on some factors

size of training data: as the size of the training data increases training time for model increases. If you want to go with less training model time you can choose a higher learning rate but may result in bad performance.
Optimizer(gradient descent) will be slow down whenever the gradient is small then it is better to go with a higher learning rate.

PS. It is always better to go with different rounds of gradient descent


What exactly are genetic algorithms and what sort of problems are they good for? I've noticed that a few questions on this site mention genetic algorithms and it made me realize that I don't really know much about those.

I have heard the term before, but it's not something I've ever used, so I don't have much idea about how they work and what they are good for. All I know is that they involve some sort of evolution and randomly changing values.

Can you give me a short explanation, preferably including some sort of practical example that illustrates the basic principles?

Evolutionary algorithms are a family of optimization algorithms based on the principle of Darwinian natural selection. As part of natural selection, a given environment has a population of individuals that compete for survival and reproduction. The ability of each individual to achieve these goals determines their chance to have children, in other words, to pass on their genes to the next generation of individuals, who, for genetic reasons, will have an increased chance of doing well, even better, in realizing these two objectives.
This principle of continuous improvement over the generations is taken by evolutionary algorithms to optimize solutions to a problem. In the initial generation, a population composed of different individuals is generated randomly or by other methods. An individual is a solution to the problem, more or less good: the quality of the individual in regards to the problem is called fitness, which reflects the adequacy of the solution to the problem to be solved. The higher the fitness of an individual, the higher it is likely to pass some or all of its genotype to the individuals of the next generation.
An individual is coded as a genotype, which can have any shape, such as a bit vector (genetic algorithms) or a vector of real (evolution strategies). Each genotype is transformed into a phenotype when assessing the individual, i.e. when its fitness is calculated. In some cases, the phenotype is identical to the genotype: it is called direct coding. Otherwise, the coding is called indirect. For example, suppose you want to optimize the size of a rectangular parallelepiped defined by its length, height and width. To simplify the example, assume that these three quantities are integers between 0 and 15. We can then describe each of them using a 4-bit binary number. An example of a potential solution may be to genotype 0001 0111 1010. The corresponding phenotype is a parallelepiped of length 1, height 7 and width 10.
During the transition from the old to the new generation, the variation operators, whose purpose is to manipulate individuals, are applied. There are two distinct types of variation operators:

the mutation operators, which are used to introduce variations within the same individual, as genetic mutations;
the crossover operators, which are used to cross at least two different genotypes, as genetic crosses from breeding.

Evolutionary algorithms have proven themselves in various fields such as operations research, robotics, biology, nuance, or cryptography. In addition, they can optimize multiple objectives simultaneously and can be used as black boxes because they do not assume any properties in the mathematical model to optimize. Their only real limitation is the computational complexity.


Has anyone been able to solve OpenAI's hardcore bipedal walker with their implementation of DDPG? As the question suggests, I'm trying to see if I can solve OpenAI's hardcore version of their gym's bipedal walker using OpenAI's DDPG algorithm.

Below is a performance graph from my latest attempt, including the hyper parameters, along with some other attempts I've made. I realise it has been solved using other custom implementations (also utilising only dense layers in Tensorflow, not convolution), but I don't seem to understand why it seems so difficult to solve using OpenAI's implementation of DDPG? Can anyone please point out where I might be going wrong? Thank you so much for any help!

Latest attempt's performance:

Average score: about -75 to -80
Env interacts: about 8.4mil (around 2600 epochs)
Batch size: 64
Replay memory: 1000000
Network: 512, 256 (relu activation on inputs, tanh on outputs)
All other inputs left to default

Similar experiments yielded similar scores (or less), and included:

Network sizes of (400,300), (256,128), and (128,128,128)
Number of epochs ranging from 500 all the way to 100000
Replay memory sizes all the way up to 5000000
Batch sizes of 32, 64, 128, and 256
All of the above, with both DDPG as well as TD3

Thank you so much for any help! It would be greatly appreciated!

You may be very interested to know that there was a bug in the v2 Lidar tracing, making the agent think there were phantom objects, and sometimes intersecting with its own legs:

https://github.com/openai/gym/pull/1789

  Finding this bug makes me even more impressed anyone has solved BipedalWalkerHardcore-v2 - it seems the observations from lidar have been inconsistent and incorrect, returning the furthest hit result instead of closest.

...

  Before fix - lidar traces through ground, and hits the side of a pit,
  giving the agent the impression of a "phantom canyon" in front of the
  pit, that only appears as it approaches the pit:
  https://i.imgur.com/XKPnRTR.png

...

  After fix - lidar is stopped by terrain, even when another object is behind it: https://i.imgur.com/Gg8B5BD.png

...

  After triple checking the docs - I've submitted a minor tweak (returning -1 instead of 1 for an object that should be ignored) - it now seems legs are correctly ignored, and the traces are accurate in all situations!

...

  It seems to me that solutions to BipedalWalkerHardcore-v2 have not just learned to deal with the complex environment - but advanced a step ahead, and are able to deal with the complex environment and sensory hallucinations causing them to jump at the slightest hint of a cube, and keep running even when it looks like the ground is not visible below their feet, relying more on the touch sensor than the lidar, or perhaps recognising the difference in "shape" between a real pit and a "fake pit" (A real pit has a floor)

BipedalWalkerHardcore-v2 has been bumped to BipedalWalkerHardcore-v3 with these fixes as of Jan 31, 2020.

You might want to try retraining your agent now! (although it is still a difficult environment to solve)

To expand on why DDPG doesn't solve it, when although buggy, BipedalWalkerHardcore-v2 is solvable: The solution landscape to this problem is as full of pits as the environment itself. To learn to leap over a pit in the environment for example, the agent must perform a complex sequence of actions that is difficult to discover by random chance. Each time it fails, it learns that being close to a pit is highly likely to result in a large penalty, and in an effort to maximise it's rewards, will often remain stationary with a naive method like DDPG as the rewards for doing that are higher than trying and falling into the pit once more. In short, vanilla DDPG lacks enough exploration power to find the complex series of actions required before it converges on not going near the pit. Not to mention all of the other things it needs to learn to be successful.

Of the very few published examples that have solved it, one used Evolutionary Strategy - a gradient free method essentially trying millions of policies and evaluating them, and a custom A3C method that was tailored to solve this particular environment. Both had high computational and sample requirements.
I speak from experience as I have personally solved this environment with a RL exploration algorithm that generalises to other environments, solves it in 4 hours on a single cpu, and can be used with any off-policy RL algorithm, but unfortunately I'm unable to publish it because of IP with the company I work for.

TLDR; The chance of vanilla DDPG solving it are infeasibly small.


What are the advantages of RL with actor-critic methods over actor-only methods? In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods?
This is not a comparison with the Q-learning series, but probably a method of learning the game with only the actor.
I think it's effective to use only actors, especially for sparse rewards. Is that correct?
Please, let me know if you have any specific use cases that use only actors.

In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods?

One practical benefit is that critics can use TD learning to bootstrap, allowing them to learn online on each step taken, plus learn in continuing problems. Pure actor algorithms like REINFORCE, cross-entropy method, and non-RL policy-only learners, such as genetic algorithms, require episodic problems. The smallest unit those can learn from is an entire episode. That is because without a critic providing value estimates, the only way to estimate return is to sample an actual return from the end of an episode.
A TD-based critic may also have lower variance, which can aid in fast learning and stability, although this is not always a benefit. TD-based critics also have bias, which can cause instability. See Why do temporal difference (TD) methods have lower variance than Monte Carlo methods? on Cross Validated for more details on this.
In practice, RL algorithm choice is a hyperparameter. As well as impacting difficulty of implementation, CPU and other resource costs, it can impact how well learning occurs depending on the problem being attempted. Usually, the only way you can tell a method is better for your problem is to try all the valid ones and measure their performance.

I think it's effective to use only actors, especially for sparse rewards. Is that correct?

The sparsity of reward is not a major factor here. The hard credit assignment problem that this brings into play means that the agent has to either assign a value, or pick an action, in states which have no direct feedback. All else being equal, both the value function and optimal action choice are hard to resolve when they depend on a large number of future policy decisions and state transitions which may vary.
Which approach is best will depend on whether it is easier for a statistics-based learner to approximate the mapping from state to expected future reward, or to the action. The two functions can have different degrees of complexity.
For an example where a policy function (for an actor) is simpler than a value function (for a critic or value-based method), you could consider a simple chase environment where a wolf tries to catch a rabbit on a simple continuous plane surface. The agent is the wolf, and gets a reward of +1 for catching the rabbit (I won't bother with other details, there are plenty of variations you could make).
In the example environment, a simple strategy for the wolf is to turn to face the rabbit and move forwards. This is easy to map from current locations and facings of the wolf and the rabbit - for ultimate ease you can express the state as the rabbit's position and velocity in polar coordinates from the wolf's perspective. Compare that to the value function - it has to predict the time it will take to reach the rabbit given a current action choice. This is a far harder function to express based on the state, and as a result may also be harder to learn.


Is there any website that allows you to choose an algorithm, code it and visualise how it works? I would like to do some practical implementation of a planning algorithm (of course, something a bit simple and easy).
Is there any website where I can pick an algorithm (e.g. A* or hill climbing), code it, and visualize how it works/executes?
The site doesn't necessarily need to be restricted to planning or search algorithms.  For example, in the context of machine learning, I would also like to be able to pick the learning algorithm and model (e.g. linear regression), code it, and visualize how it works.

Not planning, but this is a visual in-browser neural network for your interest:

http://playground.tensorflow.org/


Training the generator in a GAN pair with back propagation For the purposes of this question I am asking about training the generator, assume that training the discriminator is another topic.

My understanding of generative adversarial networks is that you feed random input data to the generator and it generates images. Out of those images, the ones which the discriminator thinks are real are used to train the generator.

For example, I have the random inputs $i_1$, $i_2$, $i_3$, $i_4$... from which the generator produces $o_1$, $o_2$, $o_3$, $o_4$. Say for example, the discriminator thinks that $o_1$ and $o_2$ are real but $o_3$ and $o_4$ are fake, I then throw away input output pairs 3 and 4, but keep 1 and 2, and run back propagation on the generator to tell it that $i_1$ should produce $o_1$, and $i_2$ should produce $o_2$ since these were "correct" according to the discriminator.

The contradiction seems to come from the fact that the generator already generates those outputs from those inputs, so nothing will be gained by running backprop on those input output pairs.

Where is the flaw in my logic here? I seem to have something wrong in my reasoning, or a misunderstanding of how the generator is trained.

Let's take a look at the images that fooled the D-network. When this happens the binary cross entropy loss of the D-network is important to look at. The network said a fake image was real. As usual, you back propagate to the very beginning of the D-network. What is the back prop doing? It is telling the weights in the D-network to change such that you lessen the loss (remember you are minimizing the loss function by going down hill --> using gradient descent). So you changed the weights of the D-network, but you really don't have to stop there.

The complex nonlinear function the D-network is calculating is really f(Image, weights of D-network). So the output of the D-network is really dependent on the Image fed in and the weights of the D-network. The cost is a function of the output f. So we have c = g(f(Image, weights of D-network)) where g is really the binary cross entropy. Looking at this, we see that the cost is also a function of the image!

We can take the partials with respect to the image just like we take the partials with respect to the weights to update the weights. And so after doing back prop to the image, we don't stop -> we take the partial of the cost function with respect to the image that is fed into the D-network and we update the image. And essentially we don't have the original image when we continue to back prop to the beginning of the G-network. Let me know if you'd like more clarification. Again this is my understanding and I haven't coded a GAN yet, so someone please confirm. Also, I am open to others' interpretations.


What are the benefits of using max-pooling in convolutional neural networks? I am reading Francois Chollet's Deep learning with Python, and I came across a section about max-pooling that's really giving me trouble.

I am unable to copy-paste the content, so I've included screenshots of the paragraph that's troubling me.

I simply don't understand what he means when he talks about "What's wrong with this setup?" (towards the end).

How does removing the max-pooling layers "reduce" the amount of the initial image that we're looking at? What are the benefits of using max-pooling in convolutional neural networks, as opposed to just using convolution layers?

MaxPooling pools together information. Imagine you have 2 convolutional layers $(F_1, F_2)$ respectively, each with a 3x3 kernel and a stride of $1$. Also, imagine your input is $I$ is of shape $(w,h)$. Let's call a max-pooling layer $M$ is of size $(2,2)$.

Note: I'm ignoring channels because, for these purposes, it's not necessary and can be extended to any amount of them.

Now you have two cases:

$O_1 = F_2 \circ F_1 \circ I$
$O_2 = F_2 \circ M \circ F_1 \circ I$

In these cases, $shape(O_1)=(w-4, h-4)$ and $shape(O_2)=\left(\frac{w-2}{2}-2, \frac{h-2}{2}-2 \right)$. If we plug in dummy values, like $w,h = 64,64$, we get the shapes become $(60,60)$ and $(29,29)$ respectively. As you can tell, these are very different!

Now, there is more of a difference than just the size of the outputs, each neuron holds a pooling of more information. Let's do it out:

Each output neuron of $F_1 \circ I$ has information from a $(3,3)$ receptive field.
Each output neuron then of $F_2 \circ F_1 \circ I$ has information from a $(3,3)$ receptive field of $F_1 \circ I$, which, if we eliminate reused nodes, is a $(5,5)$ receptive field from the initial $I$.
Each output neuron then of $M \circ F_1 \circ I$ has information from a $(2,2)$ receptive field of $F_1 \circ I$, which, if we eliminate reused nodes, is a $(4,4)$ receptive field from the initial $I$.
Each output neuron then of $F_2 \circ M \circ F_1 \circ I$ has information from a $(4,4)$ receptive field of $M \circ F_1 \circ I$, which, if we eliminate reused nodes, is a $(8,8)$ receptive field from the initial $I$.

So, let us discuss these: Using max-pooling reduces the feature space heavily by throwing out a lot of nodes whose features aren't as indicative (makes training models more tractable) along with it does extend the receptive field with no additional parameters.


How do I choose the best algorithm for a board game like checkers? How do I choose the best algorithm for a board game like checkers?

So far, I have considered only three algorithms, namely, minimax, alpha-beta pruning, and Monte Carlo tree search (MCTS). Apparently, both the alpha-beta pruning and MCTS are extensions of the basic minimax algorithm.

tl;dr:

None of these algorithms are practical for modern work, but they are good places to start pedagogically.
You should always prefer to use Alpha-Beta pruning over bare minimax search.
You should prefer to use some form of heuristic guided search if you can come up with a useful heuristic. Coming up with a useful heuristic usually requires a lot of domain knowledge.
You should prefer to use Monte Carlo Tree search when you lack a good heuristic, when computational resources are limited, and when mistakes will not have outsize real-world consequences.

More Details:

In minimax search, we do not attempt to be very clever. We just use a standard dynamic programming approach. It is easy to figure out the value of difference moves if we're close to the end of the game (since the game will end in the next move, we don't have to look very far ahead). Similarly, if we know what our opponent will do in the last move of the game, it's easy to figure out what we should do in the second last move. Effectively we can treat the second last move as the last move of a shorter game. We can then repeat this process. Using this approach is certain to uncover the best strategies in a standard extensive-form game, but will require us to consider every possible move, which is infeasible for all but the simplest games.

Alpha-Beta pruning is a strict improvement on Minimax search. It makes use of the fact that some moves are obviously worse than others. For example, in chess, I need not consider any move that would give you the opportunity to put me in checkmate, even if you could do other things from that position. Once I see that a move might lead to a lose, I'm not going to bother thinking about what else might happen from that point. I'll go look at other things. This algorithm is also certain to yield the correct result, and is faster, but still must consider most of the moves in practice.

There are two common ways you can get around the extreme computational cost of solving these kinds of games exactly:

Use a Heuristic (A* search is the usual algorithm for pedagogical purposes, but Quiescence search is a similar idea in 2 player games). This is just a function that gives an estimate of the value of a state of the game. Instead of considering all the moves in a game, you can just consider moves out to some finite distance ahead, and then use the value of the heuristic to judge the value of the states you reached. If your heuristic is consistent (essentially: if it always overestimates the quality of states), then this will still yield the correct answer, but with enormous speedups in practice.
Use Rollouts (like Monte Carlo Tree Search). Basically, instead of considering every move, run a few thousand simulated games between players acting randomly (this is faster than considering all possible moves). Assign a value to states equal to the average win rate of games starting from it. This may not yield the correct answer, but in some kinds of games, it performs reliably. It is often used as an extension of more exact techniques, rather than being used on its own.


Is AI and Big Data science recommending a shift in the scientific method from inductive to deductive reasoning? Is this true?  Are we planning to switching such reasoning methods regarding AI tech in the future?

You could argue that AI and big data are trying to switch the AI method from deductive to inductive reasoning in the sense that original AI was deductive (if...then conditionals) but deep learning implies inductive reasoning (feed the network a million images of white swans and the network will "conclude" all swans are white - the classic example of (erroneous) inductive reasoning.)


How does Ubers rating System algorithm work? i've been trying lately to search the internet to find a result for this but no useful results, unfortunately, we're making an uber-like application during our discussion we needed a way to validate a rider's rating after he completes his ride with a taxi, but the main question is, does uber use an algorithm to do so? and how?

I highly recommend that you start reading on the Netflix challenge. It has tonnes of useful and interesting examples dealing with this sort of thing.
You will need an Algorithm that builds a score on both 'quality' and 'quantity'. That is, it needs to add a 'weight' to the final rating based on the number of reviews that an individual has. This is so that, for example, an individual with 50 8 score reviews would be rated higher than an individual with only one 9 score review.
I recommend that you implement Bayesian estimates to calculate weighted voting.
IMDb (Internet Movie Database) utilizes this algorithmn to determine its IMDB top 250 movies. (Robert C 2010)

The formula for calculating the Top Rated 250 Titles gives a true Bayesian estimate:
weighted rating (WR) = (v ÷ (v+m)) × R + (m ÷ (v+m)) × C
where:
R = average for the movie (mean) = (Rating)
v = number of votes for the movie = (votes)
m = minimum votes required to be listed in the Top 250 (currently 3000)
C = the mean vote across the whole report (currently 6.9)

Please note that in addition to the rating Algorithm, Uber has a dispatch algorithm which takes into consideration factors i.e. Drivers who are online and drivers who are nearest to the passenger.


What is the benefit of using identity mapping layers in deep neural networks like ResNet? As I understand, ResNet has some identity mapping layers, whose task is to create the output as the same as the input of the layer. The ResNet solved the problem of accuracy degrading. But what is the benefit of adding identity mapping layers in intermediate layers?
What's the effect of these identity layers on the feature vectors that will be produced in the last layers of the network? Is it helpful for the network to produce better representation for the input? If this expression is correct, what is the reason?

TL;DR: Deep networks have some issues that skip connections fix.

To address this statement:

  As I understand Resnet has some identity mapping layers that their task is to create the output as the same as the input of the layer

The residual blocks don't strictly learn the identity mapping. They are simply capable of learning such a mapping. That is, the residual block makes learning the identity function easy. So, at the very least, skip connections will not hurt performance (this is explained formally in the paper).

From the paper:

Observe: it's taking some of the layer outputs from earlier layers and passing their outputs further down and element wise summing these with the the outputs from the skipped layers. These blocks may learn mappings that are not the identity map.

From paper (some benefits):

  $$\boldsymbol{y} = \mathcal{F}(\boldsymbol{x},\{W_i\})+\boldsymbol{x}\quad\text{(1)}$$The shortcut connections in Eqn.(1) introduce neither extra parameter nor computation complexity. This is not only attractive in practice but also important in our comparisons between plain and residual networks. We can fairly compare plain/residual networks that simultaneously have the same number of parameters, depth, width, and computational cost (except for the negligible element-wise addition).

An example of a residual mapping from the paper is $$\mathcal{F} = W_2\sigma_2(W_1\boldsymbol{x})$$

That is $\{W_i\}$ represents a set of i weight matrices ($W_1,W_2$ in the example) occurring in the layers of the residual (skipped) layers. The "identity shortcuts" are referring to performing the element wise addition of $\boldsymbol{x}$ with the output of the residual layers.

So using the residual mapping from the example (1) becomes:

$$\boldsymbol{y} = W_2\sigma_2(W_1\boldsymbol{x})+\boldsymbol{x}$$

In short, you take the output $\boldsymbol{x}$ of a layer skip it forward and element wise sum it with the output of the residual mapping and thus produce a residual block.

Limitations of deep networks expressed in paper:

  When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.

The skip connections and hence the residual blocks allow for stacking deeper networks while avoiding this degradation issue.

Link to paper

I hope this helps.


How do you know if an agent has learnt its environment in reinforcement learning? I'm new to reinforcement learning and trying to understand it.

If you train an agent using a reinforcement learning algorithm (discrete or continuous) on an environment (real or simulated), then how do you know if the agent has learnt its environment? Should it reach its goal on every run (episode)? (Any literature references are also welcome)

Is this related to the reward threshold defined in the environment?

What happens if you continue training after the agent has learnt the environment? Will it perform by reaching its goal every time or will there be failed episodes?

This depends on the complexity of the environment being learned, and the purpose for learning it. There is no general answer.

For the simple environments used to teach reinforcement learning (RL), often the optimal solution is obvious, or can be calculated and proven optimal. For instance, any environment that can be solved using policy iteration will have a known optimal policy and optimal value function. The goals of these environments are to teach, or to check correctness of agents - it helps in these cases to have a well known correct answer.

At the next level up in terms of complexity are well-studied environments which can have achievable targets set for learning agents. The goals of these environments include getting useful metrics for learning agents, such as how many episodes it takes a particular implementation to learn it well enough. Defining "well enough" is a matter of experience with existing agents.

Getting more complex still, in general it is not possible to know whether an agent has fully optimised against its environment. The subject area of sequential decision making that includes RL agents can cover scenarios such as driving a car or playing a computer game. We don't know when any agent, whether they are based on RL or some other approach, has fully learning an environment, and instead must construct tests of behaviour - e.g. make an agent simulate driving in a set of scenarios, and expect at least safe behaviour in each of them, essentially a driving test similar to one a person might take. In these environments, often the tests are based on "good enough to use" goals. We can say an agent has learned to drive if it drives more safely than an average human.

In the special case of competitive games, we can score agents against each other or against human players. You might say that an agent has learned its environment if it beats some standard player, but also you can rank agents against each other and declare a particular agent as the current best.

It is possible to mix and match these ideas. The Atari games learning suite has benchmark scores to reach that count as "standard human", and recently agents have been published that beat all of those scores.

  What happens if you continue training after the agent has learnt the environment? Will it perform by reaching its goal every time or will there be failed episodes?

If you include training episodes, then RL learns mainly by "trial and error". So you should expect an agent to make deliberate mistakes as it tests to see what happens. In some environments these could be critical mistakes leading to failed episodes.

If you ignore the training episodes and are interested only in performance without exploration - e.g. testing every few hundred episodes - then you can expect performance to vary depending on the type of agent and environment. Some agents even exhibit "catastrophic forgetting", which as the name implies causes performance to drop significantly - this can be caused by a successful agent over-fitting to all the recent successful episodes without errors it just experienced, and losing the ability to predict the true lower value of incorrect actions.

Neither failed episodes during training nor catastrophic forgetting are inevitable. It depends on the environment and type of agent.


2 Player Games in OpenAI Retro I have been using OpenAI Retro for awhile, and I wanted to experiment with two player games. By two player games, I mean co-op games like "Tennis-Atari2600" or even Pong, where 2 agents are present in one environment.

There is a parameter for players in the OpenAI documentation, but setting this variable to 2 does nothing in terms of the game.

How do you properly implement this? Can this even be done?
The end goal is to have 2 separate networks per one environment.

OpenAI Retro is an extension to OpenAI Gym.

As such, it does not support multiple agents in multi-player environments. Instead, the environment is always presented from the perspective of a single agent that needs to solve a control problem.

When there is more than one agent in a Gym problem, everything other than the first agent is considered part of the environment. Typically player 2 in 2-player problems is controlled by whatever game AI already exists.

You can work around this in a couple of ways, both involve you modifying existing environments, requiring a solid understanding of the integration and wrapper layers in Gym so that you can make correct edits. However you might find environments where either of these are done for you.

In fact, within the Retro environments, some multiplayer environments are supported using the second option below.

1. Supply additional player(s) as part of the environment

You could modify the environment code so that it accepts a parameter which is the agent used to control other players. Typically such an agent would not be set up to learn, it could just be some earlier version of the agent that is learning. You would need to code the interface between the supplied agent and the game engine - taking care to allow for its potentially different view of success.

If a Gym environment has been set up for you (or any human) to compete against your own agent using the keyboard, that is already halfway there to this solution. You could take a look at where the Gym environment has been set up for this, and modify the code to allow for automated input as opposed to action selection by keyboard.

This approach probably works best for competitive games where you can use a random selection of previous agents inserted into the environment to train a next "generation".

2. Present actions for multiple players as if there was a single player

Provided the game emulator allows you to do this, you can edit the code so that multiple players are controlled at once. The Gym environment does not specify how you construct your action choices, so you are then free to split the action vector up so that parts of it are chosen by one trained agent and parts of it by another.

You may need to work on different views of the environment and/or reward signal in this case. For instance in a competitive game (or a co-operative game with competitive elements such as personal scores) then you will need to add custom code to re-interpret reward values compared to single player game. For something simple like Pong, that could just be that Player 2 receives the negative of Player 1's reward signal.

This approach probably works best for cooperative games where success is combined. In that case, you have the choice of writing any number of separate agents that control different aspects of the action space - that would include having agents that separately controlled P1 and P2 game controller inputs.

Each agent would still effectively treat its partner as a part of the environment, so you may want to mix training scenarios so that agents for each controller did not get too co-dependent. For instance, if you hope to step in to control P1, you probably don't want the AI running P2 to have too specific expectations of P1's role and behaviour in the game.

This second option is what OpenAI Retro interface does. For instance Pong supports this:

env = retro.make(game='Pong-Atari2600', players=2)

I verified this using the sample code in the documentation, which controls both paddles, and receives the reward signal as a Python list mostly [0.0, 0.0] but whenever a player scores it becomes [-1.0, 1.0] or [1.0, -1.0] depending on which player got the point.

This will only be supported on retro games where someone has made the effort to make it work as above. If you have a game which does not do this, but in theory supports multiplayer on the ROM, you will need to look at and alter either the AI retro wrapper or the emulator or both.


Why is a mix of greedy and random usually "best" for stochastic local search? I read that a mix of "greedy" and "random" are ideal for stochastic local search (SLS), but I'm not sure why. It mentioned that the greedy finds the local minima and the randomness avoids getting trapped by the minima. What is the minima and how can you get trapped? Also, how does randomness avoid this? It seems like if it's truly random there's always a chance of ending up searching solutions that lead to dead ends multiple times (which seems like a waste of processing and avoidable)?

As an example of local/global minima, imagine being on a rugged, mountainous landscape, and you want to find the lowest point within some area. For a greedy search, every step you take will take you downhill. If you go downhill long enough, you'll eventually find a flat spot, which is a minimum - from here, there's no step you can take that will get you any lower. However, there's a nearby ridge, which if you crossed it, you could continue downhill to find an even lower spot, the global minimum (the true lowest point). Using your greedy approach, you'll never go uphill to cross the ridge, so you'll be stuck in the local minimum forever. If you occasionally take random steps (other than directly downhill), you have the opportunity to cross ridges that separate local minima, and you have a better chance of finding the global minimum. You are correct that in many cases, the random step won't help you cross a ridge, and will just take you up a mountain in the wrong direction, which is a waste of time. But unless we allow the algorithm to "explore" a bit, it will be content that the first minimum it finds is the best one, and will never get to the bottom.


How to detect frauds in advertising business using machine learning? I am very beginner to this world. I still learning the basics of Machine learning and AI but i have a problem at hand and i am not sure which technique or Algorithm can be applied on it.

I am working on Click-Fraud detection in advertising. I need to predict fraud and learn new frauds with ML.

The dataset I have is the view and click logs from adserver(Service Provider). This data have some fields few of them are listed below:

"auction_log_bid_id": null,
"banner": 9407521,
"browser": 0,
"campaign": 2981976,
"city": 94965,
"clickword": null,
"content_unit": 4335438,
"country": 1,
"external_profiledata": {},
"external_user_id": null,
"flash_version": null,
"id": 6665230893362053181,
"ip_address": "80.187.103.98",
"is_ssl": true,
"keyword": "string"
"mobile_device": -1,
"mobile_device_class": -1,
"network": 268,
"new_user_id": 6665230893362118717,
"operating_system": 14,
"profile_data": {},
"referrer": null,
"screen_resolution": null,
"server_id": 61,
"state": 7,
"target_url": "string"
"timestamp": 1551870000,
"type": "CLICK_COMMAND",
"user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/16D57",
"user_id": null,
"view_log_id": null

There are other fields.

I need to analyse these logs to find patterns for possible frauds but i am not sure where to start and which technique to use. e.g. Supervised, Unsupervised Semi-Supervised or Reinforcement Learning.

There are a couple of different ways you can go about this depending on what kind of data you have.

If you have labels or can separate the normal data from the fraudulent, you can perform either binary classification, or likely more usefully, anomaly detection.

In Anomaly Detection(which is typically now done via autoencoder) you train your model on the normal data, so it learns a compressed representation of that 'signal', from there it will be able to detect any sample that does not fit the learned representation(in theory).

Here is a link to a tutorial in keras: link


Object Detection: Can I modify this script to support larger images (Scaled YOLOv4)? I am looking at training the Scaled YOLOv4 on TensorFlow 2.x, as can be found at this link. I plan to collect the imagery, annotate the objects within the image in VOC format, and then use these images/annotations to train the large-scale model. If you look at the multi-scale training commands, they are as follows:
python train.py --use-pretrain True --model-type p5 --dataset-type voc --dataset dataset/pothole_voc --num-classes 1 --class-names pothole.names --voc-train-set dataset_1,train --voc-val-set dataset_1,val  --epochs 200 --batch-size 4 --multi-scale 320,352,384,416,448,480,512 --augment ssd_random_crop

As we know that Scaled YOLOv4 (and any YOLO algorithm at that) likes image dimensions divisible by 32, I have plans to use larger images of 1024x1024. Is it possible to modify the --multi-scale commands to include larger dimensions such as 1024, and have the algorithm run successfully?
Here is what it would look like when modified:
--multi-scale 320,352,384,416,448,480,512,544,576,608,640,672,704,736,768,800,832,864,896,928,960,992,1024

Yes, the functionality should is there. But, don't you think you are overdoing the scales. You have at least 18 scales mentioned here. Too much of anything is bad. There is a reason it likes things divisible by 32 because at that increase in size something more meaningful will show up in the image. Spamming sizes like this won't help you at all, it would rather waste your time.


Multi Armed Bandits with large number of arms I'm dealing with a (stochastic) Multi Armed Bandit (MAB) with a large number of arms.
Consider a pizza machine that produces a pizza depending on an input $i$ (equivalent to an arm). The (finite) set of arms $K$ is given by $K=X_1\times X_2 \times X_3\times X_4$ where $X_j$ denote the set of possible amounts of ingredient $j$.
e.g.
$X_1=\{$ small, medium, large $\}$ (amount of cheese) or
$X_2=\{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10\}$ (slices of salami)
Thus, running the pizza machine with input $i$ is equivalent to pulling arm $i\in K$. Due to the different permutations, the number of arms $|K|$ is very large (between 100,000 and 1,000,000). Depending on the pulled arm $i$, the machine generates a pizza (associated with a reward that indicates how delicious the pizza is).
However, the machine's rewards are non-static. Pulling an arm $i$ generates a reward according to an unknown (arm specific) distribution $P_i$, with all rewards drawn from $P_i$ beeing i.i.d.. In addition, it is possible to normalize all rewards to the interval [0,1].
The above problem corresponds to the standard problem of a stochastic MAB, but is characterized by the large number of arms. In the case of the pizza machine, several days of computation time are available to determine the best pizza, so the number of itarations is allowed to be large as well.
In my investigation of MAB algorithms addressing a large number of arms, I came across studies that could call up to a few thousand arms.
Are there algorithms that exist in the MAB domain, which specifically deal with large problem instances (e.g.with $|K|>100,000$)?

Without any knowledge on the references you came across, I am assuming that the authors were considering common applications of MAB (planning, online learning, etc.) for which the time horizon is usually small. In such applications, we usually cannot afford a large average regret which is inevitable for standard MAB algorithms due to the $\sqrt{|K|}$ factor.
Depending on the application or additional constraints you can impose on your problem, there are several works that consider structured stochastic MABs in which there are much better guarantees than the pessimistic $\sqrt{K}$ bounds. One variant of structured MABs is learning on graphs [1], where one can obtain regret bounds growing with $\sqrt{\beta(G)}$, where $\beta(G)$ is the independence number of a graph $G$.
[1] F. Liu, Z. Zheng, N. Shroff, "Analysis of Thompson Sampling for Graphical Bandits Without the Graphs", ArXiv. 2018.


Can agent based machine learning achieved with any ML algorithms other than neural network? I would like to know other tha neural network, is there any ML technique for agent based ML. If so how to train an agent with some predefined rules? Can we use python programming for representing those rules?

Neural networks are not inherently part of reinforcement learning (a popular agent-based framework for describing control problems). In general, if you have an agent-based scenario, you are trying to optimise a function:

Policy( State ) -> Action

Where State can be any combination of current observations and history that seem relevant to the problem. The optimisation is usually over some measure of success, achieving goals etc. Reinforcement learning formalises these terms using Markov Decision Processes, which is a very general and successful approach, but not the only one.

Finding the best policy does not require neural networks. There are a lot of basic reinforcement learning algorithms that are defined without reference to NNs. For instance, Q-learning does not need neural networks, they are an optional extra. In addition, you do not need to even use RL - you can try to search for a suitable Policy function directly using a method such as Genetic Algorithms.

Focusing more on RL methods, the usual approach without any kind of function approximator is to just store an estimate of the value of each state or state/action pair. This value is a measure of long term reward, and has a formal definition in RL. Often called the tabular approach , because it is just a table of states and their estimated values, it works for small problems just fine. For example you can train an agent to play tic tac toe, find an optimal path that moves over a grid, or discover when to hold and when to twist in a simplified blackjack game.

However, problems start to occur with the most basic methods when the state space becomes large, which happens very easily. Tabular methods are OK up to a million or maybe ten million states (depending on how easy it is to take trial actions). These are enumerated states, so typically adding a single dimension to a problem multiplies the number of possible states by the number of options in that new dimension. So after a certain level of complexity, a simple table is not good enough, and you need a function approximator. Function approximators for RL ideally have the following properties:

Can generalise from examples.
Can be updated online as new data arrives, progressively forgetting old data.
Can be differentiated with respect to their learning parameters.

Neural networks fit the bill. Some other popular supervised learning techniques do not - for instance Random Forests cannot usually be trained online.

In fact linear regression does work, and it is a very simple approach. For some problems, linear regression works very well in combination with reinforcement learning. You may need to do some careful feature engineering, and it is not always possible for more complex problems, but if you have a "medium-sized" agent-based problem - simpler than learning Go or playing video games - and you want to avoid using neural networks, then you should be able to attempt to train an agent using a combination of RL, such as Q-learning, plus linear regression.

Both Q-learning and linear regression have plenty of examples available online, and can be implemented from scratch using basic Python frameworks such as NumPy.


How to optimize neural network parameters with REINFORCE I've seen a few mentions in papers that neural network parameters can be found using REINFORCE algorithm. It was mentioned in the context of nondifferentiable operations involving e.g. step function which appears in "hard attention" or weight pruning. Unfortunately, I haven't seen how to really do this. In Markov Decision Process we have, states (S), actions (A) and rewards (R) so what is what in case of neural net? I don't see how can we find parameters or neural net if the gradient is not well defined.
Any code sample or explanation?

The term REINFORCE actually corresponds to a method of estimating gradients, it is not particular to reinforcement learning. The paper you linked doesn't appear to deal with RL at all, so the issue they're describing is not one that you should expect to find in a policy gradient application.
If you're using REINFORCE to estimate policy gradients in RL (this is the common use case), your parameters are parameterizing a policy function. The inputs are states and the outputs are actions. The issue with estimating the gradient of the parameters is as follows. In RL the objective is to maximize expected rewards. This is an expectation with respect to the policy of the cumulative reward of a trajectory. If you take the gradient of this wrt the parameters, since the expectation is wrt the policy which depends on the parameters, you can't move the gradient inside the expectation so you can't estimate the gradient from samples. Using REINFORCE however, you use the "log-derivative" trick to rewrite the objective as an expected gradient, which can be estimated from samples.


How can we create a vector space where word spelling and pronunciation can be easily compared? In natural language processing, we can convert words to vectors (or word embeddings). In this vector space, we can measure the similarity between these word embeddings.

How can we create a vector space where word spelling and pronunciation can be easily compared? For example, "apple" and "ape", "start" and "startle" are very similar, so they should also be similar in this new vector space.

I am eventually looking for a library that can do this out of the box. I would like to avoid implementing this myself.

If you only need the vector space as a way to obtain a similarity measure, you may want to consider a distance measure instead. Similarity and distance are inversely related: identical words have maximum similarity or zero distance, and as the similarity decreases, the distance increases.

For instance, the Wagner-Fischer algorithm computes the edit distance between two strings of characters. This edit distance takes into acccount insertions and deletions, as in your examples, but also substitutions (for example "gray" vs. "grey").

The article linked above includes pseudocode that should translate easily to actual code.


Why isn't the Credit Card Fraud Detection dataset from Kaggle already balanced? I am working on a credit card fraud detection problem using autoencoders. I have a question regarding the dataset I'll be using.

I've downloaded the dataset for the above problem from Kaggle, which is highly imbalanced: it contains only 492 frauds out of 284,807 transactions. Why isn't the dataset already balanced? Should I balance it before applying the auto-encoder?

I believe that the idea is to have a similar ratio of fraud/"normal transaction" as to the ones that bank encounter on real life.

If you balance it you will probably have a lot of false positive once you apply your solution to real world's data and, if that may be fine for you to play with, it's not what a bank would like as they can't block too much of the "normal" transaction or the client will change of bank. According to this post (https://www.quora.com/How-many-transactions-do-typical-banks-process-everyday) just the mastercard alone represent 3.4billion transaction/day, just imagine if 1% of the daily transaction where blocked every day, it would represent 34 million of transactions blocked without any valid reason.

It's different from a lot of classification problem where you want to have balanced dataset, here you try to detect anomalies and, by definition, they are rare so they should be as rare in your dataset.


Where are reinforcement algorithms used in financial services? One of the most common misconceptions about reinforcement learning (RL) applications is that, once you deploy them, they continue to learn. And, usually, I'm left having to explain this. As part of my explanations, I like to show where it is being used and where not.
I've done a little bit of research on the topic, but the descriptions seem fairly academic, and I'm left with the opinion that reinforcement learning is not really suitable for financial services in regulated markets.
Am I wrong? If so, I would like to know where RL is being used? Also, in those cases, are these RL algorithms adapting to new data over time? How do you ensure they are not picking up on data points or otherwise making decisions that are considered to be unacceptable?

I know the new bot in JP Morgan uses reinforcement learning:
Machine Learning in FX
There a lot of studies of Reinforcement learning can apply in crypto currency trading. Here is one of many examples:
A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem


What is the relationship between the size of the hidden layer and the size of the cell state layer in an LSTM? I was following some examples to get familiar with TensorFlow's LSTM API, but noticed that all LSTM initialization functions require only the num_units parameter, which denotes the number of hidden units in a cell.

According to what I have learned from the famous colah's blog, the cell state has nothing to do with the hidden layer, thus they could be represented in different dimensions (I think), and then we should pass at least 2 parameters denoting both #hidden and #cell_state.

So, this confuses me a lot when trying to figure out what the TensorFlow's cells do. Under the hood, are they implemented like this just for the sake of convenience or did I misunderstand something in the blog mentioned?

I had a very similar issue as you did with the dimensions. Here's the rundown:

Every node you see inside the LSTM cell has the exact same output dimensions, including the cell state. Otherwise, you'll see with the forget gate and output gate, how could you possible do an element wise multiplication with the cell state? They have to have the same dimensions in order for that to work.

Using an example where n_hiddenunits = 256:

Output of forget gate: 256
Input gate: 256
Activation gate: 256
Output gate: 256
Cell state: 256
Hidden state: 256

Now this can obviously be problematic if you want the LSTM to output, say, a one hot vector of size 5. So to do this, a softmax layer is slapped onto the end of the hidden state, to convert it to the correct dimension. So just a standard FFNN with normal weights (no bias', because softmax). Now, also imagining that we input a one hot vector of size 5:

input size: 5
total input size to all gates: 256+5 = 261 (the hidden state and input are appended)
Output of forget gate: 256
Input gate: 256
Activation gate: 256
Output gate: 256
Cell state: 256
Hidden state: 256
Final output size: 5

That is the final dimensions of the cell.