Inconsistency in TD-Leaf algorithm in KnightCap chess engine Notice that, in the following formula, at the very right, the term multiplied with $\lambda$ is $d_i$

$$
w := w + \alpha \sum_{i=1}^{N-1} \nabla r(x_i^l, w) \Big \lfloor \sum_{j=i}^{N-1} \lambda^{j-i} d_i \Big \rfloor
$$

Note that $i$ is the index of the first sum.

However, in the following formula, the term is $d_m$, which was $d_i$ in the first equation.

$$
w_j := w_j + \alpha \sum_{i=1}^{N-1} \frac{\partial r(x_i^l, w)}{\partial w_j}  \Big \lfloor \sum_{m=i}^{N-1} \lambda^{m-i} d_m \Big \rfloor
$$

Note that $m$ is the index of the second sum.

In my opinion, first equation seems reasonable. The $\lambda$ in second one actually seems to work like discount factor.

In another chess engine, Giraffe, the author cited same paper, noted the first equation, but then proceeded on to implement second one. See also KnightCap Paper.

The second equation is correct. In TD($\lambda$), the $\lambda$ parameter can be tuned to smoothly vary between single-step updates (essentially what Sarsa does) in the case of $\lambda = 0$, and Monte-Carlo returns (using the full episode's returns) in the case of $\lambda = 1$.

In the first equation, $\sum_{j = i}^{N - 1} \lambda^{j - i} d_i$ could be interpreted as a... summing up exactly the same temporal-difference term $d_i$ a number of times (specifically, $N - 1 - j$ times), but multiplied by a different scalar every time. I'm not sure how that could be useful in any way.

In the second equation, $\sum_{m = i}^{N - 1} \lambda^{m - i} d_m$ can be interpreted as a weighted combination;

$1 \times d_i$: this is the difference between what we predict our returns will be at time $i + 1$, and what we previously predicted our returns would be at time $i$.
$+ \lambda^1 \times d_{i + 1}$: a very similar difference between two of our own predictions, now the predictions at time $i + 2$ and $i + 1$. This time the temporal-difference term is weighted by the parameter $\lambda$ (completely ignored if $\lambda = 0$, full weight if $\lambda = 1$, somewhere in between if $0 < \lambda < 1$).
$+ \lambda^2 \times d_{i + 2}$: again a similar temporal-difference term, now again one step further into the future. Downweighted a bit more than the previous term in cases where $0 < \lambda < 1$.
etc.

For people who are familiar with temporal-difference algorithms like TD($\lambda$), sarsa($\lambda$), eligibility traces, etc. from Reinforcement Learning literature, this makes a lot more sense. The notation is still a bit different from the standard literature on algorithms like TD($\lambda$), but in fact becomes equivalent once you note that in this paper they discuss domains where there are only rewards associated with terminal states, and no intermediate rewards.

Intuitively, what they're doing with the $\lambda$ parameter is assigning more weight (or "credit" or "importance") to short-term predictions / short-term "expectations" (in the English sense of the word, rather than the mathematical sense of the word) or observations of rewards, over long-term predictions/observations. In the extreme case of $\lambda = 0$, you completely ignore long-term predictions/observations and only propagate observed rewards very slowly, one-by-one in single steps. In the other extreme case of $\lambda = 1$, you propagate rewards observed at the end of episodes with equal weight all the way to the beginning of the episodes, through all states that you went to, giving them all equal weight for that observed reward. With $0 < \lambda < 1$, you choose a balance between those two extremes.

Also note that Equation (5) in the KnightCap paper (where they similarly discuss the extreme case of $\lambda = 1$, like I did above) is incorrect if we take the first equation from your question, but is correct if we take the second equation.


Is there a machine learning system that is able to understand mathematical problems given in a textual description? Is there a machine learning system that is able to "understand" mathematical problems given in a textual description, such as

A big cat needs 4 days to catch all the mice and a small cat needs 12 days. How many days need both, if they catch mice together?

?

There was a lot of work on this topic at UT Austin, which has now migrated to the Alan Institute.
There is no off-the-shelf software that will answer your question (if there was, DARPA would stop funding its development!), but you can read about the latest development in a number of recent papers.
This paper (Seo et al. EMNLP 2015) discusses the techniques that are used to interpret diagrams that accompany geometry problems, while this one (Hosseini et al. EMNLP 2014) talks about how to automatically parse verbs to interpret the meaning of a question. The 2015 TACL paper (Koncel-Kedziorski et al. 2015) completes this by discussing how to extract the relevant equations from a word problem. Once you have the equations, know what question is being asked, and can interpret any diagrams, you can do most high school math problems.
However, I don't think this is yet a fully reliable system. It is one part of a larger, long running effort to create a program that can achieve higher education certifications in many subjects. You can see many projects related to this at the Alan Institute's website.


Agent in toy environment only learns to act optimally with small discount factors I have tried several environment libraries like OpenAI gym/gridworld but now I am trying to create a toy environment for experimentation. The environment I've created is as follows:

State: grid with n rows by m columns, represented by a boolean matrix. Each grid cell can be empty or filled and the grid starts empty.
Action: one of the m columns to be filled, which must have at least the top row empty.
Next state: Once a column is chosen, the lowest unfilled cell in that column is filled. This works from bottom up like a very simple version of Tetris.
Reward: after every action, a reward equal to the number of empty columns is awarded.

Therefore in a sample world of 5 rows by 3 column, starting off with an empty grid, the maximum attainable reward would be by filling column wise first. This policy will give a maximum total reward of 2*5 + 1*5 = 15. (2 free columns by 5 row action, once first column is filled then 1 free column by 5 row action.)

This very simple environment is trained using DQN with a single ff layer. The agent only took a few episodes to converge and is able to produce the maximum attainable reward.

In a next toy environment, I've made it a little more complex. I modified the very first action to be random choice of any column. I have retrained the RL model with the new environment modification. However, after convergence, the agent does not attain max score of 15 for all possible starting columns. I.e. If column 1 was randomly chosen first, max score might be 15, however column 2 or 3 was randomly chosen first, max score might only reach 11 or 9. In theory, the optimum policy would be for the agent to fill column that was randomly chosen first - i.e. repeat the first randomly chosen action.

I have tried several ways to tweak my input parameters (e.g. episilon_decay_rate, learning_rate, batch_size, number of hidden nodes) to see if the agent could act optimally for all possible starting columns. I also tried DDQN and Sarsa. The only way I could make the agent perform optimally is by reducing gamma (discount factor) to 0.5 or below. Are there any explanations to why the agent only works for small discount factors in this example? Also, are there alternative ways to obtain the optimum policy?

This is an episodic problem, and there should be no issue in theory with most learning algorithms coping without a discount factor (or setting gamma = 1).

  Are there any explanations to why the agent only works for small discount factors in this example?

The most likely explanations are:

You have a mistake in your implementation or use of DQN.
You have an incorrect setting of a neural network hyperparameter. My initial thought would be you have mistakenly put a non-linearity like softmax or sigmoid on the NN outputs (it needs to be a linear output). Or it could just be that 10 hidden neurons is not enough for this representation
Convergence requires far longer to train than you thought, once you introduce non-linear relationships between state and expected reward.

It is not surprising that the simpler puzzle trains more easily, as the agent can reduce it down to a linear counting puzzle without really needing to "look" at the representation. I would expect the Q values to be very poor approximations to the correct values after so few iterations, but the problem is simple enough that the incorrect values still produce an optimal behaviour.


Is there any way, in which we can monitor what's going on the screen in android to generate a model from it? I've been working on a project (Android Game), in which the player has to confront with some obstacles/enemies whom he has to destroy. So, is there a way in which we can monitor how the user of the game plays and accordingly to generate (and timely updating) that trained model, which can be used later in the game to make the user think of a different way to defeat an enemy unlike going in the same streamline flow which he has followed till now.

I've implemented Q-Learning and Genetic algorithms on the PC, to make the Game AI for some games like 'tetris' to make the computer play on it's own. But, haven't done it in Android till now.

Already searched: I've also referred to some websites in which they suggested using Neural Networks to encounter the same.

But, I'm unable to get an accurate procedure in which this can be done. Please suggest me a way in which I can monitor the user's input on how he plays.

Thank you.

The software that you require to provide your AI agent remote access is called Virtual Network Computing (VNC). A VNC is a desktop sharing system that transmits keyboard and mouse movements, over a network connection, from the server (the AI) to the client (training environment).

This allows your AI agent to use a computer like a human does, which is by looking at the screen pixels and operating a virtual keyboard and mouse. Infact VNC's are used conventionally for remote technical support and accessing files on one's computer remotely.

An excellent use case example is Open AI's Universe platform. Alongside Gym, which is Open AI's toolkit for developing RL algorithms. Universe launches programs behind a VNC remote desktop, which enables the training of AI agents by only showing them the screen pixels and allowing them to operate a virtual keyboard or mouse.

Since you want to train your AI agent on an Android based training environment, my suggestion is that you consider installing either Real VNC (android client) or Droid VNC on your Android device. You can then install the VNC server software on your training machine (i.e. an nvidia docker instance). Your AI agent will now have full access to the training environment screen for reinforcement learning.

I also recommend that you look at this similar question on the network.

https://askubuntu.com/questions/414189/how-to-remotely-control-my-android


Is it possible to apply deep dream technique for the audio streams? What happens if you apply the same deep dream technique which produces "dream" visuals but to media streams such as audio files?

Does changing image functions into audio and enhancing the logic would work, or will it no longer work/doesn't make any sense?

My goal is to create "dream" like audio based on the two samples.

As far as I can see, there's no reason why you couldn't (for example) take the convolutional inputs to deepdream from adjacent sample points, rather than adjacent spatial positions, as is the case with image input.

Given the 'self similar' nature of deep dream images, listening to this fractal granular synthesis technique might be of interest/inspiration.


Blocks world Game: A* heuristic solution approach I am developing a heuristic solution for blocks world problem.

I tried using number of blocks out of place as my h(n). It seems little ineffective.

Can someone please point out a suitable heuristic for the problem and explain with few examples how it will work.

Blocksworld Problem Example:

Initial(starting State):
Stack 0: D,B
Stack 1: A,E
Stack 2: C
Stack 3: F

Goal State:
Stack 0: A,B,C,D,E,F

You may start assigning penalties for undesirable conditions in a state like:

1) Number of blocks outside stack 0.

Supose you penalize with 10 units each block outside stack 0, then the starting state above adds 40 units to the penalty score

2) Number of blocks in the stack 0 in a position different than in the goal state.

Supose you penalize with 50 units each block in stack 0 in the wrong position of the stack 0, then the starting state adds 50 to the penalty score (block D).

And go on
you may add as many penalties as you find helpful for solving the problem.

Then for each step you list all possible block moves, for each one of this moves you calculates the total penalty score that this move produces and choose the one move with the minimal score (if ties choose the first one).
You keep repeating above until you reach the final state (penalty 0).


Why isn't my model learning satisfactorily? The problem to solve is non-linear regression of a non-linear function. My actual problem is to model the function "find the max over many quadratic forms": max(w.H.T * Q * w), but to get started and to learn more about neural networks, I created a toy example for a non-linear regression task, using Pytorch.
The problem is that the network never learns the function in a satisfactory way, even though my model is quite large with multiple layers (see below). Or is it not large enough or too large? How can the network be improved or maybe even simplified to get a much smaller training error?

I experimented with different network architectures, but the result is never satisfactory. Usually, the error is quite small within the input interval around 0, but the network is not able to get good weights for the regions at the boundary of the interval (see plots below). The loss does not improve after a certain number of epochs.  I could generate even more training data, but I have not yet understood completely, how the training can be improved (tuning parameters such as batch size, amount of data, number of layers, normalizing input (output?) data,  number of neurons, epochs, etc.)

My neural network has 8 layers with the following number of neurons: 1, 80, 70, 60, 40, 40, 20, 1.

For the moment, I do not care too much about overfitting, my goal is to understand, why a certain network architecture/certain hyperparameters need to be chosen. Of course, avoiding overfitting at the same time would be a bonus.

I am especially interested in using neural networks for regression tasks or as function approximators. In principle, my problem should be able to be approximated to arbitrary accuracy by a single layer neural network, according to the universal approximation theorem, isn’t this correct?

Neural networks learn badly with large input ranges. Scale your inputs to a smaller range e.g. -2 to 2, and convert to/from this range to represent your function interval consistently.


How do big companies, like Facebook, model individuals and their interaction? As a layman in AI, I want to get an idea of how big data players, like Facebook, model individuals (of which they have so many data).
There are two scenarios I can imagine:

Neural networks build clusters of individuals by pure and "unconscious" big data analysis (not knowing, trying to understand and naming the clusters and "feature neurons" on intermediate levels of the network) with the only aim to predict some decisions of the members of these clusters with highest possible accuracy.

Letting humans analyze the clusters and neurons (trying to understand what they mean) they give names to them and possibly add human-defined "fields" (like "is an honest person") if these were not found automatically, and whose values are then calculated from big data.

The second case would result in a specific psychological model of individuals with lots of "human-understandable" dimensions.
In case there is such a model, I would be very interested to know as much about it as possible.
What can be said about this:

Is there most probably such a model (that is kept as a secret e.g. by Facebook)?

Has somehow tried to guess how it may look like?

Are there leaked parts of the model?

My aim is to know and understand by which categories Facebook (as an example) classifies its users.

This article may shed some light on this question:

Facebook Doesn’t Tell Users Everything It Really Knows About Them


What are the differences between artificial neural networks and other function approximators? Modern artificial neural networks use a lot more functions than just the classic sigmoid, to the point I'm having a hard time really seeing what classifies something as a "neural network" over other function approximators (such as Fourier series, Bernstein polynomials, Chebyshev polynomials or splines).

So, what makes something an artificial neural network? Is there a subset of theorems that apply only to neural networks?

Backpropagation is classic, but that is the multi-variable chain rule, what else is unique to neural networks over other function approximators?

First of all, neural networks are not (just) defined by the fact that they are typically trained with gradient descent and back-propagation. In fact, there are other ways of training neural networks, such as evolutionary algorithms and the Hebb's rule (e.g. Hopfield networks are typically associated with this Hebbian learning rule).

The first difference between neural networks and other function approximators is conceptual. In neural networks, you typically imagine that there are one or more computational units (often called neurons) that are connected in different and often complex ways. The human can choose these connections (or they could also be learned) and the functions that these units compute given the inputs. So, there's a great deal of flexibility and complexity, but, often, also a lack of rigorousness (from the mathematical point of view) while using and designing neuron networks.

The other difference is that neural networks were originally inspired by the biological counterparts. See A logical calculus of the ideas immanent in nervous activity (1943) by Warren McCulloch and Walter Pitts, who proposed, inspired by neuroscience, the first mathematical model of an artificial neuron.

There are other technical differences. For example, the Taylor expansion of a function is typically done only at a single value of the domain, it assumes that the function to be approximated is differentiable multiple times, and it makes uses of the derivatives of such a function. Fourier series typically approximate functions with a weighted sum of sinusoids. Given appropriate weights, the Fourier series can be used to approximate an arbitrary function in a certain interval or the entire function (if the function you want to approximate is also periodic). On the other hand, neural networks attempt to approximate functions of the form $f: [0, 1]^n \rightarrow \mathbb{R}$ (at least, this is the setup in the famous paper that proved the universality of neural networks) in many different ways (for example, weighted sums followed by sigmoids).

To conclude, neural networks are quite different from other function approximation techniques (such as Taylor or Fourier series) in the way they approximate functions and their purpose (i.e. which functions they were supposed to approximate and in which context).


Which kind of data does sigmoid kernel performance well? While I was playing with some hyperparameters, I came to a wired situation. My dataset is IRIS dataset to be specific. SVM algorithm has some hyperparameters that we can tune, such as Kernels, and C value.
(All accuracy calculations and SVM are from sklearn package to be specific)
I made a comparison between kernels and noticed sigmoid kernel was performing way worse in terms of accuracy. It is more than 3 times less accuracy than RBF, Linear, and Polynomial. I do know that kernels are quite data-sensitive and data-specific, but I would like to know "Which types of data is sigmoid kernel good at any example? or is this my fault due to wrong C value for sigmoid kernel?"

The sigmoid kernel is better suited for binary classifications. As the IRIS dataset is for multi-class classification, its performance was not as good as other kernels. You can train with only 2 types of flowers to see if the sigmoid kernel can perform well.


Detect visual attention area in an image I'm trying to detect the visual attention area in a given image and crop the image into that area. For instance, given an image of any size and a rectangle of say LxW dimension as an input, I would like to crop the image to the most important visual attention area. I'm looking for a state of the art approach for that.

Do we have any tools or SDK to implement that? Any piece of code or algorithm would really help.

You can search for the following paper titles:

A Deep Multi-Level Network for Saliency Prediction.
Beyond Universal Saliency: Personalized Saliency Prediction with Multi-task CNN.

You can code in python using Pytorch framework.


Why is this variable in equation 2 of the SQAIR paper a random vector of $n$ ones followed by a zero? I've been reading the SQAIR paper lately, and the mathematics involved seems a bit complicated.
Some background, about the paper: SQAIR stands for Sequential Attend, Infer, Repeat - the paper does generative modelling of moving objects. The idea of Attend, Infer, Repeat is to decompose a static scene into constituent objects, where each object is represented by continuous latent variables. The latent variables, $z^{what}$,$z^{where}$ and $z^{pres}$ encode the appearance, position and presence of an object.
Here's a screenshot of the first of many things I'm unable to understand -

Why is $z^{pres,1:n+1}$ a random vector of $n$ ones followed by a zero? Why do we need the zero? How does it help?
Furthermore, an explanation of equation $(2)$ as in the image above, would be great.
P.S. I hope you all find the paper interesting. I'll ask other questions from the paper in separate posts, so as to not crowd one post with too many queries.

From my understanding of the paper, $Z^{pres}$ keeps track of the objects in the scene. For every step of the sequential inference, $z^{pres,i}$ takes either a 0 or a 1. A 1 represents that an object is present and it has to be explained by the remaining latent variables. 0 indicates that all the objects have been explained and inference is complete.


Why aren't there neural networks that connect the output of each layer to all next layers? Why aren't there neural networks that connect the output of each layer to all next layers?

For example, the output of layer 1 would be fed to the input of layers 2, 3, 4, etc. Beyond computational power considerations, wouldn't this be better than only connecting layers 1 and 2, 3 and 4, etc?

Also, wouldn't this solve the vanishing gradient problem?

If computational power is the concern, perhaps you could connect layer 1 only to the next N layers.

Actually, this already exists!

I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks. Just like in your question, within a dense block, the output of each layer is given as input to all subsequent layers. Put another way, in a normal feed-forward neural network the $l$th layer is a function of the previous output $x_l = H(x_{l-1})$, while in the dense net each layer is a function of all the previous outputs $x_l = H([x_0, x_1, \dots, x_{l-1}])$.

However, since it is a CNN, there is a reduction in the size of the feature maps with each pooling layer, so to keep the dimensions constant, there is an alternation between the dense block and pooling layers.

The results are clear: not only in almost all the tests the accuracy of the dense net is greater than that of the other methods, but they do so using up to 90% fewer parameters, i.e. they have a high efficiency of parameters. Moreover, as suggested by the authors themselves, the improved accuracy can be explained by the shorter connections between the layers, which allow acting during the training phase in a deep supervision fashion, solving the vanishing gradient problem. This is similar to how it was done in other methods, but with a less complicated gradient.

If you're interested you should definitely check out their paper Densely Connected Convolutional Networks (2018).


How to make convnets aware what the image actually is, not what is depicted on it? I've uploaded a picture to Wolfram's ImageIdentify of graffiti on the wall, but it recognized it as 'monocle'. Secondary guesses were 'primate', 'hominid', and 'person', so not even close to 'graffiti' or 'painting'.

Is it by design, or there are some methods to teach a convolutional neural network (CNN) to reason and be aware of a bigger picture context (like mentioned graffiti)? Currently it seems as if it's detecting literally what is depicted in the image, not what the image actually is.

This could be the same problem as mentioned here, that DNN are:

  Learning to detect jaguars by matching the unique spots on their fur while ignoring the fact that they have four legs.2015

If it's by design, maybe there is some better version of CNN that can perform better?

You seem to be wanting some description of the 'style' of an image.

To make that work in general, I'd guess that would actually require quite a lot of pre-processing to present 'texture elements' (rather than pixels) as the basic features.

This is quite speculative, but one approach might be to use Iterated Function Systems as a means of extracting these.

Whether 'spatial adjacency' (and hence CNN) is then the best approach to make higher-level decisions about these elements is (AFAIK) a matter for experiment.


AI Knowledge Based System: I am having difficulties understanding proposition logic I am having difficulties wrapping my head around how the answer is being produced. How does the solution come up with the answer 3? How does one derive the answer 3 from the truth table?

For example the following file :

  TELL  p2=> p3; p3 => p1;

  ASK p2

Produces the following result.

  Standard output is an answer of the form YES or NO, depending on
  whether the ASK(ed) query q follows from the TELL(ed) knowledge base
  KB. When the method is TT and the answer is YES, it should be followed
  by a colon (:) and the number of models of KB

The provided result is a non-sense for this input.

In a closed world paradigm (ie: prolog), where all non-provable facts are assumed as false, from "p2->p3 and p3->p1", p2 is false, and the program result should be "p2 ? No".

In an open world, as it is not possible to proof "p2" nor "not p2", the result must be "p2 ? unknown"

"p2 -> p3 and p3 -> p1" doesn't provides any information about "p2" in the same way that "if it is sunny then I'm happy" doesn't provides information about if it is sunny or raining.

The meaning of the number after the "yes" (the number of "models") is unknown. Even present two true tables for a single fact "p2 -> p3 and p3 -> p1" can be considered an error.


Is analogy necessary to artificial general intelligence? Analogies are quite powerful in communication. They allow explaining complex concepts to people with no domain knowledge, just by mapping to a known domain. Hofstadter says they matter, whereas Dijkstra says they are dangerous. Anyway, analogies can be seen as a powerful way to transfer concepts in human communication (dare I say transfer learning?).
I am aware of legacy work, such as Case-Based Reasoning, but no more recent work about the analogy mechanism in AI.
Is there a consensus on whether or not analogy is necessary (or even critical) to AGIs, and how critical would they be?
Please, consider backing your answers with concrete work or publications.

I don't think I can give you a true answer to the actual question as posed, as I don't have a strict definition of "general intelligence". Nor do I have a solid definition of "critical" in context.
However, if we lean on our naive/intuitive understanding of what "general intelligence" and what it means to be critical, you might translate your question as

Would a general intelligence system need analogy-making in order to do certain things that it couldn't otherwise do?

Or, to put it another way

Are there useful behaviors that are enabled by analogical reasoning that can't be replicated any other way?

In the strictest sense, I don't have an answer to either of those questions either, but there is at least evidence to suggest that the answer may be "yes".  See, for reference, the Copycat paper by Hofstadter and Mitchell.
From what I've seen, some of the kinds of problems Copycat solves are different from anything I've seen solved by other approaches.  Now maybe it's just a coincidence that nobody has tried solving those problems with, I don't know, let's say "deep learning" or "rule induction" or "genetic algorithms".  Or maybe they have and I just haven't stumbled across that corpus of research.
Anyway, I'll also add that there is still ongoing research into using analogy for AI/ML.  See, for example, the paper Analogical Inference for Multi-relational Embeddings (2017), where the authors talk about using analogy, but define their approach as "analogical inference" (which they claim is different from "analogical reasoning" as defined during the earlier "GOFAI period").  There is also the paper Evaluating vector-space models of analogy (2017), where another set of authors deal with a form of analogical reasoning.
I don't think there's a consensus as to whether or not some form of analogical reasoning is "critical", but it's definitely a subject that is still being researched.
And to go off on a little bit of a tangent - an interesting related question would be to ask whether or not "analogy making" would be an emergent property of a sufficiently deep/wide ANN, or would such a facility need to be designed and coded up explicitly.


How to combine input from different types of data sources? I've to train a neural network using microphone data (wav files), accelerometer sensor data and light sensor data.

Right now the approach I thought was to convert all data into images and combine them into a single image and train my neural network.

Another approach was to convert wav files into arrays and combine them along with sensor data and train my neural net.

Are my approaches correct or is there a better way to do this?

Any suggestions/ideas are welcome.

Handling Multiple Input Types

Multiple input types are common as learning technology moves from academic labs and open source examples into the real world.  The design process includes the below four steps, which related directly to this question.  There are other steps that don't centrally related not mentioned here.

First Step — List Inputs in Most Usable Form

In the case of this question, there are three in the list.

Monophonic or multichannel audio data as an array of samples or sample vectors acquired at a fixed sampling rate
Acceleration data as an array of samples in $\mathbb{R}^n$ where $n \in \{1, 2, 3\}$ acquired at a fixed sample rate.
Visible light or IR detection data as an array of samples of $c$ channels at a fixed sample rate.

Fixed sample rate is listed with each type.  Variable sample rates pose additional burden on learning.  If any of the three are acquired with a variable sample rate, they can be converted to fixed sample rate arrays provided there is a time stamp or a time-since-last-sample channel available with each sample vector.

Variable to fixed sample rate conversion can be accomplished easily via ffmpeg, programmatically via its library, or through any equivalent software designed for this purpose.  The acceleration and visible light signals can also be adapted for ffmpeg's input.

When a specific dynamic characteristic must be preserved, finer control over the interpolation can be obtained using cubic spline algorithms.  Such algorithms were originally written in FORTRAN or C and have been ported to Java and Python.

Second Step — When Spectral Data is Practically Required

With time series data, the phenomena being learned is often largely a function of vibration frequencies and amplitudes.  This is almost always the case with audio data.  The exception is when transient detection is the primary teller of the conditions being measured rather than more continuous sounds.

A note is spectral in description.  A knock is a transient and both spectra and actual rise characteristics are indicative of source.

Presenting digital samples from the transducer circuit (in this case the microphone diaphragm's position in response to pressure waves) to the learning system directly is not a best practice.  Unconverted vibration information forces the learning system to learn transformations for which optimized algorithms already exist.  This slows learning and generally pushes complexity beyond the learning system's capacity.

When the spectral features best characterize the phenomena being learned, as in hums, screeches, vowel sounds, music, horns, or beeps, the best practice is to convert the samples in the time domain to frequency domain data.

This is usually done using one of the forms of the FFT (fast Fourier transform) algorithm in combination with a RMS (root mean square) calculation of amplitude and windowing.  Windowing attenuation forms, such as the Hamming window or the cosine window, are available in most packages that have FFT capabilities.

The sample rate is also important.  Nyquist's sample rate criteria must be met by the A-to-D converter (in the sound card or instrumentation circuitry) to ensure that the spectral range includes the best indicators of the phenomena to be learned.  In machine learning, such indicators are called features.

In most cases, audio or vibration information is presented to a learning system or sub-system as spectral frequencies and amplitudes.  Vibration recognition systems and speech to text systems generally execute FFTs (with RMS and windowing) on the raw input.  In human ears, the conversion to spectral information is mechanical, using the tapering geometry of the ear's cochlea.

There also exists algorithms that can directly develop spectral information from variable sample rate data to amplitudes and frequencies, combining two transformations into one for efficiency of computing resources and additional input accuracy.

In summary of this section, whether light, IR, acceleration, or other data is treated as audio and converted to spectral information is based on whether

The signals are vibrations that have rises and falls within specific frequency ranges like hums, notes, and speech or
They are transients (as with tapping the mechanical package on which the accelerometer is mounted) or
They are binary in nature (as in the case of linear encoder signals).

Third Step — When Topology of Types

The topological orientation of the input source types to one another is key to feeding the information into any learning system.  These are the two pure extremes.

The signals from the source types are describing the same physical or data mining property.
The signals are measuring independent properties that together describe the state of the system being measured.

In the case of (1), a normalized sample from one type of input channel is, from a learning point of view, indistinguishable from a normalized sample from another.  This would be the case if the accelerometer is measuring the same vibration that the microphone is measuring.  This is the less usual case, but is more common when the data is for study of vibration to project mechanical or materials failures in advance.

With Background Clarified, the Q&A Becomes Clearer

The question asked this.

  How [does one best] combine input from different types of data sources?

Whether the source data is case (1) or case (2) or some midpoint between them is key to answering this question.  Samples from different source types that measure single phenomena can be phase adjusted, spectrally adjusted, and otherwise normalized to present a single data type at the input.  Merging into a single serial data stream would be the optimal pre-processing for training in case (1).

More often, the topological orientation of the different input source types conforms to case (2) above.  In that case, different input types should be presented in parallel to the learning system inputs.  The merging would be counterproductive in this case, for a reason that will be clarified in the next section below.

To avoid sparsity in input data, which creates additional training burden, the sample rates should be equalized for all channels of all data source types.  This may require conversion of the sample rate, which can be done using the same techniques (above) for converting from variable to fixed sample rates.

Images of Data

The question stated this.

  Right now, the approach I [was considering is] to convert all data into images and combine them into a single image and train my neural network.

The attempt to normalize and combine into one input stream is the correct approach if (1) above is the case.  However, presenting data as images requires the learning system restore the data to a less redundant, more feature encoded form, so conversion to images is not recommended.

If (2) above is the case, the provision of unique and property-independent data through the same input tensors requires the artificial network sort the input samples by type, which can be done with recurrent or attention based artificial networks, but it is unnecessary.  Here's the rule.

  If the signals from different sources are measuring independent properties to which learning will be applied, do not merge the input streams.  Instead, present them in parallel to the learning system so that the learning system does not need to also learn how to discriminate sample types.

The reason topology is mentioned in the subtitle of this section is that there is a third case, (3), which is a hybrid of (1) and (2) as hinted above.

Many excellent detection and control systems process the input channels in relation to one another prior to the actual learning system components.  This pre-processing takes advantage of known relationships in the data and the topology of signals in phase space.

The phase space of two or three signals can be visualized in 2D or 3D where each channel is assigned to a Cartesian coordinate system's axis and the time domain is not represented by any axis.  Mechanical fault detection can be optimized using phase space analysis.

An example of this more complex case (3) is in the human recognition of both consonant and vowel sounds and various other non-speech audio phenomena.  After the cochlear conversion to spectra, the transient information (as from a 'k' sound or a click) and tone information (as from an 'e' or a whistle) is presented to speech recognition nets in the brain in an integrated form.

This decreases the learning time newborns take to learn voice and household sounds and probably improved evolutionary viability in times of global stress or local predatory activity.

In automated vehicles and smart robotics, the same is true.  Some sounds, accelerations, and visible light changes are transient and some are vibrations.  Studying the topology of the surface created by the input signal types and their spectra in phase space provides much information.  Armed with the results of such studies allows excellence in the preparation of input to a learning system.

Fourth Step — Normalizing and Standardizing

Once the above is complete, understanding how to present the data to the input layer of an artificial network becomes clearer.  The goal of this design step is to have relatively equal ranges for the components of each input tensor, whether presented serially or in parallel depending on the above criteria.

Doing so reduces the burden on the learning system.  All signals are presented at full strength but within the clipping range of the signal.  This is a general principle and is important for both data center (batch) learning and real time learning and adaptive systems.


How does NEAT find the most successful generation without gradients? I'm new to NEAT, so, please, don't be too harsh. How does NEAT find the most successful generation without gradient descent or gradients?

NEAT is a genetic algorithm (GA). A genetic algorithm maintains a population of individuals (or chromosomes) and evolves it using operations like the crossover or the mutation, so that the fittest individuals keep living and most other individuals die. The nature of the individuals depends on the problem. For example, in the case of NEAT, the individuals are neural networks. However, these individuals need first to be encoded into a (compressed) reprentation (for example, a vector) that allows the operations like mutation to be efficiently applied: this representation is often called the genotype (or chromosome).

How do you decide which individuals are the fittest? A function called the fitness function needs first to be defined. The fitness function measures the fitness (or quality) of the individuals (or solutions). In the case of neural networks, a fitness function could, for example, be the accuracy of the neural networks on a validation dataset.

Why don't GAs need gradients? The fitness (or quality) of the solutions is given by the fitness function in the case of GAs, so gradients are not strictly needed, even though they can also be used.

NEAT is a little bit more complex (and it is described in detail in the paper that introduced it Evolving Neural Networks through Augmenting Topologies), but this is the basic idea behind all genetic algorithms (including NEAT).


How could we solve the TSP using a hill-climbing approach? How could we solve the TSP using a hill-climbing approach?

I will give you a basic idea of an approach.

The basic idea behind hill climbing algorithms is to find local neighbouring solutions to the current one and, eventually, replace the current one with one of these neighbouring solutions.

So, you first need to model your problem in a way such that you can find neighbouring solutions to the current solution (as efficiently as possible). But what is a solution to a TSP? It is a sequence of nodes (or vertices), such that the first node is equal to the last one (given that the travelling salesman needs to return to its initial position), no other vertex is repeated, and all vertices of the graph are included.

So, given a sequence of nodes $x_1, x_2, \dots, x_{n}, x_1$, how can you create a neighbouring solution that is valid, that is, there is not repeated vertex (apart from the initial and the last one) and all vertices are included?

Hint: recall that, in the case of TSP, we can assume that every node (or city) is connected to every other node.


Do we have to use the IOB format on labels in the NER dataset? If so, why? Do we have to use the IOB format on labels in the NER dataset (such as B-PERSON, I-PERSON, etc.) instead of using the usual format (PERSON, ORGANIZATION, etc.)? If so, why? How will it affect the performance of the model?

There's nothing stopping you from training a model with whatever tags you want.
Using what you describe as "usual" format means you would have approx half as many tags as using the IOB format. In theory this means your model will develop higher accuracy faster and with less training data. On the downside, you will need to do more work when interpreting the results in order to be confident where one named entity ends and another one begins.
I made a notebook to do some empirical tests on this which uses 17 tags in the IOB format and 9 in the "usual" format.
TL;DR using the "usual" format did not produce a noticeable improvement in model quality.


How can we print weights per iteration in a simple feed forward MLP for an specific class? im working on a project in which I have to make a multi-layer perceptron with two hidden layers with 3 nodes in each. The target value in my data contains 8 unique values/classes. One of the tasks states "For the most popular class CYT plot weight values per iteration for the last layer (3 weights and bias)".  My question is "does this statement make sense"?  I can access the weights and biases of a layer but I don't get what are  weight values for a specific class and how to access them

A common model used for this kind of classification task is to have one output neuron per class. So, for example, neuron 1 may have a loss function that is related to outputting "1" for examples of class 1, and "0" for examples of other classes. Neuron 2 may be asked to do the same, but for class 2 rather than class 1.

If you use a model of this kind, you can pull the weights for each neuron in the final output layer. It sounds like this is what you are being asked to plot.


Doesn't the number of explored nodes with IDA* increase linearly? I think I'm misunderstanding the description of IDA* and want to clarify.
IDA* works as follows (quoting from Wiki):

At each iteration, perform a depth-first search, cutting off a branch when its total cost exceeds a given threshold. This threshold starts at the estimate of the cost at the initial state, and increases for each iteration of the algorithm. At each iteration, the threshold used for the next iteration is the minimum cost of all values that exceeded the current threshold.

Suppose that we have the following tree:

branching factor = 5
all cost are different

Say we have expanded 1000 nodes. We pick the lowest cost of the nodes that we 'touched' but didn't expand. Since all costs are unique, there is now only one more node which satisfies this new cost bound, and so we expand 1001 nodes, and 'touch' 5 new ones. We now pick the smallest of these weights, and starting from the root expand 1002 nodes, and so on and so forth, 1003, 1004...
I must be doing something wrong here right? If not, the complexity is $n^2$, where n is the number of nodes with cost smaller than the optimum, compared to n for normal A*.
Someone pointing out my misunderstanding would be greatly appreciated.

No misunderstanding. IDA* is indeed n^2 vs n for A* in certain situations.
See
https://pdfs.semanticscholar.org/388c/0a934934a9e60da1c22c050566dbcd995702.pdf
for a reference.
IDA* only works in certain (not uncommon) situations. If there are many states with the same costs, and this number grows exponentially with the cost, then IDA* will be O(n). An example of such a problem is the 8 sliding piece puzzle.
IDA* will however, as noted, become exponential when all costs differ. Examples include continuous costs, or discrete values spanning a large range.
If you really wanted to apply IDA* to e.g. a map, a workable approach would likely be to round all values - or perform a log+floor operation.


Why is the last layer of a DBN or DBM used for classification task? I understand why deep generative models like  DBN ( deep belief nets ) or DBM ( deep boltzmann machines ) are able to capture underlying structures in data and use it for various tasks ( classification, regression, multimodal representations etc ...).

But for the classification tasks like in Learning deep generative models, I was wondering why the network is fine-tuned on labeled-data like a feed-forward network and why only the last hidden layer is used for classification?

During the fine-tuning and since we are updating the weights for a classification task ( not the same goal as the generative task ), could the network lose some of its ability to regenerate proper data? ( and thus to be used for different classification tasks ? )

Instead of using only the last layer, could it be possible to use a partition of the hidden units of different layers to perform the classifications task and without modifying the weights? For example, by taking a subset of hidden units of the last two layers ( sub-set of abstract representations ) and using a simple classifier like an SVM?

Thank you in advance!

One of the big realizations that deep learning models brought in recent years was that we can train the feature extractors and classifiers simultaneously. In fact most people have stopped separating the 2 tasks and simply refer to all the process as training the model.

However, if you dive in to every single model architecture, it will always be constructed from the first part which is the feature extractor which outputs the embedding output - (which is basically the x encoded features of the input), and second part consisting of the final layer the model - the classifier which uses the embedding layer encoding to predict the class of the input.

The goal of the first part is to reduce the dimensionality of the input to just the most impotent features for the final task. The goal of the classifier is to use those features to output the final score/class etc.

This is why usually only this layer is fine-tuned, because we don't want to damage the trained feature extractor, just update the classifier to fit a slightly different distribution.

I'm pretty sure that in your mentioned case, for generation they do not use the classification layer, so updating it shouldn't have any affect on the model's generative abilities.

Regarding your last question, yes it is possible, ones you extracted the features with the model, you can use any kind of classifier on them.


How do I choose the number of neurons in the fully-connected layer before the softmax layer? I am solving a classification problem with CNN. The number of classes is 5.

How can I decide the number of neurons in the FC layer before the softmax layer?
Is it $N * 5$, where $N$ is the number of classes?
Is there any documentation for deciding the number of neurons in the FC layer (before SoftMax layer)

How can I decide the number of neurons in the FC layer before the softmax layer?

Train different network architectures (with different numbers of neurons in the last FC layer). Use cross-validation - a set of data you have not trained on - to measure the performance of the network, using a metric that you have decided beforehand is a good proxy for your experiment goals. For instance, you might choose getting the highest classification accuracy as your goal - but might choose something different for an unbalanced dataset, because 90% accuracy is not meaningful.

There is not usually a good reason to over-tune your network. Trying some variations with 1.5 x or 2 x geometric series (e.g. 5, 10, 20, 40 neurons in layer) is probably enough to find a good hidden layer size.

  Is it $N * 5$, where $N$ is the number of classes?

No, in general. That may work just fine for your problem though.

  Is there any documentation for deciding the number of neurons in the FC layer (before the softmax layer)

Not really. The most direct thing to do is find solutions that work well for similar problems to yours, and adapt as necessary. So you can search for any similar problem domain and see what the researchers used there.

There is not much theory to guide you here. However, if it helps with your intuition, more neurons can make a better fit to higher frequency variations in the target function you are learning, whilst more layers will mean better handling of complex function spaces (e.g. where rules about the mapping between input and output can be made according to combining simpler rules) - provided in both cases that you have enough training data that it is possible for the NN to learn the function approximately. These are not strictly defined traits of functions for machine learning, so many NN users will work with this intuitive view.

If in doubt, assuming this is not an image, NLP or other well-studied problem, then I might just guess at e.g. 64 neurons per hidden layer, and try 1, 2, and 3 hidden layers as a starting point (all the same size). I cannot say if that will work for you and your problem, but it might help get past the "blank page effect" and start you training and testing some variations.


Does Monte Carlo tree search qualify as machine learning? To the best of my understanding, the Monte Carlo tree search (MCTS) algorithm is an alternative to minimax for searching a tree of nodes. It works by choosing a move (generally, the one with the highest chance of being the best), and then performing a random playout on the move to see what the result is. This process continues for the amount of time allotted.

This doesn't sound like machine learning, but rather a way to traverse a tree. However, I've heard that AlphaZero uses MCTS, so I'm confused. If AlphaZero uses MCTS, then why does AlphaZero learn? Or did AlphaZero do some kind of machine learning before it played any matches, and then use the intuition it gained from machine learning to know which moves to spend more time playing out with MCTS?

Monte Carlo Tree Search is not usually thought of as a machine learning technique, but as a search technique. There are parallels (MCTS does try to learn general patterns from data, in a sense, but the patterns are not very general), but really MCTS is not a suitable algorithm for most learning problems.

AlphaZero was a combination of several algorithms. One was MCTS, but MCTS needs a function to tell it how good different states of the game might be (or else, it needs to simulate entire games). One way to handle this function in a game like chess or Go is to approximate it by training a neural network, which is what the Deep Mind researchers did. This is the learning component of AlphaZero.


How can we define common sense in an AI agent? In our lives, we meet different people and describe their common sense based on how they act on a situation. For example, highly extrovert people are able to deal with people without any awkwardness. For them, an action in how to deal with people come as common sense. But, in the case of scientists, approach to solving a problem may be common sense which ordinary people cannot see.

How can we define common sense in an AI agent?

I came up with a few ideas I would argue are valuable in motivating the idea of common sense for a machine learning model.

Common sense is retrospective. We define it in terms of a past (sensible) actions and conditions, and we can say someone has good common sense on the basis of their behavior, which we can view as the sum of their historical actions and the degree to which they were sensible.
The actions alone are not sufficient for demonstrating common sense; the mental model(s) that generated those actions is important as well. Why did the individual take those sensible actions? Is their reasoning behind their action sensible as well, or did they merely get lucky (put another way, does their generally not sensical mental model produce sensible actions in certain cases and non sensical actions in most others)? This hints that common sense is contingent on the rationalization of the actions themselves.
Given the former, common sense is contingent on the enumeration of possible actions and choosing the right one, given the context of the situation. For example, I’m walking on a path and see a snail. What are some actions? I could keep walking, stop and admire it for a while, step on it, or eat it. The first two options are sensible if I value the snail as a living being. The first option is sensible if I’m in a rush. The second and fourth may be sensible if I’m a chef exploring nature for new potential ingredients. The third option if I recognize the snail as invasive. We say someone has common sense when, given the context, the actions chosen are sensible and we can reason about them intuitively.

My guess is that, the intuition behind the perceived sense of an action is what you’re after. I’d argue that, ultimately, the intuition of common sense is defined by the person developing the model and will have to do with the formulation of the model (e.g it’s assumptions, the objective function, etc). After all, common sense is subjective and context specific.

Concretely, a model can have common sense if the developer bakes it in and we can use inferential methods to demonstrate this. For example, in a Word2Vec model, we might see that $\mathsf{Paris} \mapsto \mathsf{France}$ and would expect that $\mathsf{Tokyo} \mapsto \mathsf{Japan}$. To interrogate this, we might do some vector math and find that $\mathsf{Paris} - \mathsf{France} + \mathsf{Tokyo} = \mathsf{Japan}$. How and if the AI model develops the larger association between $\mathsf{Capital} \mapsto \mathsf{Country}$, however, comes down to how the developer built and trained the model to recapitulate their own common sense.


How should we interpret this figure that relates the perceptron criterion and the hinge loss? I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.2 Relationship with Support Vector Machines says the following:

The perceptron criterion is a shifted version of the hinge-loss used in support vector machines (see Chapter 2). The hinge loss looks even more similar to the zero-one loss criterion of Equation 1.7, and is defined as follows:
$$L_i^{svm} = \max\{ 1 - y_i(\overline{W} \cdot \overline{X}_i), 0 \} \tag{1.9}$$
Note that the perceptron does not keep the constant term of $1$ on the right-hand side of Equation 1.7, whereas the hinge loss keeps this constant within the maximization function. This change does not affect the algebraic expression for the gradient, but it does change which points are lossless and should not cause an update. The relationship between the
perceptron criterion and the hinge loss is shown in Figure 1.6. This similarity becomes particularly evident when the perceptron updates of Equation 1.6 are rewritten as follows:
$$\overline{W} \Leftarrow \overline{W} + \alpha \sum_{(\overline{X}, y) \in S^+} y \overline{X} \tag{1.10}$$
Here, $S^+$ is defined as the set of all misclassified training points $\overline{X} \in S$ that satisfy the condition $y(\overline{W} \cdot \overline{X}) < 0$. This update seems to look somewhat different from the perceptron, because the perceptron uses the error $E(\overline{X})$ for the update, which is replaced with $y$ in the update above. A key point is that the (integer) error value $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$ can never be $0$ for misclassified points in $S^+$. Therefore, we have $E(\overline{X}) = 2y$ for misclassified points, and $E(X)$ can be replaced with $y$ in the updates after absorbing the factor of $2$ within the learning rate.

Equation 1.6 is as follows:

$$\overline{W} \Leftarrow \overline{W} + \alpha \sum_{\overline{X} \in S} E(\overline{X})\overline{X}, \tag{1.6}$$
where $S$ is a randomly chosen subset of training points, $\overline{X} = [x_1, \dots, x_d]$ is a data instance (vector of $d$ feature variables), $\overline{W} = [w_1, \dots, w_d]$ are the weights, $\alpha$ is the learning rate, and $E(\overline{X}) = (y - \hat{y})$ is an error value, where $\hat{y} = \text{sign}\{ \overline{W} \cdot \overline{X} \}$ is the prediction and $y$ is the observed value of the binary class variable.

Equation 1.7 is as follows:

$$L_i^{(0/1)} = \dfrac{1}{2} (y_i - \text{sign}\{ \overline{W} \cdot \overline{X_i} \})^2 = 1 - y_i \cdot \text{sign} \{ \overline{W} \cdot \overline{X_i} \} \tag{1.7}$$

And figure 1.6 is as follows:

Figure 1.6 looks unclear to me. What is figure 1.6 showing, and how is it relevant to the point that the author is trying to make?

I can't fully explain the part because I forgot what it talks about.
However, regarding the hinge loss, it is basically allowing your SVM to tolerate misclassifications without increasing the cost function.
For example, you give someone 1 dollar or 1 euro. You can forgive them, you tolerate it. Your hinge loss is 0 for lending someone 1d dollar. However, if you give them 10 dollars or 100, you will ask them to refund you ASAP because you can't tolerate that much loss!


Why can't LSTMs keep track of the "important parts" of a sequence? I keep reading about how LSTMs can't remember the "important parts" of a sequence which is why attention-based mechanisms are required. I was trying to use LSTMs to find people's name format.

For example, "Millie Bobby Brown" can be seen as first_name middle_name last_name format, which I'll denote as 0, but then there's "Brown, Millie Bobby" which is last_name, first_name middle_name, which I'll denote as 1.

The LSTM seems to be overfitting to one classification of format. I suspect it's because it's not paying special attention to the comma which is a key feature of what format it could be. I'm trying to understand why an LSTM won't work for a case like this. It makes sense to me because LSTMs are better at identifying sequence to sequence generation and things such as summarization and sentiment analysis usually require attention. I suspect another reason why the LSTM is not able to infer the format is that the comma can be placed in different indexes of the sequence, so it could be losing its importance in the hidden state the longer the sequence is (not sure if that makes sense). Anyone else has any theories? I'm trying to convince my fellow researchers that a pure LSTM won't be sufficient for this problem.

The problem is not that RNN flavours such as LSTMs are incapable of keeping track of the "important" parts of the input. They also do not have much trouble recognizing commas in different places.

To prove this point, I recommend reading Andrej Karpathy's excelllent write-up about the behaviour of individual RNN "neurons".

Addressing specifically this comment in your question:

  I suspect another reason why the LSTM is not able to infer the format is that the comma can be placed in different indexes of the sequence, so it could be losing its importance in the hidden state the longer the sequence is (not sure if that makes sense).

If commas are relevant to solve the task at hand, LSTMs can learn to remember its position or related information. This information is not necessarily diluted by repeated application of reccurrence with long sequences: networks can learn to propagate and promote crucial information from one recurrent state to the next.

Input sequences have arbitrary length, which means that LSTMs need to compress information about seen sequence elements

Rather, the input sequence has an arbitrary length, while the LSTM state vectors have a fixed size. State vectors are the only way for an LSTM to "keep track of important parts". This means that those fixed-size vectors are a bottleneck and there is an information-theoretic upper bound on the amount of information about "important parts" that can be kept by an LSTM.

LSTMs potentially take multiple decisions. For each decision, something else in the input sequence is most important

For tasks such as summarization that you mention in the question, an LSTM makes a series of predictions (predicting the tokens of the summary one token at a time). For each prediction, different things in the input sequence might be important. Put another way, for each decision, another view of the input may be most helpful.

This is a key motivation for using attention networks. Each time an LSTM is making a prediction, an attention network can provide a dynamic, optimally helpful view of the input sequence.


AI with conflicting objectives? A recent question on AI and acting recalled me to the idea that in drama, there are not only conflicting motives between agents (characters), but a character may themselves have objectives that are in conflict.

The result of this in performance is typically nuance, but also carries the benefit of combinatorial expansion, which supports greater novelty, and it occurs to me that this would be a factor in affective computing.

(The actress Eva Green is a good example, where her performances typically involve indicating two or more conflicting emotions at once.)

It occurs to me that this can even arise in the context of a formal game where achieving the most optimal outcome requires managing competing concerns.

Is there literature or examples of AI with internal conflicting objectives?

There are multi-objective optimization problems, where the objective functions may be in conflict with each other, which can potentially have multiple Pareto-optimal solutions. The paper Multi-objective optimization using genetic algorithms: A tutorial (2006) gives a good overview of the multi-objective optimization problem with genetic algorithms, which can be called evolutionary multi-objective optimization (EMO) or multi-objective optimization evolutionary algorithms (MOEAs).

A common multi-objective genetic algorithm is NSGA (or NSGA-2 and NSGA-3), which stands for Non-dominated Sorting Genetic Algorithm, which is based on the concepts of non-dominated sorting, Pareto front and optimality, niches (sub-populations), and elitism (the best individuals of the current population are carried over to the next generation).

If you want to play with MOEAs, you may wanna try the Python deap package, which supports, for example, the NSGA-2 algorithm.


Why RNNs often use just one hidden layer? Did I get it right, that RNNs most often have just one hidden neuron layer? Is there a reason for that? Will RNNs with several hidden layers in each cell work worse?
Thank you!!

Definitely you can have multiple hidden layers in RNN. One the most common approaches to determine the hidden units is to start with a very small network (one hidden unit) and apply the K-fold cross validation ( k over 30 will give very good accuracy) and estimate the average prediction risk. Then you will have to repeat the procedure for increasing growing networks, for example for 1 to 10 hidden units or more if needed.

However, in my experience, if you are interested to get best possible accuracy, you should start with small number of hidden layers and more simple structure, and if you are not satisfied with the corresponding accuracy, then we should go on increasing the learning rate by fixed but small small steps and each time start training fresh.


What is the best programming language to learn to implement genetic algorithms? What is the best and easiest programming language to learn to implement genetic algorithms? C++ or Python, or any other?

There is no "best language" for any problem. There are too many variables to consider, even when advising a single person with a single project plan.

If the choice is between Python and C++, I would generally advise:

If you want to implement from scratch and learn how the algorithm works, use Python with numeric/accelerated libraries such as NumPy or PyTorch. Python script is quicker to prototype and try different ideas, due to loose variable typing and built-in high-level structures such as dict and list.
If you want to write core, efficient libraries, then C++ will out-perform Python, but writing these will take longer. There are plenty of C++ libraries available (e.g. TensorFlow is available with a C++ API), but the community around them is less focused than with Python toolkits.

You can also combine both approaches and write specific libraries in C or C++ to improve performance at any time.

With Genetic Algorithms, the speed bottleneck is most often population assessment - e.g. running the environment simulation to get a fitness measure for each individual - and not the GA itself. So if you have a specific problem or problem domain in mind, may want to orient your choice around a language that already has support for the kind of environments where you want to run your GA. GAs usually benefit greatly from parallelisation, so if you are aiming for something ambitious you will want to look into GPU support and/or distributed computing toolkits.

Most researchers/hobbyists working in AI-related fields end up using multiple languages over time. You may end up with a favourite language environment, which might be Python, Julia, Java, C++, C, C#, Lua, LISP, Prolog, Matlab/Octave, R . . . but you will end up needing a smattering of other languages, and usually skills with specific toolkits such as TensorFlow, Scikit-Learn, Hadoop etc in order to complete projects.

Don't be afraid that you will "waste time" learning in one language initially then needing to transfer to another. Your learning of algorithms will be transferable, and your first attempts will most likely not be that re-usable as library code anyway, so you are going to re-implement your ideas, perhaps many times. My first simulated annealing project was written in Fortran 77 . . . 20 years later I dug that knowledge up again and implemented in Ruby/C - nowadays I work in Python for the AI/hobby stuff even though my professional career sees me working mostly in Ruby.


Can prior knowledge be encoded in deep neural networks? I was reading Gary Marcus's a Critical Appraisal of Deep Learning. One of his criticisms is that neural networks don't incorporate prior knowledge in tackling a problem. My question is: have there been any attempts at encoding prior knowledge in deep neural networks?

Neural nets incorporate prior knowledge. This can be done in two ways: the first (most frequent and more robust) is in data augmentation. For example in convolutional networks, if we know that the "value" (whatever that is, class/regression) of the object we are looking is rotational/translational invariant (our prior knowledge), then we augment the data with random rotations/shifts. The second is in the loss function with some additional term.


What does "death" intuitively mean in the paper "Death and Suicide in Universal Artificial Intelligence"? In the paper Death and Suicide in Universal Artificial Intelligence, a proposal is given for what death could mean for Artificial Intelligence.

What does this mean using English only? I understand that mathematical notation is useful for giving a precise definition, but I'd like to understand what the definition really means.

The authors do actually give an English definition in terms of the well-known agent formulation of AI:

  We intend this usage to be intuitive: death means that one sees
  no more percepts, and takes no more actions.

It would seem that this becomes possible for a reinforcement learning agent such as AIXI in a formulation that uses semi-measures of probability (which need not sum up to 1), rather than the more traditional notion.


AI in dota2 vs AI in starcraft I am just curious what AI would be harder to create from a strictly engineering point of view. AI which would win 1vs 1 game with the best player in starcraft or AI which would control a team the whole team in dota2 and win against the best team?

I can't answer definitively without a detailed breakdown of game mechanics of dota2 vs. Starcraft, but assuming the games have similar complexity, the AI controlling multiple in-game agents that form a team would be more complex, and therefore more difficult to create, than a single agent "team".

My supposition is based on the fundamentals of combinatorics and computational complexity:

in dota2, the AI has to not only to identify winning strategies, but coordinate multiple agents.

by contrast:

the Starcraft AI only has to identify a winning strategy, with no coordination between discrete agents which form a team.

Coalitions have traditionally been difficult to analyze in Game Theory and Combinatorial Game Theory (in the latter case especially, restricting mathematical study, for the most part, to games involving only 2 players.)  In this case the issue is not coalitions--the multiple game agents are already on the same team--but complexity is nevertheless dramatically increased by the existence of multiple agents on a team as compared to a team consisting of a single agent.


What is the impact of using multiple BMUs for self-organizing maps? Here's a sort of a conceptual question. I was implementing a SOM algorithm to better understand its variations and parameters. I got curious about one bit: the BMU (best matching unit == the neuron that is more similar to the vector being presented) is chosen as the neuron that has the smallest distance in feature space to the vector. Then I updated it and its neighbours.
This makes sense, but what if I used more than one BMU for updating the network? For example, suppose that the distance to one neuron is 0.03, but there is another neuron with distance 0.04. These are the two smallest distances. I would use the one with 0.03 as the BMU.
The question is, what would be the expected impacts on the algorithm if I used more than one BMU? For example, I could be selecting for update all neurons for which the distance is up to 5% more than the minimum distance.
I am not asking for code. I can implement it to see what happens. I am just curious to see if anyone have any insight on the pros and cons (except additional complexity) of this approach.

The following is my intuition about the behavior of the algorithm considering your question, based on my knowledge and experience with single BMU SOMs. I didn't verify it experimentally.
At early stages of training:
It should disturb the SOM topology preserving properties, as you're assigning the same pattern to different (and probably distant) locations in the lattice, making the distribution multimodal. This is not terrible on its own, as different positions could encode different relationships to neighbors, overcoming the limitations of a single 2D/3D neighborhood. But if your SOM is not large enough, the multiple modes may collapse, resulting in unimodal distributions covering large portions of the map, wasting space. It all depends on your training schedule.
At later stages of training:
If you started with a single BMU at early stages, it wouldn't provide any meaningful impact, as a SOM, by definition, clusters similar patterns at close positions of its lattice. In other words, the neurons closest to your input are already inside the BMU update radius. If you trained with multiple BMUs since the beginning, you would keep finding / activating the multiple maxima of your multimodal distributions for each pattern. The dimensionality reduction would become even more nonlinear than it already is with single BMUs. This may or may not be a problem, depending on your application.


Is a genetic algorithm efficient for a snake game? I am working on a DIY project in which I want to be able to train a neural network to play Snake.

Is a genetic algorithm an efficient way of training a network for this application?

For a GA, what should the inputs of the network be? (distance to walls and fruit or the squares in the proximity of the snake head as a vector)

What would the difference in efficiency be depending on the algorithm and what limitation does each one have? Are there any other alternatives I should consider?

Q-Learning and Genetic algorithm are both good algorithms to create an IA that plays Snake.
The one you use depends mostly on how you understand and model your IA environment.

Q-Learning algorithm is an algorithm that needs a State (give by
the environment), Actions it can take, and Rewards to give
him according to how it performs.
Genetic algorithm needs to have intrinsic parameters (could be caracteristics of the network taking the decisions, or more simple things like leg size / muscle strenght if you want to make an IA that runs), how to merge 2 parents to create children, and a Metric to evaluate how the network performed.

I assume your snake is make of a grid (like old snake where you only have 4 directions, you can't go in diagonal directions).
In snake example, here is how I would define it in both case (careful, this is how I would model the problem, there might be more efficient models) :
Q-Learning

State : what is in each block of the grid (snake tail, snake head, food)
Actions : Up, Down, Right, Left
Rewards : Not dying : +0.01; dying : -10; eating piece of food: +1.

Genetic Algorithm

Intrinsic Parameters : Parameters of the network that takes decisions (inputs of the network could be whole state of the environment as described in Q-Table).
How to merge : ?
Metric : Lenght of the snake or time lived.

I can't find exactly how to make a genetic algorithm, but it should be possible.
Hope it helps. Q-Learning is usually easier so this is what I would use, but feel free to use whatever you want.


How can genetic programming be used in the context of auto-encoders? I am trying to understand how genetic programming can be used in the context of auto-encoders. Currently, I am going through 2 papers

Training Feedforward Neural Networks Using Genetic Algorithms (a classific one)

Training Deep Autoencoder via VLC-Genetic Algorithm

However, these papers don't really help me to grasp the concept of genetic programming in this specific context, maybe because I'm not very familiar with GP.
I understand that autoencoders are supposed to reconstruct the instances of the particular classes they have been trained on. If another fed instance is not reconstructed as expected, then it could be called an anomaly.
But how can genetic programming be used in the context of auto-encoders? You are still required to create a neural network, but, instead of a feed-forward one, you use autoencoder, but how exactly?
I would appreciate any tutorials or explanations.

The first classic paper your present uses a genetic algorithm (GA), while you mention genetic programming (GP). In short, a GA uses an evolutionary process to evolve a fixed-length vector, normally a bit string, while a GP evolves variable-length decision trees, which could be a computer program or mathematical function.
An autoencoder is something that automatically encodes or decodes something. For example, a noisy image is decoded to the original image (very simplified I know), the point here is automatically, so, without any defined process, an autoencoder can "learn" how to remove the noise from the image.
Generally, an autoencoder is made using a neural network (NN). The noisy image is read in as input and the output is a clear image. To train the NN, backpropagation is used. For backpropagation to work, an error is needed to calculate the weight changes. Alternatively, the weights could be encoded as a vector. A fixed-length vector is then evolved using a GA to evolve the weights of the NN. This is what the "classic paper" mentioned does.
Using a GA to evolve the weights of a NN is not really efficient, because weights affect each other which means each time crossover happens the weights are totally messed up and the GA can easily get stuck in a local optimum. Gradient descent is probably more efficient.
The structure of a multi-layered network is important. Here, gradient descent is still used to update the weights, but a GP could be used to evolve the structure of the NN.
Alternatively, the GP itself could be used to evolve an autoencoder without using a NN. Here, the chromosome will contain nodes such as mathematical operations or computer code that would alter the input to produce the output.


How to show temporal difference methods converge to MLE? In chapter 6 of Sutton and Barto (p. 128), they claim temporal difference converges to the maximum likelihood estimate (MLE). How can this be shown formally?

The convergence and optimality proofs of (linear) temporal-difference methods (under batch training, so not online learning) can be found in the paper Learning to predict by the methods of temporal differences (1988) by Richard Sutton, specifically section 4 (p. 23). In this paper, Sutton uses a different notation than the notation used in the famous book Reinforcement Learning: An Introduction (2nd ed.), by Sutton and Barto, so I suggest you get familiar with the notation before attempting to understand the theorem and the proof. For example, Sutton uses letters such as $i$ and $j$ to denote states (rather than $s$), $z$ to denote (scalar) outcomes and $x$ to denote (vector) observations (see section 3.2 for example of the usage of this notation).

In the paper The Convergence of TD($\lambda$) for General $\lambda$ (1992), Peter Dayan, apart from recapitulating the convergence proof provided by Sutton, he also shows the convergence properties of TD($\lambda$) and he extends Watkins' Q-learning convergence theorem, whose sketch is presented in his PhD thesis Learning from Delayed Rewards (1989), and defined in detail in Technical Note: Q-learning (1992), by Dayan and Watkins, to provide the first strongest guarantee or convergence proof for TD(0).

There is much more research work on the convergence properties of TD methods, such as Q-learning and SARSA. For example, in the paper On the Convergence of Stochastic Iterative Dynamic Programming Algorithms (1994), where Q-learning is presented as a stochastic form of dynamic programming methods, the authors provide a proof of convergence for Q-learning by making direct use of stochastic approximation theory. See also Convergence of Q-learning: a simple proof by Francisco S. Melo. In the paper Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms, the authors provide a proof of the convergence properties of on-line temporal difference methods (e.g. SARSA).


What is an agent in Artificial Intelligence? While studying artificial intelligence, I have often encountered the term "agent" (often autonomous, intelligent). For instance, in fields such as Reinforcement Learning, Multi-Agent Systems, Game Theory, Markov Decision Processes.

In an intuitive sense, it is clear to me what an agent is; I was wondering whether in AI it had a rigorous definition, perhaps expressed in mathematical language, and shared by the various AI-related fields.

What is an agent in Artificial Intelligence?

The acclaimed book Artificial Intelligence: A Modern Approach (by Stuart Russell and Peter Norvig) gives a definition of an agent

  An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.

This definition is illustrated by the following figure

This definition (and illustration) of an agent thus does not seem to include the agent as part of the environment, but this is debatable and can be a limitation, given that the environment also includes the agent.

According to this definition, humans, robots, and programs are agents. For example, a human is an agent because it possesses sensors (e.g. the eyes) and actuators (e.g. the hands, which, in this case, are also sensors) and it interacts with an environment (the world).

A percept (or perception) is composed of all perceptual inputs of the agents. The specific definition of the percept changes depending on the specific agent. For example, in the case of a human, the percept consists of all perceptual inputs from all sense organs of the human (eyes, ears, tongue, skin, and nose). In the case of a robot only equipped with a camera, the percept consists only of the camera frame (at a certain point in time). A percept sequence is a sequence of percepts.

An action is anything that has an effect on the environment. For example, in the case of a legged robot, an action can be "move forward".

An action is chosen by the agent function (which is illustrated by the white box with a black question mark in the figure above), which can also be called policy. The agent function highly determines the intelligent or intellectual capabilities of the agent and differentiates it from other agents.

Therefore, there are different agents depending on the sensors and actuators they possess, but, more importantly, depending on their policy, which highly affects their intellectual characteristics. A possible categorization of agents is

rational agents do the "right" thing (where "right", of course, depends on the context)
simple reflex agents select actions only based on the current percept (thus ignoring previous percepts)
model-based reflex agents build a model of the world (sometimes called a state) that is used to deal with cases where the current percept is insufficient to take the most appropriate action
goal-based agents possess some sort of goal information that describes situations that are desirable; for example, in the case of a human, a situation that is desirable is to have food
utility-based agents associate value with certain actions more than others; for example, if you need immediate energy, chocolate might have more value than some vegetable
learning agents update their e.g. model based on the experience or interaction with the environment

More details regarding these definitions can be found in section 2 of the book mentioned above (3rd edition). However, note that there are other possible categorizations of agents.

A reinforcement learning (RL) agent (which is a program) is an agent that interacts with an environment and can learn a policy (a behavioral strategy or agent function) or value (or utility) function (from which the policy can be derived) from this interaction, where the agent takes an action from the current state (or model) of the environment and the environments emits a percept, which, in the case of RL, consists of a reinforcement (or reward) signal and the next state. The goal of the RL agent is to maximize the cumulative reward (or reinforcement) signal. An RL agent can thus be considered a rational, goal, utility-based and learning agent. It can also be (or not) a simple reflex and model-based agent.


How to process data in a data stream for a LSTM How can a data stream for a RNN (LSTM) be handled, when the stream contains data sets belonging to different prediction classes?

Training phase:
I have trained a LSTM to predict a class out of a sequence of Letters. For the training phase I used a fixed data array where the beginning an the ending of a sequence belonged to a class. Of course there is a little noise but the whole data set was labled with a class. E.g:

Seq.    is  Class
ABC     is  One
CBA     is  Two
ABD     is  Three

The network predicts well when it sees a static data array.

Problem in Prediction Phase:
During prediction the LSTM will receive a data stream where there is a sequence off arrays but there is no delimiter. The data set can not be distinguished or separated. I am not sure how it would perform when I have a data stream for different classes like
    ABCABCCBAABD.

I guess in speech recognition one must face similiar problems.

One thing I might add is not to expect the ML model to take care of everything, that is where ML engineering comes into play. One suggestion I have, without knowing much about your data stream, is that you need to implement a real-time data ingestion/pipeline, like Apache Kafka Stream (so similar implementation), where you can decompose your stream into output that your model was trained for. You could split by dictionary, pattern or whatever you need, then push the split output into another real-time stream that can be ingested by your model. Not a lot to go for, but hopefully it helps a little.


Which machine learning algorithm could I use to break up a poem by lines? I want to create a network to predict the break up of poetry lines. The program would receive as input an unbroken poem, and would output the poem broken into lines.
For example, an unbroken poem could be
And then the day came, when the risk to remain tight in a bud was more painful

which should be converted to
And then the day came,
when the risk
to remain tight
in a bud
was more painful
than the risk
it took
to Blossom.

How should I go about this? I have been using classifiers for various tasks, but this seems to be a different type of task.
I'm thinking of it as an array of words (doesn't matter how they're represented for now), which would look like
[6, 32, 60, 203, 40, 50, 60, 230 ...]

and needs to map into an array representing line breaks
[0, 0, 1, 0, 0, 0, 1, 0, 0, 1 ...]

where 1 (at optimal) means there should be a line break after the word in that index (in this idea, the two arrays are of the same length).
Unfortunately, I couldn't find an algorithm that could train a network of this shape.
What machine learning or deep learning algorithm can be used for this task?

You should try to use an RNN. You feed in letter by letter and have a binary output of linebreak - no linebreak. If you have enough data it might actually work.


Why am I getting the incorrect value of lambda? I am trying to solve for $\lambda$ using temporal-difference learning. More specifically, I am trying to figure out what $\lambda$ I need, such that $\text{TD}(\lambda)=\text{TD}(1)$, after one iteration. But I get the incorrect value of $\lambda$.

Here's my implementation.

from scipy.optimize import fsolve,leastsq
import numpy as np

 class TD_lambda:
        def __init__(self, probToState,valueEstimates,rewards):
            self.probToState = probToState
            self.valueEstimates = valueEstimates
            self.rewards = rewards
            self.td1 = self.get_vs0(1)

        def get_vs0(self,lambda_):
            probToState = self.probToState
            valueEstimates = self.valueEstimates
            rewards = self.rewards
            vs = dict(zip(['vs0','vs1','vs2','vs3','vs4','vs5','vs6'],list(valueEstimates)))

            vs5 = vs['vs5'] + 1*(rewards[6]+1*vs['vs6']-vs['vs5'])
            vs4 = vs['vs4'] + 1*(rewards[5]+lambda_*rewards[6]+lambda_*vs['vs6']+(1-lambda_)*vs['vs5']-vs['vs4'])
            vs3 = vs['vs3'] + 1*(rewards[4]+lambda_*rewards[5]+lambda_**2*rewards[6]+lambda_**2*vs['vs6']+lambda_*(1-lambda_)*vs['vs5']+(1-lambda_)*vs['vs4']-vs['vs3'])
            vs1 = vs['vs1'] + 1*(rewards[2]+lambda_*rewards[4]+lambda_**2*rewards[5]+lambda_**3*rewards[6]+lambda_**3*vs['vs6']+lambda_**2*(1-lambda_)*vs['vs5']+lambda_*(1-lambda_)*vs['vs4']+\
                                (1-lambda_)*vs['vs3']-vs['vs1'])
            vs2 = vs['vs2'] + 1*(rewards[3]+lambda_*rewards[4]+lambda_**2*rewards[5]+lambda_**3*rewards[6]+lambda_**3*vs['vs6']+lambda_**2*(1-lambda_)*vs['vs5']+lambda_*(1-lambda_)*vs['vs4']+\
                                (1-lambda_)*vs['vs3']-vs['vs2'])
            vs0 = vs['vs0'] + probToState*(rewards[0]+lambda_*rewards[2]+lambda_**2*rewards[4]+lambda_**3*rewards[5]+lambda_**4*rewards[6]+lambda_**4*vs['vs6']+lambda_**3*(1-lambda_)*vs['vs5']+\
                                        +lambda_**2*(1-lambda_)*vs['vs4']+lambda_*(1-lambda_)*vs['vs3']+(1-lambda_)*vs['vs1']-vs['vs0']) +\
                    (1-probToState)*(rewards[1]+lambda_*rewards[3]+lambda_**2*rewards[4]+lambda_**3*rewards[5]+lambda_**4*rewards[6]+lambda_**4*vs['vs6']+lambda_**3*(1-lambda_)*vs['vs5']+\
                                        +lambda_**2*(1-lambda_)*vs['vs4']+lambda_*(1-lambda_)*vs['vs3']+(1-lambda_)*vs['vs2']-vs['vs0'])
            return vs0

        def get_lambda(self,x0=np.linspace(0.1,1,10)):
            return fsolve(lambda lambda_:self.get_vs0(lambda_)-self.td1, x0)

The expected output is: $0.20550275877409016$, but I am getting array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

I cannot understand what am I doing incorrectly.

TD = TD_lambda(probToState,valueEstimates,rewards)
TD.get_lambda()
# Output : array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

I am just using TD($\lambda$) for state 0 after one iteration. I am not required to see where it converges, so I don't update the value estimates.

$TD(\lambda)$ return has the following form:
\begin{equation}
G_t^\lambda = (1 - \lambda) \sum_{n=1}^{\infty} \lambda^{n-1} G_{t:t+n}
\end{equation}
For you MDP $TD(1)$ looks like this:
\begin{align}
G &= 0.64 (r_0 + r_2 + r_4 + r_5 + r_6) + 0.36(r_1 + r_3 + r_4 + r_5 + r_6)\\
G &\approx 6.164
\end{align}
$TD(\lambda)$ looks like this:
\begin{equation}
G_0^\lambda = (1-\lambda)[\lambda^0 G_{0:1} + \lambda^1 G_{0:2} + \lambda^2 G_{0:3} + \lambda^3 G_{0:4} + \lambda^4 G_{0:5} ]
\end{equation}
Now each $G$ term separately:
\begin{align}
G_{0:1} &= 0.64(r_0 + v_1) + 0.36(r_1 + v_2) \approx 7.864\\
G_{0:2} &= 0.64(r_0 + r_2 + v_3) + 0.36(r_1 + r_3 + v_3) \approx -5.336\\
G_{0:3} &= 0.64(r_0 + r_2 + r_4 + v_4) + 0.36(r_1 + r_3 + r_4 + v_4) \approx 25.864\\
G_{0:4} &= 0.64(r_0 + r_2 + r_4 + r_5 + v_5) + 0.36(r_1 + r_3 + r_4 + r_5 + v_5) \approx -11.936\\
G_{0:5} &= 0.64(r_0 + r_2 + r_4 + r_5 + r_6 + v_6) + 0.36(r_1 + r_3 + r_4 + r_5 + r_6
 + v_6) \approx -0.336
\end{align}
Finally, we need to find $\lambda$ so that the return is equal to $TD(1)$ return, we have:
\begin{equation}
6.164 = (1 - \lambda)[7.864 - 5.336\lambda + 25.864\lambda^2 - 11.936\lambda^3 - 0.336\lambda^4]
\end{equation}
When you solve this equation, one of the solutions is  $0.205029$ which is close to what you needed to get considering the numerical errors. Your problem was that you only considered probability in first state, but that decision prolongs to future states as well.

EDIT

As pointed out by bewestphal, this is not a full solution, it misses one crucial step to get it fully correct. Hint for it can be found in his answer and that's the correct solution.


How well can a CNN distinguish an object from its class? A convolutional neural network (CNN) can easily predict the class of an object in an image.

Can a CNN distinguish the Pisa Tower from other buildings, or Hagia Sophia from other mosques easily? If it can, how many training images can be sufficient? Do I need thousands of training images of that specific thing to distinguish it?

(This is a term project recommendation about deep neural networks, so I need to understand its feasibility.)

To help you understand the feasibility of your project these posts could be a good start:
https://datascience.stackexchange.com/questions/13181/how-many-images-per-class-are-sufficient-for-training-a-cnnenter link description here

https://stats.stackexchange.com/questions/226672/how-few-training-examples-is-too-few-when-training-a-neural-network

That being said, the short answer would be it depends. It depends how precise you want to be, what is the difficulty of the task, the infrastructure you have for the training etc.

For the images, you should not worry, you could start with the ImageNet dataset:
For construction and buildings:
http://www.image-net.org/explore?wnid=n04341686

For mosquees:
http://www.image-net.org/synset?wnid=n03788195

You can then use data augmentation techniques to enhance the size of you training set. Here is a library I have used in the past which helped me greatly to achieve this task: https://github.com/aleju/imgaug

Hope that helps!


Why use a recurrent neural network over a feedforward neural network for sequence prediction? If recurrent neural networks (RNNs) are used to capture prior information, couldn't the same thing be achieved by a feedforward neural network (FFNN) or multi-layer perceptron (MLP) where the inputs are ordered sequentially?

Here's an example I saw where the top line of each section represents letters typed and the next row represents the predicted next character (red letters in the next row means a confident prediction).

Wouldn't it be simpler to just pass the $X$ number of letters leading up to the last letter into an FFNN?

For example, if $X$ equaled 4, the following might be the input to the FFNN

S, T, A, C => Prediction: K

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this may work but the network just treat the data as a bunch of data without any specific indication of time. The network can learn the time representation only through gradient descent. RNN or LSTM however have "time" as a mechanism built into the model. The model loops through the model sequentially and have a real "sense of time" even before the model is trained. The model also have "memory" of previous data points to help the prediction. The architecture is based on the progress of time and the gradient are propagated through time as well. This is a much more intuitive way to process time-series data.

A 1D CNN also will work for the task. An example of CNN in time series data is wavenet, which uses CNN for generating incredibly life like speech using dilated convolution neural network. For whether LSTM or CNN works better, it depends on the data. You should try experimenting with both networks to see which works best.

Suppose you need to classify a video's genre. It is much simpler to watch it in sequence then seeing frames of it playing randomly in front of your eyes. This is why an RNN or an LSTM works better in time series data.


What is the next state for a two-player board game? I'm using Q-learning to train an agent to play a board game (e.g. chess, draughts or go).

The agent takes an action while in state $S$, but then what is the next state (that is,  $S'$)? Is $S'$ now the board with the piece moved as a result of taking the action, or is $S'$ the state the agent encounters after the other player has performed his action (i.e. it's this agent's turn again)?

If your opponent has fixed knowledge (it doesn't learn), then the next state after your agent did an action is the state when your turn is back. So the actions of other players are considered as an environment reaction to your actions.

But if your opponent can learn, you may create a Multi-agent Reinforcement Learning