How many iterations are required for iterative-lengthening search when step costs are drawing from a continuos range [ϵ, 1]? This is AI: A Modern Approach, 3.17c. The solution manual gives the answer as $\frac{d}{\epsilon}$, where $d$ is the depth of the shallowest goal node.

Iterative lengthening search uses a path cost limit on each iteration, and updates that limit on the next iteration to the lowest cost of any rejected node.

I have seen this question posted elsewhere as, "What is the number of iterations with a continuous range $[0, 1]$ and a minimum step cost $\epsilon$?" In that case, I agree that the minimum number of iterations is $\frac{d}{\epsilon}$ because you would need to increase the path cost limit by a minimum of $\epsilon$ with each iteration.

However, with a continuous range of $[\epsilon, 1]$, it seems there is an infinite range and that the number of iterations is potentially infinite, since there is no minimum step cost. Should this solution actually be infinite?

In the title of the question, you write (emphasis mine):

  step costs are drawn from a continuous range $[\epsilon, 1]$

If step costs are drawn from that range, it means that every step has a cost of at least $\epsilon$, and at most $1$. This leads back to the case that described in the question that you already understand.

Note that the book (at least, the third edition of the book) does state in the question that $0 < \epsilon < 1$. Of course, the question would not make much sense in the first place if negative costs were allowed.


How does one make it obvious that the structure of a neural network should be what it is? I am a beginner: I've only read a book about neural network and barely implemented one in C.

In short:

A neural network is built out of nodes,
Each node holds an output: activation.(sum.(x * w)),
We then compute the total error out of the network output.

From a beginner perspective, hyper-parameters, such as the number of layers needed, seem to be defined arbitrarily in most tutorials, books. In fact, the whole structure seems to be quite arbitrarily defined. In practice, hyper-parameters are often defined based on some standards.

My question is, if you were to talk to a total beginner, how would you explain to him the structure of a neural network in such a way that the whole thing would appear as obvious ? Is that even possible ?

Here, the word structure refers to a neural network being a configuration of nodes inside layers.

Thanks to anyone pointing out ambiguities or spelling errors.

Edit: note that I actually understand the whole back-propagation algorithm. I have no problem visualizing a nn.

While, as you begin to hit on, there are general guidelines to follow when building a neural network, they are far from standardized. This is because even though AI is a reasonably old field(1950s), neural networks have only been the tool of choice for less than a decade. Before, NNs did horribly, due to lack of data, and computation along with some less than efficient architectures.

With that being said Hinton's general rule is to add nodes/layers till the model begins to overfit and then add dropout(pretty reasonable in practice).

As such, the whole field is essentially an art as much as a science currently with only basic guidelines to follow based on your problem and data. This is part of the beauty though imo, with there being so much left to discover.

Hope that helped answer your question!


What are some resources to improve practical expertise in machine learning? I studied machine learning when I was in university, a couple of years ago. I used it for my master thesis (decision trees, ensembles, SVM, and word embedding mostly) and for other projects either personal and academics (genetic algorithms, Q-learning).
Then I studied neural networks by myself, from the pure theory of the perceptron, backpropagation, gradient descent, etc. (with Bengio's deep learning book) to the various ramifications of ConvNets, DeConvNets, auto-encoders, LSTM, deep reinforcement learning, and other various papers. I took a several months break and there are already new types of networks that I don't know, so hard to keep up with the state-of-the-art.
However, my knowledge is purely theoretical. I want to embed machine learning in my future career, but getting a job in the field requires practical expertise, which I lack.
I am working full time now as a software engineer. So, what do you think may be a possible path to follow in order to gain expertise? Given that I have a strong mathematical background and more than 10 years of coding on my back, is there any resource on practical rules of thumbs, suggestions, real case studies, etc., which I can invest time onto? I mean something beyond the vanilla "introduction to machine learning", something more real and concrete.
Do you think that attempting Kaggle challenges from scratch would be stimulating? Or too confusing? Do you think delivering a personal project (i.e. not a byproduct of my current job) in which I employ one such method to solve a potentially real problem would be enough to be relevant in my cv?

Since you are a software engineer, you already have some experience with one or more programming languages, so this is already a good thing. Moreover, you say that you have a strong mathematical background, so you will probably learn certain machine learning concepts more easily.
When it comes to machine learning, there are open source projects that you can contribute to via Github, and this really boosts your skills and experience.
I also am contributing to an open-source conversation agent via Github.
If you are able to understand and contribute to such projects, then implementing ML/DL algorithms is possible in your target language. I myself have been using Python, but that's just a personal choice, and it's been utilized by many AI engineers and data scientists, globally.

Do you think that attempting kaggle challenges from scratch would be
stimulating? Or too confusing? Do you think delivering a personal
project (i.e. not a byproduct of my current job) in which I employ one
of such methods to solve a potentially real problem would be enough to
be relevant in my cv?

Contributing to open-source projects will link you to other engineers in the same field, and this will eventually open you many paths.
Also, try the book: Artificial Intelligence: A Modern Approach Stuart Russel, Peter Norvig, Prentice Hall, 3rd edition, 2009


Is any classifier not subject (or less susceptible) to fooling? Is any classifier not subject to fooling as in here?

I agree that the question is related to the other one as commented by Philip. But I guess it is not completely a duplication as pointed out by hisairnessag3. What I wanted to ask is that any classifiers inherently do not subject (or less prone) to attack. I have a feeling that non-linear classifiers should be less susceptible to attack. Btw, any benchmark on say, simple K-nearest neighbor classifiers is available?

I would say this is not necessarily a duplicate but quite similar to some other questions. However, I will answer the question posed here.

At a theoretical level, what you are asking is there any algorithm that cannot be tricked into predicting the wrong class?

The answer is: No

It is analogous to asking whether there is a perfect architecture for an arbitrary classification problem, and that answer is quite obviously not. It would also not likely to be terribly difficult to show this is provably the case, at least for a class of algorithms(i.e connectionist models).


How do I improve accuracy and know when to stop training? I am training a modified VGG-16 to classify crowd density (empty, low, moderate, high). 2 dropout layers were added at the end on the network each one after one of the last 2 FC layers.

network settings:

training data contain 4381 images categorized under 4 categories (empty, low, moderate, high), 20% of the training data is set for validation. test data has 2589 images.
training is done for 50 epochs.(training validation accuracy drops after
50 epochs)
lr=0.001, decay=0.0005, momentum=0.9
loss= categorical_crossentropy
augmentation for (training, validation and testing data): rescale=1./255, brightness_range=(0.2,0.9), horizontal_flip

With the above-stated settings, I get the following results:

training evaluation loss: 0.59, accuracy: 0.77
testing accuracy 77.5 (correct predictions 2007 out of 2589)

Regarding this, I have two concerns:

Is there anything else I could do to improve accuracy for both training and testing?
How can I know if this is the best accuracy I can get?

Is there anything else I could do to improve accuracy for both training and testing?

Yes, of course, there are a lot of methods if you want to try to improve your accuracy, some that I can mention:

Try to use a more complex model: ResNet, DenseNet, etc.
Try to use other optimizers: Adam, Adadelta, etc.
Tune your hyperparameters (e.g. change your learning rate, momentum, rescale factor, convolution size, number of feature maps, epochs, neurons, FC layers)
Try to analyze your data, with ~75% and 4 categories, is that possible there is one category that difficult to classified?

In essence, you have to do a lot of experiments with your model until you think "it is enough" (if you have a deadline). If you don't have a hard deadline, you can keep improving updating your ML model.

  How can I know if this is the best accuracy I can get?

No, you can't until you compare it with other models/hyperparameters. If you do more experiments (some ways like the one I mentioned above) or compare with other people's experiments that using the same data, you'll find which one is the best. For an academic paper, for example, you need to compare at least 3 to 4 models that similar or experiment with hundreds of different hyperparameters combination.


Power efficiency of human brains vs. neural networks How big artificial neural networks can we run now (either with full train-backprop cycle or just evaluating network outputs) if our total energy budget for computation is equivalent to human brain energy budget (12.6 watts)?

Let assume one cycle per second, which seems to roughly match the firing rate of biological neurons.

126 million artificial neurons at 12.6 Watts, with IBM's True North

Back in 2014, IBM's True North chip was pushing 1 million neurons at less than 100mW.

So that's roughly 126 million artificial neurons at 12.6 Watts.

A mouse has 70 million neurons.

IBM believes they can build a human-brain scale True North mainframe at a "mere" 4kW.

Once 3D transistors come to market, I think we'll catch up to animal brain efficiency pretty fast.


If I am interested in theoretical computer science, is AI a bad choice? I hope this question is ok here, but since I have found a tag which deals with these issues (profession), I'll ask away. I also hope this may be useful to other people with similar doubts, since I am failing to find valuable information on this topic online.

I am interested in the theoretical side of CS, such as computability, logic, complexity theory and formal methods. At the same time, I am deeply fascinated by Artificial Intelligence and the questions it poses to our understanding of the notion of intelligence and what does it mean to be a human being.

In general, is AI a more "applied"/engineeristic field, or are there theoretical aspects to research in?

In short: If I prefer formal/theoretical compsci, is AI a bad career choice?

(note: I am asking this because I am a CS undergrad considering getting into a AI MSc).

There are certainly results in theoretical computer science / pure math with deep implications for AI. But to my knowledge these results typically aren't labeled as results of artificial intelligence, but as something more congruent in that particular field (For example in CS, we might say "agent with unbounded computational power"; in math might say some statement is "decidable/undecidable" with respect to some system). Of course they still matter in the field of AI, but you need to know what you are looking for.

See my question What are some implications of Gödel's theorems on AI research? for some examples. Or you can look up MIRI's research guide for a better idea of what existing work is out there that links formal math / CS to AI research.

Another point to raise is that there is no good definition of AI in fields outside of normal discourse (or even within, perhaps), so its difficult to decide what discussions pertains to the study of AI. Questions like whether ZFC with/without choice is expressive enough might not be on the mind of most AI researchers, but could still have some implications.

So to answer you question more directly, there is certainly a field of study regarding theoretical AI. Regarding whether or not its a good choice is something for you to decide, but it is (in my humble and not-very-well-educated opinion) very difficult field that isn't very popular, and has not seen major progress in many years.


Understanding Generalized Advantage Estimate in reinforcement learning I was reading the paper on Generalized Advantage Estimate. It first introduces a generalized form of policy gradient equation without involving $\gamma$ and then it says the following:

We will introduce a parameter $\gamma$ that allows us to reduce variance by downweighting rewards corresponding to delayed effects, at the cost of introducing bias. This parameter corresponds to the discount factor used in discounted formulations of MDPs, but we treat it as a variance reduction parameter in an undiscounted problem.

I know the Monte Carlo estimate of value function is given as:
$$V(s_t)=\sum_{l=t}^\infty \gamma^tr_t$$
The bootstrapped estimate of value function is given as:
$$V(s_t)=r_t+\gamma V(s_{t+1})$$
(In both equations, $\gamma$ is a discount factor.)
The bootstrapped estimate is biased because it based on $V(s_{t+1})$ which is usually a biased estimate by some estimator such as a neural network. The Monte Carlo estimate is unbiased because it contains all rewards sampled from the environment. In this case, however, because the agent might take a lot of actions over the course of an episode, it's hard to assign credit to the right action, which means that a Monte Carlo estimate will have a high variance.
Does this contradict what the paper says: "a parameter $\gamma$ that allows us to reduce variance"? Or does it simply mean the following: lower $\gamma$ gives smaller weights to distant future rewards, thus making value estimate less dependent on them, reducing variance in comparison to larger $\gamma$, which make distant future rewards contribute significantly to value estimate. So introduction/existence of $\gamma$ itself does not reduce the variance but gives way to increase or decrease the variance.

Bootstrapped estimate is biased because it based on $V(s_{t+1})$ which is usually a biased estimate by some estimator such as neural network.

I don't think this statement is totally correct in the context of the paper. Quoting the paper:

Taking $\gamma < 1$ introduces bias into the policy gradient estimate, regardless of the value function’s accuracy. On the other hand, $ \lambda < 1$ introduces bias only when the value function is inaccurate.

So the initial $\gamma$-discounting introduces bias regardless of the accuracy of the $V(s)$.
Same point is reiterated in the footnote on the page 3:

Note, that we have already introduced bias by using $A^{\pi,\gamma}$ in place of $A^\pi$; here we are concerned with obtaining an unbiased estimate of $g^\gamma$, which is a biased estimate of the policy gradient of the undiscounted MDP.

To summarize the above:

the original MDP under consideration is undiscounted
authors introduce $\gamma$-discounting to reduce variance (at cost of bias) of the PG estimate - this happens even if we had accurate $V(s)$
later, authors introduce another $\lambda$-discounting which also controls bias-variance tradeoff - now due to inaccuracy in $V(s)$

As for why lower $\gamma$ reduces variance (p.2 above) - your explanation is on point: smaller $\gamma$ $\Rightarrow$ smaller weights for distant rewards $\Rightarrow$ less variance (but more bias).


Could style transfer be used to transfer the style of a website from one to another? Given that one website has a particular style and another has another style, could a style transfer be done such that the style of one website was transferred to the other website?

Or in a more simple case, consider just part of a website, a box.

The correct way for website styling to be encapsulated and centralized is through the use of one or more CSS style sheets.  For instance, the old  tag is frowned upon and using a text-align:center directive a proper class or ID based CSS selector is considered the proper way.  In such a case, the copying of a style is simply done by the copying of the appropriate style sheet file.

The place where AI and/or machine learning tools may be best used is in the conversion of non-CSS styling to CSS styling.  Once they are in that form, the copying and layering of styling becomes trivial.

To convert a legacy web site to proper CSS style sheet use, the input would be a set of all documents likely to be involved in the styling of a specific page (i.e. its style dependencies).  The output would be a set of corresponding documents with the styling moved into classes and IDs that remove styling redundancy (simplify and centralize the style).

The transformation of the document set could be best accomplished with current technique through transformations, much like code refactoring in typical IDE refactoring algorithms in that the output document set must be functionally the same as the input document set.  Furthermore, the appearance and responsive (size sensitive) aspects must be preserved in addition to functionality.

The greatest usability could be obtained by adding some automation to these refactoring transformations, as in Maxima simplification functions like trigsimp.  This is a typical search of options, perhaps with predictive branching, as in compiler optimizations.

The transformations, to guarantee preservation of function and appearance must essentially be elementary equivalent transformations, but the selection of transformation would probably need to be more advanced than basic antecedent and consequence productions.

Heuristically meta rules, fuzzy logic, neural nets, Bayesian inference, or other tools could be compared in terms of effectiveness in the realm of prioritizing transformations and in terms of scoring results in the search leaves.  Here's where machine learning would probably be necessary to have a truly useful tool.

Training could be accomplished through use of an array of input document sets and the careful scoring of transformations and/or outputs of trials could be performed with a software engineer with appropriate expertise in AI and/or machine learning, in conjunction with a CSS expert or a focus group containing two or more of them.

Once this process and the resulting rule set works, it would then be possible to simplify the CSS further by finding styling redundancy across pages and doing transformations that reduce an entire site to a minimal set of style sheets and style specifications within them.

Combining such tools with a simple crawler like wget could permit the copying of a desirable web site style as a starting point for another web site's styling, drastically cutting down the time to create a properly CSS styled result.


What is the difference between simulated annealing and deterministic annealing? Not sure if this is the right place, but I was wondering if someone could briefly explain to me the differences & similarities between simulated annealing and deterministic annealing?

I know that both methods are used for optimization and both originate from statistical physics with the intuition of reaching a minimum energy (cost) configuration by cooling (i.e. slowly reducing the temperature in the Boltzmann distribution to calculate probabilities for configurations).

Unfortunately, Wikipedia has no article about deterministic annealing and the one about simulated annealing does not mention any comparison.

This resource has a brief comparison section between the two methods, however, I do not understand why the search strategy of DA is

  based on the steepest descent algorithm

and how

  it searches the local minimum deterministically at each temperature.

Any clarification appreciated.

After diving deeper into the material I am able to answer my own question:
Simulated Annealing tries to optimize a energy (cost) function by stochastically searching for minima at different temparatures via a Markov Chain Monte Carlo method. The stochasticity comes from the fact that we always accept a new state $c'$ with lower energy ($\Delta E < 0$), but a new state with higher energy ($\Delta E > 0$) only with a certain probability
$$p(c \to c') = \text{min}\{1, \exp(-\frac{\Delta E}{T}) \},$$
$$\Delta E = E(c') - E(c).$$
Where we used the Gibbs distribution $p(c) = \frac{1}{Z}\text{exp}(\frac{-E(c)}{T})$ to calculate probabilities for each state, with $Z$ being the partition sum. The temperature $T$ plays the role of a scaling factor for the probability distribution. If $T \to \infty $ we have a uniform distribution and all states are equally possible. If $T \to 0$ we have a Dirac delta function around the global optimum. By starting with a high $T$, sampling states and gradually decreasing it, we can make sure to sample enough states from the state space and accepting energetic higher states in order to escape local minima on the way to the global optimum. After sampling long enough while slowly decreasing the temperature, we theoretically arrive at the global optimum.
Deterministic Annealing on the other hand directly minimizes the free energy $F(T)$ of the system deterministically at each temperature, e.g. by Expectation-Maximization (EM-algorithm). The intuition behind it is that we like to find an optimum at a high temperature (where it is easier to find one because there are fewer local minima), accept this as intermediate solution, lower the temperature, thus scaling the cost function such it is more peaked around it's optima (making optimization a bit more difficult) and start deterministically looking for an optimum again. This is repeated until the temperature is low enough and we (hopefully) found a global solution to our problem. Major drawback is that there is no guarantee to arrive at a global optimum in contrast to simulated annealing. The whole idea of scaling the energy function is based on the concept of homotopy: "Two continuous functions [...] can be "continuously deformed" into each other."


Developing character tactics via repeated trials Let's assume a common game scenario of several characters in a combat arena. Each character has different strengths and weaknesses. The arena has traps and tools. Suppose the characters had only very basic moves such as step in a direction, shoot, climb, duck, pick up item,  use item, drag heavy object. Each move has a chance of success  based on the context (e.g. range to target). What AI, machine learning, or evolutionary approach could be used to generate personalized tactics for each character based on repeated runs of the scenario?

There are a few ways to tackle this. You could make an AI that is simply a series of IF statements, or you could actually make an AI that would actually take in the situation and come up with a sensible solution.

IF Approach - You make a series of IF statements that come up with a sensible action to execute. This is the method that Minecraft uses. The resulting action were recorded from some of the best players.
True AI - Have your Character execute random actions and learn the consequences of them. The, train it to execute various actions for certain scenarios.

The main difference between these two approaches is that IF statements have a constant and predictable behavior, while the AI approach has a very bad startup value but ends up improving over time.

There is no "best" method, it is up to you to choose one or the other or a mix of both.


Understanding TD(0) algorithm implementation I came across the $TD(0)$  algorithm from Sutton and Barto:

Clearly, the only difference of TD methods with the MC methods is that TD method is not waiting till the end of the episode to update the $V(s)$ or $Q(s,a)$, but according to David Silver's lecture (Lecture 4- ~34:00),

The $TD(0)$ algorithm learns from incomplete episodes, but in the earlier algorithm we can see that the loop repeats until $s$ is terminal which mean completion of episode.

So does David Silver by learning from incomplete episodes, mean learning of $V(s)$ even when the episode is not completed? Or Did I interpret the algorithm wrong? If so what is the correct interpretation?

The $TD(0)$ algorithm learns from incomplete episodes, but in the earlier algorithm we can see that the loop repeats until $s$ is terminal which mean completion of episode.

In the pseudocode, you have two loops: one for each episode and one (nested) for each step of the episode. The until $S$ is terminal means that you perform the updates until you end the episode (that is, you end up in a terminal state, e.g. checkmate in the game of chess). For each step of the episode, you perform the TD(0) update.

Apparently, you're confusing two things: the fact that each episode ends in a terminal state and the fact that TD learns from incomplete information. Each episode ends in a terminal state (otherwise it would not be called an episode), but this does not mean that it collects a full rollout before updating $V$. In fact, at each step of the episode, it updates $V$.

The information in the David Slider's slide is consistent with the pseudocode. TD learns from experience because it uses the given policy $\pi$ to behave.

  learning from incomplete episodes, mean learning of $V(s)$ even when the episode is not completed

Yes, essentially you're updating the value function during each step of each episode.


Tensorboard problems When trying to run tensorboard locally to show my logs with tensorboard --logdir logs/ it always shows nothing but the regular tensorboard menu options, such as orange bar at the top, and different section buttons at the top like graphics, etc. however never shows any data regarding my agents. I am using tensorflow 0.11

The correct syntax is tensorboard --logdir=$PATH, with an equals sign between the switch and the path. Then I would suggest you make sure you are actually telling to the Saver to save whatever data you want to save.


What are the consequences of layer norm vs batch norm? I'll start with my understanding of the literal difference between these two. First let's say we have an input tensor to a layer, and that tensor has dimensionality $B \times D$, where $B$ is the size of the batch and $D$ is the dimensionality of the input corresponding to a single instance within the batch.

Batch norm does the normalization across the batch dimension $B$
Layer norm does the normalization across $D$

What are the differences in terms of the consequences of this choice?

This is how I understand it.
Batch normalization is used to remove internal covariate shift by normalizing the input for each hidden layer using the statistics across the entire mini-batch, which averages each individual sample, so the input for each layer is always in the same range. This can be seen from the BN equation:
$$
\textrm{BN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta
$$
where $\gamma$ and $\beta$ are affine parameters learned from data; $\mu(x)$ and $\sigma(x)$ are the mean and standard deviation, computed across batch size and spatial dimensions independently for each feature channel. First, we normalize each channel with 0 mean and standard deviation of 1 according to batch statistics. We then scale and shift each channel with $\gamma$ and $\beta$.
This is fine if you want to classify an average object on an image from different viewing angles and lighting conditions. It is defined similarly to BN:
$$
\textrm{LN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta
$$
but now $\mu(x)$ and $\sigma(x)$ are computed across all channels for each individual sample. Here's an illustration of the difference:

So layer normalization averages input across channels (for 2d input), which preserves the statistics of an individual sample. In some cases, we want to penalize the weights norm with respect to an individual sample rather than to the entire batch, as was done in WGAN-GP.
In terms of style transfer for images, it is also important to preserve the individual color statistics of a sample. Therefore, StyleGAN uses adaptive instance normalization, which is an extension of the original instance normalization, where each channel is normalized individually.
In addition, BN has several problems: the batch size must be large enough to capture overall statistics, which is sometimes impossible if you are working with large images since the model won't fit in memory. The concept of a batch is not always present, or it may change from time to time.
I strongly encourage you to read the original BN paper and also:
Adaptive instance normalization
Group Normalization


Reinforcement learning with hints or reference model In Reinforcement Learning, when I train a model, it comes up with its own set of solutions. For example, if I am training a robot to walk, it will come up with its own walking gait, such as this Deep Mind robot that has learned to walk in a bizarre gait. It can surely walk/run although the movements does not quite look like a human.

I was wondering how can I train a model by providing it some kind of reference motion data? For example, if I collect motion data from a walking human and then provide it to the training, can the training be made learn the walking movements that looks similar to the reference motion data?

Searching online I did find some links that shows this is possible. For example, here is a research where the researchers did exactly what I am trying to do, they fed motion data captured from humans to a simulation and made it learn the movements.

So, my question is: How can I give some kind of hints or reference data to a reinforcement learning model instead of just leaving it all by itself? how exactly is this done? What is it even called? What are some terms and keywords that I can search for to learn more?

Many thanks in advance

You can look into the techniques used in GANs (genereative adverserial networks). These networks work by having 2 learning agents. 1 to create images and 1 to learn the difference in a human made image and a computer generated image. This works because the 2 agents drive each other to be better and ultimately make the generator create images which can't be distinguished from real life images.

In your case you can make a agent trying to tell if the data is human or computer generated. The agent learning to move will then get negative rewards when the other agent can identify it as computer movement. This way the mover will learn to move like your reference data.

UPDATE:

I just found this video and paper which does exactly the same as you asked. Instead of using a GAN like structure they use a task specific reward and a imitation reward, which is based on reference motion data they have.

https://www.youtube.com/watch?v=vppFvq2quQ0


Using a neural network to identify a stable region within a set of data? I am working on a problem in which I am attempting to find a stable region in a spiral galaxy. The PI I'm working with asked me to use machine learning as a tool to solve the problem. I have created some visualizations of my data, as bellow.

In this image, you can see there is a flat region between 0 and roughly 30 pixels, and between 90 pixels and 110 pixels. I have received suggestions to use an RNN LSTM model that can identify flat regions, but I wanted to hear other suggestions of other neural network models as well.

The PI I'm working with suggests to feed my data visualization images into a neural network and have the neural network identify said stable regions. Can this be done using a neural network, and what resources would I have to look at? Moreover, can this problem be solved with RNN LSTM? I think the premise of this was to treat the radius as some temporal dimension. I've been extensively looking for answers online, and I cannot quite seem to find any similar examples.

If you're really just trying to find long contiguous flat regions in a sequence, you do not need machine learning. Your PI is mistaken. You would be better off simply writing a short data processing program. Your program could find the finite differences between adjacent datapoints, and then count whether a long string of them are below some threshold to identify long flat regions. This will be faster, simpler, and perhaps more accurate than using ML on data visualizations for this task.

If you are trying to find something more complex than these long flat regions, you could instead train an LSTM on the raw sequential data that you are using to generate the images. Again, that will probably be more accurate than trying to train a CNN, or any non-sequential model, on the image data itself.


What is the "Hello World" problem of Reinforcement Learning? As we all know, "Hello World" is usually the first program that any programmer learns/implements in any language/framework.
As Aurélien Géron mentioned in his book that MNIST is often called the Hello World of Machine Learning, is there any "Hello World" problem of Reinforcement Learning?
A few candidates that I could think of are multi armed bandits problem and Cart Pole Env.

MNIST (along with CIFAR) may be the "Hello World" of supervised learning for image classification, but it is definitely not the "Hello World" of all machine learning techniques, given that RL is also part of ML and MNIST is definitely not the "Hello World" of RL.
I don't think there is a single "Hello World" problem for RL. However, if you are looking for simple problems (or environments) that are usually used as baselines to assess the quality of RL agents, then I would say that the simple grid worlds where you need to move from one place to the other, the CartPole, MountainCar, Pendulum or other environments listed here are often used.
The environment that you choose to train and test your RL agent depends on your goals. For example, if you designed an algorithm that is supposed to deal with continuous action spaces, then an environment where you can take only a discrete number of actions may not be a good option.
The mentioned environments are very simple (i.e. toy problems). In my opinion, we need more serious environments that can show the applicability of RL to other areas other than (relatively simple) games.


Framework for Joining Multiple Modular Artificial Neural Networks I'm looking for an industry standard framework for joining multiple neural networks in a modular way.

Assume we have two or more neural networks trained to perform certain tasks. By feeding the outputs of some networks into the inputs of others we might obtain higher functionality, but to test multiple hypotheses, we would need a way to rapid prototype those configurations.

The source of this image is here — a proposition of this modular ANN architecture in a thesis

I'd be interested in knowing:

Do any frameworks or libraries like this even exist?
If so, do they support distributed models, so that models don't have to be hosted in the same process or on the same machine?
Do they allow hosting models to be generated from different deep learning frameworks?

In recent years the focus has been on layers rather than the more biologically inspired individual nodes.  As stated in the comment by thecomplexitytheorist you could use a computational graph, although then you have issues with distribution and you're limited to one framework.

I created something in my PhD about the same time as the thesis you reference that was a simulation of biological structure and connected in a way not dissimilar to figure b above using ML to determine the weights of all the connections.  The control module of the network was distributed across several neurons, but allowed asymmetric connections between the inputs and outputs.  This was a large effort, and similar results could now be achieved more quickly using the modern frameworks.  This is, I believe, the main reason that biologically inspired neural networks are not as actively researched.  I still believe that there are tricks in biology that we've not encapsulated and would love the head space to create a framework to test this hypothesis!

Multiple Neural Networks are the simplest way to go and I've created several of these over the years (sadly not open source).  These can be distributed and use different frameworks - just wrap up each of your networks in an API that has common inputs and outputs and you can daisy chain them without worrying about the internals.  This would give you the power and flexibility you need and allow you to swap in and out multiple models.  I know I've made that sound simple but at this stage it's more a software engineering problem.  You do need to be wary of compound errors between your networks if you're trickling down as it's basically a decision tree of NNs at that point.


What topologies support recognition of action sequences? The ability to recognize an object with particular identifying features from single or multiple camera shoots with the temporal dimension digitized as frames has been shown.  The proof is that the movie industry does face replacement to reduce liability costs for stars when stunts are needed.  It is now done in a substantial percentage of action movie releases.

This brings up the question of how valuable recognizing a stop sign is compared to the value of recognizing an action.  For instance, in the world of autonomous vehicles, should there even be stop signs.  Stop signs are designed for lack of intelligence or lack of attention, which is why any police officer will tell you that almost no one comes to a full stop per law.  What human brains intuitively looks for is the potential of collision.

Once what we linguistically perceive as verbs can be handled in deep learning scenarios as proficiently as nouns can be handled, the projection of risk becomes possible.

This may be very much the philosophy behind the proprietary technology that allows directors to say, "Replace the stunt person's face with the movie's protagonist's face," and have a body of experts execute it using software tools and LINUX clusters.  The star's face is projected into the model of the action realized in the digital record of the stunt person.

Projected action is exactly what our brain does when we avoid collisions, and not just with driving.  We do it socially, financially, when we design mechanical mechanisms, and in hundreds of other fields of human endeavor.

If we consider the topology of GANs as a loop in balance, which is what it is, we can then see the similarity of GANs to the chemical equilibria between suspensions and solutions.  This gives us a hint into the type of topologies that can project action and therefore detect risk from audiovisual data streams.

Once action recognition is mastered, it is a smaller step to use the trained model to project the next set of frames and then detect collision or other risks.  Such would most likely make possible a more reliable and safe automation of a number of AI products and services, breaking through a threshold in ML, and increased safety margins throughout the ever increasing world population density.

... which brings us back to ...

What topologies support recognition of action sequences?

The topology may have convolution, perhaps in conjunction with RNN techniques, encoders, equilibria such as the generative and discriminative models in GANs, and other design elements and concepts.  Perhaps a new element type or concept will need to be invented.  Will we have to first recognize actions in a frame sequence and then project the consequences of various options in frames that are not yet shot?

Where would the building blocks go and how would they be connected, initially dismissing concerns about computing power, network realization, and throughput for now?

Work may have been done along this area and realized in software, but I have not seen that degree of maturity yet in the literature, so most of it, if there is any, must be proprietary at this time.  It is useful to open the question to the AI community and level the playing field.

This is an old area of AI called "Plan Recognition", which has about 3.5 million results in Google Scholar.

A lot of the modern work is done with classical search techniques coupled with expert domain knowledge, or related reasoning concepts like Hierarchical Task Networks.

I'm not aware of or able to find recent research using deep neural networks for this problem, but I think there are some data-drive approaches to the related work in video-game player modeling.


Should we use a pre-trained model or a blank model for custom entity training of NER in spacy? Further to my last question, I am training a custom entity of FOODITEM to be recognized by Spacy's  Name Entity Recognition engine. I am following tutorials online, following is the advise given in most of the tutorials;

Load the model or create an empty model

We can create an empty model and train it with our annotated dataset or we can use the existing spacy model and re-train with our annotated data.

But none of the tutorials tell how/why to choose between the two options. Also, I don't understand how will the choice affect my final output or the training of the model.
How do I make the choice between a pre-trained model or a blank model? What are the factors to consider?

The reason you would load a pre-existing model is that it offers something of value to your task (e.g. named entity recognition for food) and the cost of training it from scratch is not worth it.  For example, to train GPT-3 from scratch would cost several million dollars.  Typically someone will use a model like BERT and fine tune it.  This is called transfer learning.  With spaCy you will typically use en_core_web_sm which was trained on the OntoNotes corpus and includes named entities.  Making a custom food NER using en_core_web_sm should be more accurate than making one from scratch. You should be able to build a good model with and without transfer learning fairly quickly if you have a GPU.


Why can a neural network use more than one activation function? From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and weights as their constants, like Fourier sums and other orthogonal basis functions.
How, then, can one neural network layer use activation function, like a sigmoid, and another one like the output using softmax? How do we know a linear combination of sigmoids and something else can still build that function no matter what? To me, it's like saying a function is approximated using sine functions with $N$ different $k$ values and then also randomly a few Hermite polynomials are thrown in as well. In this case, Hermite polynomials and the sine function aren't even orthogonal (to be honest I haven't checked but I'd assume they're not).
This question highlights some misconceptions I have about activation functions, perhaps, and I'd like to know where I'm going wrong here.

First of all, it looks like you are under impression that a neural network is structured like this (example for 4 inputs and outputs):
$$ \begin{array}{rcl}
y_1 & = & \text{sigmoid}(w_{11}x_1 + w_{12}x_2+w_{13}x_3+w_{14}x_4+b_1)\\
y_2 & = & \text{sigmoid}(w_{21}x_1 + w_{22}x_2+w_{23}x_3+w_{24}x_4+b_2)\\
y_3 & = & \text{softmax}(w_{31}x_1 + w_{32}x_2+w_{33}x_3+w_{34}x_4+b_3)\\
y_4 & = & \text{softmax}(w_{41}x_1 + w_{42}x_2+w_{43}x_3+w_{44}x_4+b_4)\\
\end{array}
$$
If that is how you understand it, then you've missed the point of "deep" neural networks. (Sequential) deep neural network are structured in several layers:
$$
 l_i^1 = \sigma^1(\sum_jw_{ij}^1x_j+b^1_i) \\
 l_i^2 = \sigma^2(\sum_jw_{ij}^2l^1_j+b^2_i) \\
 \ddots\\
 l_i^\alpha = \sigma^\alpha(\sum_jw_{ij}^\alpha l^{\alpha-1}_j+b^\alpha_i) \\
 y_i = \sigma^{out}(\sum_jw_{ij}^{out} l^\alpha_j+b^{out}_i) \\
$$
The activation function $\sigma^l$ of a single layer stays the same.

To me, it's like saying a function is approximated using sine functions with N different k values and then also randomly a few Hermite polynomials are thrown in as well.

To follow this analogy - you are mistaken by thinking that we are simultaneously trying to decompose over $\sin$ functions and hermitian polynomials. What actually happens is that we first decompose over $\sin$ functions and then decompose over hermitian polynomials. This actually can makes sense from practical point of view.

Hermite polynomials and the sine function aren't even orthogonal (to be honest I haven't checked but I'd assume they're not).

They are not, but sigmoids (or ReLUs) are not orthogonal to each other either. Orthogonality has noting to do with any of it.


Spontaneous emergence of replicators in Artificial Life One of the corner stones of The Selfish Gene (Dawkins) is the spontaneous emergence of replicators, i.e. molecules capable of replicating themselves.

Has this been modeled in silico in open-ended evolutionary / artificial life simulations?

Systems like Avida or Tierra explicitly specify the replication mechanisms; other genetic algorithm/genetic programming systems explicitly search for the replication mechanisms (e.g. to simplify the von Neumann universal constructor)

Links to simulations where replicators emerge from a primordial digital soup are welcome.

Systems Approach

Let's set out to replicate a real time system $S: \mathcal{X} \Rightarrow \mathcal{Y} \; | \; I$, where $\mathcal{X}$ is a empirical continuous history of input and $\mathcal{Y}$ empirical continuous history of output, conditioned upon a real initial system state $I$.  Based on some definition, we require $S$ to be alive.

We cannot simulate a replication of a theoretical model of life, with a selfish gene or any other such attribute, simply because no mathematically terse model on which the simulation could be based exists.  As of this writing, only hints to and minutia of such a model are known.

Furthermore, models are mathematical representations that, throughout human history, are found to be approximations of complexities once anomalies are addressed and new models develop to incorporate them into the theory.1

Simulation Roughly Defined

If we examine a general algorithm $\mathcal{A}$ to replicate $S$, replication can be roughly sketched as follows.

Estimate system $S$, essentially forming hypothesis $H$.
Simulate initial state $I$.
Initiate a series of discrete stimuli $\mathcal{X}_t$ approximating the real and continuous $\mathcal{X}$.
Acquire resulting system behavior $\mathcal{Y}_t$ as discrete observations of $\mathcal{Y}$.
Verify the difference between simulated and actual systems to be within allowable error $\epsilon$.

Defining Spontaneous Emergence

By spontaneous emergence is meant that such an astronomically large array of initial states and sequences of stimuli occurred that there is a high probability of one of the permutations being alive, based on some specific and reasonable definition of what is living.

Defining What Life Is

Reviewing several definitions of living organisms, the most reasonable definitions include these:

The organism can be distinguished from its environment.
The organism can acquire and cache potential energy and materials required to operate.
Its operation includes continued acquisition, producing a bidirectional and sustainable relationship with its environment.
The organism can roughly reproduce itself.
The reproduction is similar to but not exactly like the parent(s).
The method of energy and materials acquisition may include the consumption of other organisms or its energy and materials.

Competing for resources, natural selection, and all the other features of evolutionary theory are corollary to the above five requirements.  In addition to these, the current trend toward recognizing symbiogenesis as a common theme in the emergence of species should not be dismissed.

Replication of one organism may be influenced by the composition of another organism through forms of assimilation or symbiosis such that traits are passed across categories of organisms.

Artificial Life as a Simulation

These seven criteria poses a challenge for humans attempting to artificially generate life.  It is easy to create a computer model such that life is simulated in some way.  Consider how.

The environment contains virtual energy and virtual matter.
The model of the organism, distinguished from its environment, can acquire its operational requirements from the environment through a set of operations on it.
Mater and energy are conserved because the temperatures are far below nuclear thresholds.
The model of the organism allows acquisition only if enough of the energy and materials acquisition has occurred to maintain the cache.
Mater and energy acquired by one organism cannot be acquired by another organism except by consumption or absorption of an organism that acquired it or produced it from that which was acquired.
The model of the organism can self-replicate in such a way that stochastic differences in the replication is introduced in small quantities.
Operational information, including replication information, may be acquired through consumption or symbiotic relationship under some conditions.

Magical Genes for Spontaneous Life

Notice that the selfish gene is not mentioned above.  Selfishness, the prerequisite of which is intention, is not a requirement for life.  An amoeba does not think selfishly when it moves or eats.  It operates witlessly.  We should not anthropomorphize every organism we study, or develop theory based on anthropomorphic conceptions.

Similarly, symbiotic relationships form that are neither loving nor altruistic.  They exist because there is a mutual benefit that appeared as an unintended byproduct of normal operations and both symbiotic parents happened to pass that symbiotic connection to their respective offspring.
The mutual benefit, the symbiosis, and the replication are witless and unintended.

There need not be a control mechanism distinct from all other replicated mechanisms to control either symbiotic collaboration or competition.  They too are natural consequences of living things sharing an environment.  Whether an organism dies because it

Lost its symbiont,
Starves because other organisms consumed its necessities,
The organism itself depleted its own resources, or
Those needed resources were otherwise rendered unavailable,

it is still unable to replicate, so its traits die with it.

Note also that there is no known molecule that can replicate itself.  Complex systems of molecules in a variety of chemical states and equilibria are required for reproduction to take place.

Returning to Simulating an Already Existing Organism

Running a time sharing system or distributing these simulated organisms in a parallel processing arrangement may some day simulate a biosphere, but it is not one in that only transistor electro-chemistry is involved.  There is no actual direct relationship between the energy and mater of the system used to assemble the simulation and the energy and mater of the simulated environment in which the simulated systems $S$ reside.

Certainly genetic algorithms, such as Avida and Tierra, have been developed.  Compare those simulations to the modelling scenario described above, and their deficiencies become clear.  Human researchers have not yet found $\mathcal{A}$ to replicate $S$ in a way that aligns with biological reality.

Open-endedness Requires Verification to Have Merit

The most significant limitation on implementations in silico, is that they can never be truly open-ended.

There is no way as of this writing to replicate that which was simulated outside the simulation system.  Until nanotechnology reaches a point where 3D construction and assembly can migrate alive simulations into the unsimulated universe, these simulations are closed-ended in that way and their viability in vito is untested.  The value of open-ended simulations without any way to validate them is essentially zero except for amusement.

Even in the space of digital simulation, as far as that technology has progressed, nothing even close to von Neumann's universal constructor has been accomplished.  Although generic functional copy constructors are available in Scheme, LISP, C++, Java, and later languages, such is a minuscule step toward living objects in computers.

Digital Soup

The simulation of life's origins is considerably more difficult than finding an algorithm $\mathcal{A}$ to replicate $S$, where $S$ is a single life form and a sufficient portion of its environment to be representative of the biosphere on earth with an organism in it.

The issue with primordial digital soup is one of the combinatory explosion.  There are 510 million square Km on the earth's surface, and there are only three categories of life origin time frames possible.

The current estimates are close to correct, that the earth formed 4.54 billion years ago and extremely primitive life emerged 3.5 billion years ago
The organic material found in Canada that is allegedly 3.95 billion years old shortens the gap between planetary formation and life formation on it and older terrestrial life may be found
Vladimir Vernadsky's comment that life may have preexisted earth is more than just a possibility

If we go with the 1.04 billion year gap, then $(4.54 - 3.5) \cdot 10^9 \cdot 510 \cdot 10^6$ Km-years of soup must be simulated, since we cannot assume that life started in the ocean or a puddle or even on the surface.  It could have started underground or in the atmosphere.  The biosphere is currently thought to be 1,800 m above to 8,372 m below thick.

With nanobes being 20 nm in diameter and the possibility that the emergence may have only taken one second we have to simulate in three dimensions over time the following space-time domain in finite elements with at least 50% overlap in all three dimensions.

$$\dfrac {2^3 \cdot (4.54 - 3.5) \cdot 10^9 \cdot 510 \cdot 10^6 \cdot (1,800 - 8,372) \cdot 365.25 \cdot 24 \cdot 60 \cdot 60} {(20 \cdot 10^{-9})^3} \\ = 170,260,472,379 \cdot 10^{9+6+27} = 1.7 \cdot 10^{56}$$

With a quantum computer two stories high the size of Switzerland, the computing time would vastly exceed the duration of the average species on earth.  Humans are likely to be extinct before the computation completes.

As the dating of the oldest found fossils converges on the dating of earth, it may seem that life emerged quickly on earth, but that is not a logical conclusion.  If life formed as soon as the earth cooled sufficiently and no evidence of continuous emergence is found in the remaining billions of years, then Vernadsky's inference that life arrived on earth through one or more of the bodies that struck it becomes more probable.

If that is the case, then one must ask the question, if all assumptions are dropped, whether life had a beginning at all.

Simulating Life Versus Simulating Its Formation

We may simulate what life is, that is, find an algorithm $\mathcal{A}$ to replicate $S$, where $S$ is a single live organism.  It is not realistic to, by brute force, simulate how life began without learning more about what conditions can lead to its formation theoretically to drastically reduce the soup simulation space.  It is that learning that is an ongoing area of research in the genetic algorithm field.

Early musings about the possibility of an algorithm $\mathcal{B}$, which can provide the conditions that allow an arbitrary organism $S$ conforming to the above definition of life to form with out a parent or parents were interesting. Given algorithm $\mathcal{A}$ that simulates a life form and algorithm $\mathcal{B}$ that simulate the formation of life, it may be the later algorithm that proves significantly more difficult.

Conforming physics outside a computer to the simulation may be impossible.  Whether simulated life, when embodied in a robotic system is actually going to be considered life will be left to our descendants, should the species endure sufficiently.

Footnotes

[1] Classic cases include the heliocentric Copernican system giving way to the Law of Gravity, that law being shown an approximation of general relativity as shown by the proper prediction of the orbit of Mercury and light's curvature near the sun, the Four Elements dismissed in light of Lavoisier's discovery of oxygen, and absolute provability of truth within a closed symbolic system disproved by Gödel in his second incompleteness theorem and then recouped partially (in terms of computability) by Turing's completeness theorem.


How do I apply reinforcement learning to a game with infinitely many actions? I am trying to figure out how to use a reinforcement learning algorithm, if possible, as a "black box" to play a game. In this game, a player has to avoid flying birds. If he wants to move, he has to move the mouse on the display, which controls the player position by applying a force. The player can choose any position on the display for the mouse. A human can play this game. To get a sense what needs to be done, have a look at this Youtube video.

I thought about using an ANN, which takes as input the information of the game (e.g. positions, speeds, radius, etc.) and outputs a move. However, I am unlikely to ever record enough training data to train the network properly.

Therefore, I was thinking that a RL algorithm, like Q-learning, would be more suited for this task. However, I have no clue how to apply Q-learning to this task. For example, how could the coder possibly know what future reward another move will bring?

I have a few questions:

In this case, the player has infinitely many actions. How would I apply RL to this case?
Is RL a good approach to solving this task?
Is there a handy Java library which would allow me to use a RL algorithm (as a block-box) to solve this problem?
Are there alternatives?

This looks like the kind of game that a Reinforcement Learning algorithm could learn to play. If you are learning from the game objects directly (as opposed to wanting to have the agent learn from viewing the screen), then it should be relatively straightforward to do so - by which I mean that it will involve some new concepts, a little maths, and maybe take you a few weeks to learn enough and run enough experiments on the AI to the point where you have something good. But at least you should not need a cluster of GPU servers to run your learner.

  However, I have no clue how to apply Q-learning to this task. For example, how could the coder possibly know what future reward another move will bring?

That is a major part of RL - the algorithms that are part of RL all solve this issue. Therefore the coder does not need to know how to calculate future reward, just an immediate reward. For the example game, it looks like you can grant a reward of $+1$ for every frame where the player does not crash. An RL algorithm that can solve this environment should figure out through "trial and error" which actions will keep the game running.

Q Learning achieves this by keeping track of estimates of future value and updating them across time steps - if action $a_1$ in state $s_1$ consistently leads to a later state $s_2$ with a high value, then Q learning will update its estimate of the value of $Q(s_1,a_1)$ to be high as well. The estimates start very poorly, but updates work their way back through time steps as the agent learns, and provided things don't go wrong, eventually it will learn that one action will lead to higher value than another, and it be able to consistently choose it.

This is such a basic question that if you want to use RL, even if you find a library that implements the whole thing for you, that you clearly need to study some foundations of RL. There are a few different places you can do that. I can suggest two that I have used:

The book by Sutton & Barto. Reinforcement Learning: An Introduction
UCL lectures by David Silver. These include videos on YouTube.

Both are online, free and comprehensive introductions to the subject. You can find shorter, more direct introductions online if you search for e.g. "Q Learning Tutorial".

Be prepared to spend hours just learning enough about the topic that you understand how to structure your data and interface with the game in order to create a learning environment for your AI.

  In this case, the player has infinitely many actions. How would I apply RL to this case?

This is a good question. Some algorithms are better than others. Q Learning cannot work with infinitely many actions. However, you could choose a discrete set of mouse points - e.g. maybe 64 - at different angles and distances from the player (red dot) and try and get the values of those. That would allow you to try with Q learning, and is worth considering because RL that properly solves large action spaces is harder to understand and work with. You could look into the DQN algorithm for this. In short, the DQN algorithm uses history of actions and their immediate results to train a neural network to estimate the values of each action.

If you really want to solve large action spaces properly, then you need to be looking into Policy Gradient methods. Two popular and effective PG algorithms are A3C and DDPG. These are tougher to learn and implement. They use neural networks to learn multiple functions, one of which is the policy - this directly selects which action to take, given a current state/observation.

  Is RL a good approach to solving this task?

It is definitely possible to use RL to do what you want.

  Is there a handy Java library which would allow me to use a RL algorithm (as a block-box) to solve this problem?

I could not find one on a brief search. To get started, you might want to look at java-reinforcement-learning which implements some of the algorithms from Sutton & Barto, so would be a good starting point. Most of the other Java packages I found covered basics and learning material, but would not be plug-and-play for your game.

If you do find a library, you will need to have learned enough basic RL to be able to follow how to use it.

  Are there alternatives?

One alternative that might work well for your game environment, and be a little easier to understand and implement would be to combine a policy function (like a neural network) with an evolutionary algorithm that tries out variations of the policy. The go-to algorithm here is probably NEAT, and it is capable of solving simple game agents effectively. Your game should be a reasonable fit for NEAT because the scoring system of survival time makes for a good fitness function.

One thing that could really be worth trying before diving into the (very hard to understand) A3C and DDPG algorithms is the Cross-Entropy Method. It is a really simple RL agent that has some similarities to genetic algorithms (it selects "best" items, but instead of from the population, it is from random behaviour of the same individual), but doesn't always scale too well. However, it might work well for your game.


Can artificial intelligence classify textual records? I am a records manager and I am being asked if I recommend Office 365.  I'm having a hard time making a recommendation because I am missing an essential piece of information: can Office 365 replace the manual process of placing records into categories based on organizational function?  It is important that this is done accurately, because the category determines how long the records should be kept before they are irretrievably destroyed.  James Lappin seems to say that yes, Office 365 is underwritten by Project Cortex, and it is capable of doing this.

My sense is that artificial intelligence is not yet capable of determining a conceptual category for records.  For this to be true, a machine would have to replicate a complex human process: reading a free-text, free-form document; identifying the relevant pieces of information in a document, while ignoring others to determine what the document is "about"; then taking the answer of what the document is "about" and matching it to a predefined set of major organizational activity.

Are there any AI experts who can comment on how realistic it is to expect Project Cortex to do this?

AI can categorize documents very accurately.  It is not a new application but in the last year the accuracy of the underlying algorithms such as text classifiers, and language models in general, has significantly improved.  There are applications of language models which now surpass human performance. Microsoft is one of the leaders in this area.  For example, see their Turing Natural Language Generation (T-NLG) language model: Turing-NLG: A 17-billion-parameter language model by Microsoft. The T-NLG was developed in a research group called Project Turing which works on AI tools for Microsoft's products including Office-365.  My guess is this same research group makes the AI for Project Cortex.


Are the ideas in the paper "Governance by Glass-Box: Implementing Transparent Moral Bounds for AI Behaviour" novel? In the paper Governance by Glass-Box: Implementing Transparent Moral Bounds for AI Behaviour, the authors seem to be presenting a black box method of testing. Are these ideas really new? Weren't these ideas already proposed in Translating Values into Design Requirements (by Ibo Van de Poel)? Black-box testing had already been proposed much earlier .

Poel's paper on Translating Values into Design Requirements articulates a framework for mapping abstract values and norms into concrete design constraints that an engineer could work with. The example used in the paper is mapping beliefs about animal welfare to design constraints on chicken coups.

The newer paper by Tubella et al. on Governance by Glass-Box builds on Poel's idea (and in fact, cite's Poel several times). It basically suggests that we should use Poel's design process, but that in something like an AI system, we also need to use an "observation phase" to validate the system, because unlike the problem of engineering a chicken coup, an AI system may appear to have met design constraints, but routinely violate them in production.

So, you're right that the Tubella et al. paper is essentially proposing the combination of Poel's framework for translating values into design constraints with the old idea of black-box testing, but this combination itself appears to be a new, if modest, contribution.


Difficulty in understanding identifiability in the "Dueling Network Architectures for Deep Reinforcement Learning" paper I have difficulty understanding the following paragraph in the below excerpts from page 4 to page 5 from the paper Dueling Network Architectures for Deep Reinforcement Learning.

The author said "we can force the advantage function estimator to have zero advantage at the chosen action."

For the equation $(8)$ below, is it correct that $A - maxA$ is at most zero?

  ... lack of identifiability is mirrored by poor practical performance when this equation is used directly.

  To address this issue of identifiability, we can force the advantage
  function estimator to have zero advantage at the chosen action. That is, we let the last module of the network implement the forward mapping

  $$Q(s, a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \left( A(s, a; \theta, \alpha) - \max_{a' \in | \mathcal{A} |} A(s, a'; \theta, \alpha) \right). \tag{8}$$

  Now, for $a^∗ = \text{arg max}_{a' \in \mathcal{A}} Q(s, a'; \theta, \alpha, \beta) = \text{arg max}_{a' \in \mathcal{A}} A(s, a'; \theta, \alpha)$, we obtain $Q(s, a^∗; \theta, \alpha, \beta) = V (s; \theta, \beta)$.  Hence, the stream $V(s; \theta, \beta)$ provides an estimate of the value function, while the other stream produces an estimate of the advantage function.

I would like to request further explanation on Equation 9, when the author wrote what is bracketed between the red parentheses below.

  An alternative module replaces the max operator with an average:

  $$Q(s, a; \theta, \alpha, \beta) = V (s; \theta, \beta) + \left( A(s, a; \theta, \alpha) − \frac {1} {|A|} \sum_{a' \in \mathcal{A}} A(s, a'; \theta, \alpha) \right). \tag{9}$$

  On the one hand this loses the original semantics of $V$ and $A$ because they are now off-target by a constant,
   but on the other hand it increases the stability of the optimization: with (9) the advantages only need to change as fast as the mean, instead of having to compensate any change to the optimal action’s advantage in (8).

In the paper, to address identifiability issue, there are two equation used. My understanding is both equations are trying to fix Advantage part - the last module. For equation $(8)$, are we trying to make $V(s) = Q*(s)$, as the last module is zero. And for equation $(9)$, the resulting $V(s)$ = true $V(s)$ + mean$(A)$? - as the author said ‘On the one hand this loses the original semantics of $V$ and $A$ because they are now off-target by a constant’. And the constant refers to mean$(A)$? Is my understanding correct?

I believe that is explained on the prior page:

"Intuitively, the value function $V$ measures the how good it is to be in a particular state $s$. The $Q$ function, however, measures the the value of choosing a particular action when in this state. The advantage function subtracts the value of the state from the $Q$
function to obtain a relative measure of the importance of
each action."

Then two paragraphs above were you started your quote:

"However, we need to keep in mind that $Q(s, a; \theta, \alpha, \beta)$ is only a parameterized estimate of the true $Q$-function. Moreover, it would be wrong to conclude that $V (s; \theta, \beta)$ is a good estimator of the state-value function, or likewise that $A(s, a; \theta, \alpha)$ provides a reasonable estimate of the advantage function.
Equation (7) is unidentifiable in the sense that given $Q$ we cannot recover $V$ and $A$ uniquely. To see this, add a constant to $V (s; \theta, \beta)$ and subtract the same constant from $A(s, a; \theta, \alpha)$. This constant cancels out resulting in the same $Q$ value. This lack of identifiability is mirrored by poor practical performance when this equation is used directly."

Another way of looking at it would be:

You receive answers to your question

Answers receive votes

Answerers have reputation

In a perfect world people could vote based on reputation, with a weighing based upon the correctness of the answer.
You could simply look at which answer received the most votes and choose it as correct.
In the real world things don't work that way, things are correct or incorrect whether they are measured or not (think quantum mechanics) and measurement doesn't always reveal the true answer.
See: Parameter Estimation.
The estimate of the advantage is only so good, sometimes it's useful to consider it and in other instances it's useful to reject it - intelligently doing both maximizes it's usefulness.


Neural network for pattern recognition in audio I'm trying to make a neural network that detects certain instruments in a song. I don't know for sure if I should use an RNN, CNN OR DNN. Which one is best for this situation?

I don't know for sure if I should use an RNN, CNN OR DNN. Which one is best for this situation?

This question or variations of of crop up a lot in DataScience stack exchange too. To paraphrase:

  I am trying to do something with X data, and I have a lot of choices for the model. Which is best?

Unfortunately, the answer is generally:

It depends on all the fine details of your project
Unless someone has done almost your exact project recently (so they were using latest techniques of the model type), then no-one knows a priori which model will get the best result.
Optimising machine learning is very much an empirical subject. If you want to know whether A is better than B, you have to try both and measure their performance.

In your case, I think that CNNs and RNNs are both applicable (and you might want to look at a WaveNet-like architecture, which is a variant of CNN, but that could be a bit too advanced to start with). You might have a slight preference for RNN as a starting point if the sequence length to process varies significantly, such that padding input to a CNN would be inefficient. You may also prefer an RNN if the output of your model needs to be a sequence, and doubly so if the output sequence varies in length and is not directly related to the input sequence length (think of natural language translation).


Does a trajectory in reinforcement learning contain the last action? From what I learn from CS285 and OpenAI's spinning up, a trajectory in RL is a sequence of state-action pairs:
$$\tau = \{s_0, a_0, ..., s_t, a_t\}$$
And the resulting trajectory probability is:
$$ P(\tau \mid \pi)=\rho_{0}\left(s_{0}\right) \prod_{t=0}^{T-1} P\left(s_{t+1} \mid s_{t}, a_{t}\right) \pi\left(a_{t} \mid s_{t}\right) $$

From CS285: http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-4.pdf

From spinning up: https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#trajectories

However, from my derivation, the above trajectory probability actually corresponds to the following sequence where the last action $a_t$ is absent:
$$ \tau = \{s_0, a_0, ..., s_t\} $$
Can someone please help me clarify this confusion?

If we use $T$ as the notation for the terminal state, then the last action is $a_{T-1}$. This is because when you reach state $s_T$ you don't take another action, which would be $a_T$, because the episode is finished upon reaching the terminal state.


Is there any example of using Q-learning with big data? Could we even use reinforcement learning with big datasets?

Or in RL does the agent built its own dataset ?

Or in RL does the agent built its own dataset?

Essentially this is the case.

RL is a very general learning mechanism, based on trial-and-error. You could create an environment where the agent's goal is to correctly predict classification or regression problems, where the agent is rewarded for getting close to the correct prediction. However, these problems are most often better addressed using supervised learning techniques. Using RL in such cases will most of the time make the learning slower, less efficient and less accurate.

The relationship between RL and supervised learning is more about how RL generates the target data for learning. In some cases, such as Deep Q Learning (DQN), you can see quite clearly from looking at the algorithm or code, that the RL agent contains a supervised learning component. The "supervised learning" in DQN learns a regression problem to predict the action values for the current agent's target policy. Whilst the outer RL logic that contains this supervised learning model, is built around testing and changing that behaviour in order to act optimally in the longer term.

Using RL to predict from a big data set would involve using the same kind of supervised learning model on the inside, with RL using that to guess at the correct result, and sometimes choosing to guess at some other random prediction instead. This guessing process is extra processing, plus it will add noise and variance to the error signal used to train the model. In a few cases, this might even be helpful, but in the majority of cases studied for regression and classification tasks, it will result in a poorer training process, less accurate model, or both.


Why is the sample size of stochastic gradient descent a power of 2? I watched in the video lecture of cs224: Stanford CS224N: NLP with Deep Learning | Winter 2019 | Lecture 2 – Word Vectors and Word Senses.

They take the sample size of the window to be $2^5 = 32$ or $2^6 = 64$. Why is the sample size of stochastic gradient descent a power of 2? Why not we can take 42 or 53 as the sample window size?

Btw, how to identify best minimum window sample size?

You can take any size you want since most memories are now larger than that, if you were to map the indices to a register, 64 values will take 6 bits whereas we have registers upto 64 bits, so it wouldn't matter much even if you choose a  different sample size.

As far as resolution is concerned, layers like maxpool which reduce the size by half (2x2 kernel is generally used), the resolutions of outputs of layers are multiplied/divided by powers of 2, so keeping that in mind, resolutions as power of 2 makes sense, but then again, we have padding and effects of convolution on borders.


Combining deep reinforcement learning with alpha-beta pruning I will explain my question in relation to chess, but it should be relevant for other games as well:

In short terms: Is it possible to combine the techniques used by AlphaZero with those used by, say, Stockfish? And if so, has it been attempted?

I have only a brief knowledge about how AlphaZero works, but from what I've understood, it basically takes the board state as input to a neural net, possibly combined with monte carlo methods, and outputs a board evaluation or prefered move. To me, this really resembles the heuristic function used by traditional chess engines like stockfish.

So, from this I will conclude (correct me if I'm wrong) that AlphaZero evaluates the current position, but uses a very powerful heuristic. Stockfish on the other hand searches through lots of positions from the current one first, and then uses a less powerful heuristic when a certain depth is reached.

Is it therefore possible to combine these approaches by first using alpha-beta pruning, and then using AlphaZero as some kind of heuristic when the max depth is reached? To me it seems like this would be better than just evaluating the current position like (I think) AlphaZero does. Will it take too much time to evaluate? Or is it something I have misunderstood? If it's possible, has anyone attempted it?

Yes it's possible to to combine AlphaZero with Minimax methods (including alpha-beta pruning). AlphaZero itself is combination of  Monte Carlo Tree Search (MCTS) and Deep Network, where MCTS is used to get data to train network and network used for tree leafs evaluation (instead of rollout as in classical MCTS). It's possible to combine selection-expansion part of AlphaZero MCTS with Minimax the same way as it was done for classical MCTS - "Monte-Carlo Tree Search and Minimax Hybrids", pdf.


Can a convolutional network predict states for a RL Agent During the course of training a DQN agent, all visited states are stored in a replay buffer. Therefore would it be practically possible for a CNN, given a reasonable amount of data, to predict the next RL state (in the form of an image) for a given action?
For example, take the game of Atari as shown below -

Here the agent can take 2 major actions - go left and go right. Would a CNN be generate the image of the bar going right/left for the respective actions?
My practical knowledge of CNNs is quite limited and therefore I'm trying to gauge the abilities of CNNs before I take up a project.

It sounds like what you're suggesting is similar to what is done in methods that use a planner. These methods looks to learn the dynamics of the MDP to use to plan during training; that is they want to be able to learn the transition probabilities $p(s'| s, a)$.
In this paper that I read recently they note that learning to predict environment dynamics when the state/action space is high dimensional, as is the case with images, is difficult; so whilst it may be possible in theory it would be difficult to do and if you were predicting many steps into the future then the error would compound.
A way around this, as is done in the referenced paper, is to use predict environment dynamics in a latent space. This means that they use a latent variable to predict the next state using e.g. Variational Autoencoders.


Neural network to get input attributes using only the output value I have an idea about how to use neural networks but I'm not sure if it is possible or not.

In supervised learning we have a set of attributes labeled with an output value. I can use these set to train my network.

Now I have a network trained to get an output value from an random set of attributes but, can I use this trained network to get the input attributes using only the desired output?

I will have N input values and only 1 output value. I've thought that I can use the weights for that network into a new one with 1 input value and N output values but I'm not sure if I can do that.

Now I have a network trained to get an output value from an random set of attributes but, can I use this trained network to get the input attributes using only the desired output?

It depends:

If you are happy to find any inputs, even non-realistic ones, that get your desired output, then you can use your trained network, with a minor modification. Freeze all the weights, and allow back-propagation to determine the gradient of the input (which should now be a variable to optimise, not source data). Start with a noise input, back-propagate the error to find gradient to make the input better at creating your desired output, then take a gradient step towards it in the input data. This is essentially how Deep Dream works. Like Deep Dream, you will not necessarily get realistic input values, but will get semi-random ones that cause your network to predict a specific class.
If you want the newly generated input to be a best guess at something from the original dataset, then you have to look at one of more advanced models:

Restricted Boltzmann Machines (RBMs).
Variational Autoencoders (VAEs)
Generative Adversarial Networks (GANs). These have become popular recently, and there are many variations on the basic idea, including VAEGANs that combine VAEs with GANs.

These network types are quite advanced, and can be tricky to understand and train successfully. You will want to spend some time researching each type.

To generalise terribly: A GAN will tend to generate realistic "noise" in the generated items, but at the expense of overall structure and cohesion (images tend to look distorted but with realistic textures). A VAE will tend to produce smooth, coherent inputs, but at the expense of lack of fine detail (VAE images tend to look smoothed and/or blurred).

If not sure what to try, probably GAN is a reasonable choice, since there are lots of tutorials available, and recent advances with image generation can look very impressive.


Is music/sound similarity comparison feasible on neural networks? I wonder on the following concept:

  A given neural network gets two audio input (preferably music) and gives a real number between 0 and 1 which describes "similarity" between the second and the first track.

As far as my understanding of neural networks go, the problem fits the concept of NNs, as pattern recognition in music can help determine similarities and discrepancies in audio, see voice recognition.

However, due to the nature of long and complex inputs, and the vague nature of learning datasets (how similar, for instance, Diana Ross "It's your move", and The Vaporwave legend "Floral Shoppe" exactly are? 0.9? 0.6? other?), such a network would be extremely slow and convoluted.

Is it possible today to build and train such a model? If yes, how would it look like?

Yes, it is possible, even if the best approach could be different from  neural networks. Anyway, you should extract some significant features from the audio (energy, onsets, root frequencies, and other). Usually, more features than those really needed are extracted and afterwards the most sigificant are selected through some algorithm (e.g. PCA). In this way you will obtain an array of features (say between 10 and 100 features) with which you can train your NNs.

Note that NNs do not tell you why two audio are similar but only if they are or not. This is a big disadvantage. Instead, algorithms based on grey-box modeling such as rule or case based algorithm (maybe using fuzzy logic) could be more useful, provided that you have a deeper knowledge of the problem.

References and deepening sources: SMC Lab from University of Padua education material


When is content-based more appropriate than collaborative filtering? I know the difference between content-based and collaborative filtering approach in recommender systems. I also know some of the articles said collaborative filtering have some advantages than content-based, some of them also suggest to use both method (hybrid) to make a better system recommendation.

Is there a specific case where the use of one method (content-based, specifically) is better than another? Because if there is no case at all, why both methods are considered to be on the same "level", why not focus on just one method? For example, focus on collaborative filtering or hybrid method (as an extension for collaborative filtering).

Some of the cases content-based filtering is useful is:

Cold-start problem: it happens when no previous information about user history is available to build collaborative filtering, so in this case, we offer to the user some items then recommend based on the similarity between these items and other items in the dataset alternate of recommending any items that maybe not with the user taste.
Transparency: collaborative method gives you the recommendation because some unknown users have the same taste as you that cause a problem if your data is biased towards one taste makes new user haven't enough similar users have the same taste, but the content-based method can tell you they recommend the items based on what features, that helps you to determine which factors affect in the recommendation.


Using a MLP to predict a 12x12 matrix So, i need to use an MLP to predict a 12x12 matrix composed of floating points.
The matrices are as this one that follows:

Most matrices have this "pattern".

As input, i have 7 floating points, such as these:

"2.0", "0.23", "239.10","3.5", "12,0", "10.6", "0.62".

To simplify the output matrix, i've converted it in a array with the 144 elements of the matrix.
So far i'm using MLPRegressor from Scikit.The problem is, there's absolutely no pattern in the predicted results.The predicted results include negative numbers, numbers really big and no "pattern" for the indexes whatsoever.
Is there a way to adjust these things in the model or the problem is on my dataset?
Thank you very much!

Update:

I wasn't very clear about the problem, so i'll try to explain it better.

About the problem: Predict water distribution in irrigation systems. I already have a program to simulate this, and works with rather good precision. The challenge now was to make an AI model to simulate this, instead of the old methods.

About the inputs: Inputs are one float representing the pressure of the sprinkler used to distribute water, two floats representing the speed and angle of the wind at the time data was collected. The 4 remaining are numbers representing some configuration of the sprinker (these numbers change very little between instances, but they do have impact in the result, which is why i decided to mantain them in the dataset).

About the output: Output is an array of 144 elements, representing the 12*12 matrix. Each element of the matrix contain a number representing the amount of water that was collected in that point. These collectors where evenly spread around the sprinkler. So the position of the matrix matter a lot, since in most cases the first and last lines and columns will have 0 or a close to 0 (but positive) number - this may vary depending especially on the speed and angle of the wind, but also on the sprinkler.

About the dataset: I have available 75 instances. They are all stored in a CSV file, where the 7 inputs and the 144 outputs are, each in one line.

The short answer to your question is: you probably do not fully know your data. remember that ML is no magic wand. It needs your understanding of the data and the behavior of it. Although it is approved that neural networks with at most two hidden layers can approximate any model with an acceptable degree precision, setting up the structure of the neural network is still your task; also note that the cleaner your data is, the more acceptable the degree of precision will be for a structured NN.

the following more detailed explanation might also help. understanding your data that I pointed above, means:

the design of your machine learning pipeline is the first and most important thing to take care of. Are you sure you have handled training set and test set appropriately? are the ratios standard? have you performed random sampling well?
the size of the dataset is also a matter of consideration. remember a ratio of n/p>10 (n: the number of instances, p: the number of features) is necessary. So if you have 7 features, you should have more than around 100 instances. check how big is your dataset
the distribution of your dataset is also important. dealing with badly distributed data should be considered if you have an issue here. use plots and histograms to understand the distribution of your dataset over feature axes. There is a high risk of overfitting if your data is distributed badly
also take care of your output distribution. with a bad distribution over your output data (focused outputs in a zone) you loose the power of generalization of your model
check the relations between different features. check if there are any pairs of dependent variables. you can use scatter plot or calculating pairwise covariances. PCA or more complicated methods (like auto-encoders) will be of help if there is any issue here
one possibility is maybe your data doesn't carry any latent information at all. maybe the underlying pattern is so stochastic and nonlinear that it seems to be random. in other words: you can not predict what kind of food your grandma has cooked by looking at results of world cup games, no matter how big your dataset is.

Now about your output matrix:

reshaping your matrix to an array of 144 might be the best or the worst thing to do. it depends on the characteristics of your data. maybe the position of each element in the matrix is also important in predicting the value. so you might want to include the i,j of each element in the input array too. maybe each output also takes effect from neighbors. there are some other options to consider:

you can use 144 different MLP networks, one for each element (a little bit like what you have already done, when reshaping your matrix. since your library constructs a single MLP network that has 144 neurons in the output layer. so it seems like 144 networks that only vary in the output layer, and note that the weight correction step is different too, since the training error for all of the outputs propagate back in the layers simultaneously)
you can create 144 instances out of each instance, each of which has 7+2 inputs and 1 output. the 2 more inputs are the position of the output value (i,j). it might be a poor decision to do that but this method has helped me a couple of times, so test it
Do the second step, but include the neighbor values too. so you'd have for example 7+2+4 inputs (7 initial + 2 i,j + 4 neighbor values)
also consider using RNNs (recurrent neural networks). if you consider the matrix to be a sequence of values which has internal dependency plus the external dependency to those 7 features, then you can use a RNN network like LSTM.

feel free to update your post with more specific experiments and issues.


Motivation that drives a rogue AI agent This is a kind of biological and philosophical question. So, the recent concern in AI is that an AI agent may go rogue with prominent people voicing their concerns.

Now say, we have created an AI (you are free to use your own definition of what makes an AI intelligent) which has gone rogue with powers given in this question.

Now, the broad view of today's biology is that everything we do is to further our genes down the future (leaving aside small technical details). It is even widely accepted that we are just machines whose controller are the genes. Everything we do is controlled/hardwired by the genes with some avenue of learning from experiences. Also genes only further their own interest. Scientist George Price even wrote a mathematical equation proving all our acts are selfish and only furthering the interest of our genes (article). Also Richard Dawkins is a pioneer of this idea (this is only to show I haven't pulled the idea out from air).

Now, my question is that what will possibly be the motivation of an AI agent to go rogue? It doesn't have genes whose interest it needs to further. We all do something for an end result. What is the end result a rogue AI might try to achieve/attain and why?

Today, prominent machine learning techniques involve trying to minimize some cost function. In many simple cases this cost function is easy to specify, for instance, linear regression is simply trying to minimize the distance between input data and a line of best fit. No matter what the cost function, the agent is trying to minimize it (or maximize a reward function). That is its motivation.

However, as problems become harder it becomes more challenging for humans to design a cost/reward function such that an system/agent is actually trying to do what the humans want it to do. For instance, one might want a cup of coffee and rewards the agent for getting it to them very quickly. In this case, the agent might make the coffee and then throw it at the human which isn’t what the human actually wanted. Something was misspecified (eg. don’t throw or spill it).

Problems like these could result in a rouge AI and its sole motivation would be to minimize its cost function. For instance, this coffee-AI may think that it would never screw up getting coffee (thus get bad reward) if there were no humans to ask for one.


Clarification regarding "Image Crowd Counting Using Convolutional Neural Network and Markov Random Field" I am currently reading the research paper Image Crowd Counting Using Convolutional Neural Network and Markov Random Field  by Kang Han, Wanggen Wan, Haiyan Yao, and Li Hou.
I did not understand the following context properly:

  We employ the residual network, which is trained on ImageNet dataset for image classication task, to extract the deep features to represent the density of the crowd. This pre-trained CNN network created a residual item for every three convolution layer to bring the layer of the network to 152. We resize the image patches to the size of 224 × 224 as the input of the model and extract the output of the fc1000 layer to get the 1000 dimensional features. The features are then used to train 5 layers fully connected neural network. The network's input is 1000dimensional, and the number of neurons in the network is given by 100-100-50-50-1. The network's output is the local crowd count

Can anyone explain the above part in detail?

I will try to do it part by part :

  We employ the residual network, which is trained on ImageNet dataset for image classication task, to extract the deep features to represent the density of the crowd.

If you look at the figure 2, you can see that they use the Neural network architecture ResNet. This is a deep network, here is the paper. It has good performance and do image classification.

  This pre-trained CNN network created a residual item for every three convolution layer to bring the layer of the network to 152

If you are in the layer k, it means this layer has in input, the ouput of the k-3th layer. See the paper, figure 5 explains it well without much explications needed. Furthermore, Resnet has 3 different architectures wih different number of layers, and they take the deeper one, the 152 layers deep Resnet.

  We resize the image patches to the size of 224 × 224 as the input of the model and extract the output of the fc1000 layer to get the 1000 dimensional features

The input of Resnet is images of size 224x224, so they need to resize them to fit the input requirement of Resnet. The output of Resnet is 1000 because Imagenet is a dataset of 1000 classes.

  The features are then used to train 5 layers fully connected neural network. The network's input is 1000dimensional, and the number of neurons in the network is given by 100-100-50-50-1.

Then they give the output of Resnet to their own Network, which is 5 layer deep. See figure 2 of their paper. Obviously, the input layer has 1000 inputs because of the output of Resnet. The network have layers of 100, 100, 50 and finally 1 neuron. See figure 2.

  The network's output is the local crowd count

I don't think I need to explain it, they want only the number of people in the crowd, so they need only one output. This is obviously not a classifcation problem, but a regression problem.

Has you don't really point out what you don't understand, I don't explain it in details. Feel free to ask more precise question if some part are still blurred to you !


What is the difference between learning and non-learning agents? What is the difference between learning agents and other types of agents?

In what ways learning agents can be applied? Do learning agents differ from deep learning

A learning agent is an agent which has a problem generator, a learning element, a critic and a performance element. ProbGen just creates different instances of a problem w.r.t. a domain and a set of parameters in jurisdiction. A learning element is responsible for making efficient choices and performance element chooses actions for the selected choice. It also changes some of the knowledge base. A critic is basically a metric to tell which is a better choice.

Deep learning is a bit complicated than that. Then again, let's simplify. You see, deep learning is a specific arrangement or subsection of neural networks that helps the learning agent learn faster. That's as simply as I could explain it. What we do is that we arrange the layers in a neural network with a purpose and densely pack it to make it more task specific. Obviously, I am leaving out a lot of implementation details but for them, I am afraid, you need to look up a tutorial.

How are they used in AI?
That's problem specific. Learning agents are everywhere from spam detection to roombas to artificial dog pets. Deep learning on the other hand is currently focused on user pattern recognition and personalization in every front. Please let me know if you need me to expand upon this answer.