Why is search important in AI? Why is search important in AI? What kinds of search algorithms are used in AI? How do they improve the result of an AI?

State space search is a general and ubiquitous AI activity that includes numerical optimization (e.g. via gradient descent in a real-valued search space) as a special case.

State space search is an abstraction which can be customized for a particular problem via three ingredients:

Some representation for candidate solutions to the problem (e.g. permutation of cities to represent a Travelling Salesman Problem (TSP) tour, vector of real values for numeric problems).
A solution quality measure: i.e. some means of deciding which of two solutions is the better. This is typically achieved (for single-objective problems) by having via some integer or real-valued function of a solution (e.g. total distance travelled for a TSP tour).
Some means of moving around in the space of possible solutions, in a heuristically-informed manner. Derivatives can be used if available, or else (e.g. for black-box problems or discrete solution representations) the kind of mutation or crossover methods favoured by genetic algorithms/evolutionary computation can be employed.

The first couple of chapters of the freely available "Essentials of Metaheuristics" give an excellent overview and  Michalewicz and Fogel's "How to Solve It - Modern Heuristics" explains in more detail how numerical optimization can be considered in terms of state-space.

How shall the "search through possible plans" occur? The idea is to choose all 3 of the above for the planning problem and then apply some metaheuristic (such as Simulated Annealing, Tabu Search, Genetic Algorithms etc). Clearly, for nontrivial problems, only a small fraction of the space of "all possible plans" is actually explored.

CAVEAT: Actually planning (in contrast to the vast majority of other problems amenable to state-space search such as scheduling, packing, routing etc) is a bit of a special case, in that it is sometime possible to solve planning problems simply by using A* search, rather than searching with a stochastic metaheuristic.


Dealing with "blank" inputs in prediction of a neural network? Say I'm training a neural net to compute the following function:

(color_of_clothing, body_height) -> gender

When using this network for prediction, I can obviously plug in a pair (c, b) to receive a predicted g, but say I want to get a prediction only based on c or only based on b, can I use the same neural net somehow? Or would I need to train two separate neural nets c -> g and b -> g previously?

Or more generally, can I use a neural net that was trained to predict A -> B to make predictions on values from a subset of A, or should I train separate neural nets on all subsets of A that I'm interested in?

I think the answer to your question would be "yes". Though inference would always be best if you provide representative training data. For example you can train your net with (c, b) pairs, (blank, b) pairs, and (c, blank) pairs. That would make the net more robust and likely to support your use case. Training separate nets for each case would be more efficient and accurate - but I'm not sure what your goals and constraints are.

The point is a "blank" input can be trained for as well in the same net. In your case it's a new kind of body_height or color_of_clothing. I also suspect you might have issues with how you encode blank because e.g. rgb(0, 0, 0) would be black, and perhaps you intended to model blank as zero which would mean a blank is black.


Are model-free and off-policy algorithms the same? In respect of RL, is model-free and off-policy the same thing, just different terminology? If not, what are the differences? I've read that the policy can be thought of as 'the brain', or decision making part, of machine learning application, where it stores its learnings and refers to it when a new action is required in the new state.

In respect of RL, is model-free and off-policy the same thing, just different terminology?

No, they are entirely different terms, with the only thing they have in common is that they are both ways in which an RL agent can vary. An agent is generally either working off-policy or on-policy, and is generally either model-based or model-free. These things can otherwise appear in all four combinations.

  If not, what are the differences?

Model-based vs model-free

A model-based learning agent uses knowledge of the environment dynamics in order to make predictions of expected outcomes. A model-free learning agent does not use such knowledge. The model here might be provided explicitly by the developer - that could be code for physics to predict a mechanical system, or it might be the rules of a board game that the agent is allowed to know and query to predict outcomes of actions before taking them. Models can also be learned statistically from experience, although that is harder to make effective.

On-policy vs off-policy

An on-policy agent learns statistically about how it is currently acting, and assuming a control problem, then uses that knowledge to change how it should act in future. An off-policy agent can learn statistically from other observed behaviours (including its own past behaviour, or random and exploratory behaviour) and use that knowledge to understand how a different target behaviour would perform.

Off-policy learning is a strict generalisation of on-policy learning and includes on-policy as a special case. However, off-policy learning is also often harder to perform since observations typically contain less relevant data.

  I've read that the policy can be thought of as 'the brain', or decision making part, of machine learning application, where it stores its learnings and refers to it when a new action is required in the new state.

That's basically correct when considering how an agent learns how to behave in an environment.

You are assigning a bit too much to the word policy here. A policy is strictly only the mapping from a state to an action (or probability distribution over actions), and often written $\pi(a|s)$, i.e. the probability of taking action $a$ given the agent is in state $s$. The "brain" part might include how the agent learns that policy. That could include storing past experience or some summary of past experience in e.g. a neural network.

However, outside of machine learning context, a really simple function containing and if/then statement would also be a policy, if the input to the function was a state of the environment, and the output was an action or probabilities of taking a range of actions. Behaving completely randomly is also a policy, but outside of very specific environments (e.g. Rock/Paper/Scissors) it is usually not the optimal thing to do.


What is the concept of Tensorflow Bottlenecks? What is the concept and how does one calculate Bottleneck values? How do these values help image classification? Please explain in simple words.

The bottleneck in a neural network is just a layer with less neurons then the layer below or above it. Having such a layer encourages the network to compress feature representations to best fit in the available space, in order to get the best loss during training.

In a CNN (such as Google's Inception network), bottleneck layers are added to reduce the number of feature maps (aka "channels") in the network, which otherwise tend to increase in each layer. This is achieved by using 1x1 convolutions with less output channels than input channels.

You don't usually calculate weights for bottleneck layers directly, the training process handles that, as for all other weights. Selecting a good size for a bottleneck layer is something you have to guess, and then experiment, in order to find network architectures that work well. The goal here is usually finding a network that generalises well to new images, and bottleneck layers help by reducing the number of parameters in the network whilst still allowing it to be deep and represent many feature maps.


Using tensor networks as machine learning models Tensor networks (check this paper for a review) are a numerical method originally introduced in condensed matter physics to model complex quantum systems. Roughly speaking, such systems are described by a very high-dimensional tensor (where the indices take a number of values scaling exponentially with the number of system constituents) and tensor networks provide an efficient representation of the latter as an outer product and contraction of many low-dimensional tensors.

More recently, a specific kind of tensor network (called Matrix Product State in physics) found interesting applications in machine-learning through the so-called Tensor-Train decomposition (I do not know of a precise canonical reference in this context, so I will abstain from citing anything).

Now, over the last few years, several works from the physics community seemed to push for a generalized use of tensor networks in machine learning (see this paper, a second one and a third one and this article from Google AI for context). As a physicist, I am glad to learn that tools initially devised for physics may find interdisciplinary applications. However, at the same time, my critical mind tells me that from the machine learning research community's perspective, these results may not look that intriguing. After all, machine learning is now a very established field and it takes probably more than a suggestion for a new machine learning model and a basic benchmarking on a trivial dataset (as the MNIST one) -which is what the papers essentially do in my humble opinion- to attract any attention in the area. Besides, as I believe to know, there already exists quite a solid body of knowledge on tensor analysis techniques for machine learning (e.g.  tensor decompositions), which may cast doubt on the originality of the contribution.

I would therefore be very curious to have the opinion of machine learning experts on this line of research: is it really an interesting direction to look into, or is it just about surfing on the current machine learning hype with a not-so-serious proposal?

I do not feel comfortable proclaiming that I would be a machine learning expert.
But I want to point out that there is indeed interest in applying tensor networks in machine learning settings.
Let me highlight three particular settings in the following.
They can be used to find the governing equations behind dynamical systems analogous to the SINDy algorithm.
The reasoning is that the dynamics of your system may not be sparse but may indeed have a low-rank structure. (See this paper for a comparison of sparse and low-rank structures and this paper for a discussion about why low-rank matrices so often appear in big data.)
The resulting modification of SINDy is the appropriately named MANDy algorithm.
But if your dynamical law exhibits additional structure (despite being low-rank) then this structure is not used. (Requiring you to collect unnecessary samples.)
This work tackles this issue by allowing you to specify the structure that you expect your law to exhibit.
Another area of interest is the application of tensor networks as a means of compression.
Tensor networks can be used to approximate multivariate functions.
Either directly (see this article or this article for further details) or by representing their high-dimensional coefficient tensor (see e.g. here).
The last paper also highlights where such functions may occur: when solving stochastic or parametric PDEs.
"What does this have to do with machine learning?" you may rightfully ask.
These PDEs are hard to solve and it is often easier to minimize a residual than to apply a Galerkin method. The results can be shown to be equivalent with high probability. (See this paper for a proof of this statement and an application to some stochastic PDEs or this paper for an application to an optimal control problem.)
Moreover, there is the problem of tensor completion - a generalization of matrix completion to higher dimensions. (This and this are references for algorithms.)
These methods find application in data science where they can be used to decompose or denoise data. (Unfortunately, I don't have a reference with examples but I remember a presentation where these methods where used on EEG signals.)
Additionally, I recently found this interesting symposium covering even more potential applications of tensor networks in machine learning:
https://itsatcuny.org/calendar/quantum-inspired-machine-learning


Transformers: How to use the target mask properly? I try to apply Transformers to an unusual use case - predict the next user session based on the previous one. A user session is described by a list of events per second, e.g. whether the user watches a particular video, clicks a specific button, etc. Typical sessions are around 20-30 seconds, I pad them to 45 seconds. Here's a visual example of 2 subsequent sessions:

x axis is time in seconds, y axis is the list of events (black line divides the 2 sessions). I extend the vocabulary with 2 additional tokens - start and end of a session (<sos> and <eos>), where <sos> is a one-hot vector at the very beginning and <eos> - similar vector at the end of the session (which makes this long red line).
Now I use these extended vectors of events as embeddings and want to train a Transformer model to predict the next events in the current session based on previous events in this (target) session and all events in the previous (source) session. So pretty much like seq2seq autoregressive models, but in a bit unusual settings.
Here's the problem. When I train a Transformer using the built-in PyTorch components and square subsequent mask for the target, my generated (during training) output is too good to be true:

Although there's some noise, many event vectors in the output are modeled exactly as in the target. After checking train-val-test split is correct, my best guess is that the model cheats by attending to the same day in the target, which the mask should have prevented. The mask is (5x5 version for brevity):
[[0., -inf, -inf, -inf, -inf],
 [0., -inf, -inf, -inf, -inf],
 [0., 0., -inf, -inf, -inf],
 [0., 0., 0., -inf, -inf],
 [0., 0., 0., 0., -inf]]

Note that since I use <sos> in both - source and target - mask[i, i] is set to -inf (except for mask[0, 0] for numerical reasons), so the output timestamp i should not attend to the target timestamp i.
The code for the model's forward method:
def forward(self, src, tgt):
    memory = self.encoder(src)
    out = self.decoder(tgt, memory, self.tgt_mask.type_as(tgt))
    out = torch.sigmoid(out)
    return out

I also tried to avoid the target mask altogether and set it to all -inf (again, except for the first column for numerical stability), but the result is always the same.
Am I using the mask the wrong way? If the mask looks fine, what other reasons could lead to such a "perfect" result?

After shifting the target to the right as suggested in the accepted answer I get the following result:

Which is much more realistic. One suspicious thing is that out[t] now resembles tgt[t - 1], but it can be explained by the fact that the user state tends to be "sticky", e.g. if a user watches a video at t - 1, most likely he will watch it at t as well.

The main issue during training is that you haven't right-shifted the input of the decoder, which is probably why you set the diagonals of mask to -inf (when it should be $0$).
Also, just an FYI, although you haven't focused on evaluation/prediction yet, I will explain the evaluation/prediction here as well for completeness, since it works so differently than training, and also since you will need it when generating the graphs for debugging.
Training
Both tgt and tgt_mask should be changed to simulate auto-regressive properties.
You are feeding in tgt as the input to the decoder, where tgt is the ground truth target sequence to predict. tgt should have dimension length(sequence) x batchSize x $\|dict\|$. Additionally, you are feeding in mask where the diagonals are -inf.
Instead, you should do the following:

Add 2 new special tokens to your vocabulary that signals the start and end of decoding, i.e. <START> and <END>. So your vocabulary will need to be extended to 16 events.
Every input to the decoder (training, evaluation, prediciton) should have <START> as the first token. So during training, when you want to feed in tgt to the decoder, you should right-shift tgt by adding in the <START> token at the beginning.

tgt_shifted will now be of dimension length(sequence)+1 x batchSize x $\| dict \|$
tgt_shifted_mask will now be of dimension length(sequence)+1 x length(sequence)+1. Diagonal should be all $0$
the inputs to the decoder should be tgt_shifted, tgt_shifted_mask, and memory
the output of the decoder will have dimension length(sequence)+1 x batchSize x $\| dict\|$, and will not be right-shifted because it will not have <START> as the first word. But it should have <END> as the last word.
Append <END> to the ground truth tgt, and turn it into one-hot encoding, so tgt should have dimension length(sequence)+1 x batchSize x $\|dict\|$. Your loss operator should be some sort of element-wise comparison between tgt and the output of the decoder.

Note that you only run the decoder once per training batch during training, i.e. the decoder simultaneously predicts the logits of all length(sequence) tokens at the same time. On the other hand, during evaluation/prediction, you must run the decoder length(sequence) times, since you can only use the decoder to predict one token at a time.
Evaluation/Prediction
During evaluation/prediction, the model should generate its own sentence entirely from scratch, one word at a time:

out should be initialized as a single token, <START>, and have dimensions $1$  x batchSize x $\|dict\|$.
Do the following in a loop until termination or max output length is reached:

Generate a out_mask for the current version of out. out_mask should be a matrix with dimension length(out) x length(out). Diagonal should be all $0$
Pass out, out_mask, and memory into the decoder layer. The output of the decoder will be a length(out) x batchSize x $d_{model}$ tensor (note that in your case, $d_{model} = \|dict\|$), which indicates tokenIdx x batchIdx x vocabularyIdx
Convert the last time step (e.g. decoderOutput[-1,:,:] if all sequences in your batch have the same length) into probabilities, and take all the events that surpasses some probability threshold and set them to $1$. Set all other events to $0$. This is the new time step that the decoder layer predicts. So add this time step to out to append it's length by $1$.
Repeat the previous 3 steps in a loop until you reach the <END> token, which signals the end of decoding, or until the max output length is reached.

During evaluation, you can calculate the validation loss by using the logits of only the final iteration of the decoder (which should have dimension length(sequence)+1 x batchSize x $\|dict\|$), and compare those logits with the ground truth with the <END> token added at the end (resulting in dimensions length(sequence)+1 x batchSize x $\|dict\|$).


What is the intuition behind the number of filters/channels for each convolutional layer? After having chosen the number of layers for a convolutional neural network, we must also choose the number of filters/channels for each convolutional layer.
The intuition behind the filter's spatial dimension is the number of pixels in the image that must be considered to perform the recognition/detection task.
However, I still can't find the intuition behind the number of filters. The numbers 128 and 256 are often used in the literature, but why?

The channel sizes 32, 128, etc. are used because of memory and efficiency. There is nothing holy about these numbers.
The intuition behind choosing the number of channels is as follows-
The initial layers extract low-level features- they consist of edge detectors, etc. There aren't many such features. So, we won't gain much by adding a lot of filters(of course, if we use a 3x3 filters on an RGB image, we would have 2^27 different filters even if our neurons have only 0 and 1 as their values. However, most of them are quite similar/meaningless for our job). Using a lot of filters might even lead to overfitting.
The latter layers are responsible for detecting more nuanced features, like elbows/nose shape from the lower level features extracted previously. So, we might do better if we increase the number of channels. Also, note that the resultant layers become more and more sparse as we go deeper.
Though it might differ in applications like super resolution image, in general, the number of channels stays the same or increases when we go deeper.
A nice experiment would be to try and increase the number of channels until you get no more benefit from it. I believe there was a paper that did exactly this(please cite it if someone remembers). You could even try to visualise the filters at this stage and see if the filters are similar or not.
Hope it helps.


Structure of LSTM RNNs I have some very basic questions here. This is probably because I didn't read the relevant documents closely enough. If I used some terminology incorrectly, please point them out. Thank you!

For units/neurons in the hidden layers, they are referred to as memory blocks and each memory block can contain multiple memory cells?
And one memory cell looks like these two?

For units/neurons in the input and output layers, they are simply regular neurons? Like those in a feedforward network?
Regarding the network structure, the structure of an RNN essentially looks like that of a feedforward network, except the neurons in the hidden layer have been replaced by neurons with recurrent connections. Is this an accurate description?
LSTM RNNs then replaces those neurons in the hidden layers with memory blocks (which contain memory cells that have recurrent connections). Is this an accurate description?

Structure of LSTM Networks

[Are] units/neurons in the hidden layers ... referred to as memory blocks and each memory block can contain multiple memory cells?

Although an early guess at the way biological neurons work inspired the Perceptron design that later became multilayer (MLP) and then recurrent (RNN), these artificial network building blocks are not very much like biological neurons. The term is still used by some, but, because of this increased interdisciplinary awareness, the more common terms are cell, unit, or block.
These last two terms can be a little ambiguous when entire networks are used as building blocks in larger AI architectures. This is not a show stopper and also occurs in other engineering disciplines where sub-circuit blocks are components in more complex blocks, and sub-assemblies can be parts of larger mechanical assemblies.

And one memory cell looks like these two?

The images in the questions aren't clear representations. This visualization of an LSTM cell1 shows the attenuation parameters, each labeled with a capital W.

These LSTM cells are arranged as a layer and each pair of adjacent cells within the same layer are connected $(C_t, \, h_t) \rightarrow (C_{t+1}, \, h_{t+1})$. The input in this articular diagram is $x_t$ and the output to the next layer is $h_t$.

For units in the input and output layers, they are simply regular neurons, like those in a feed forward network?

They can be, but there are many topological designs in use that use LSTM cells in some layers and Perceptrons in others, along with side-by-side networks, bi-directional designs, recursive topologies, and completely different cell types, such as the very popular GRU cell.

[Is] the structure of an RNN essentially like that of a feed-forward network, except the neurons in the hidden layer have been replaced by neurons with recurrent connections ...?
LSTM RNNs then replaces those neurons in the hidden layers with memory blocks (which contain memory cells that have recurrent connections) ...?

No general rule can be assumed, especially since new designs emerge nearly every week, and significant topological advancements occur about once every one or two years. This is an example2 of a topology organized to support a specific processing use case.

References
[1] Visualizations of RNN units — Diagrams of RNN unrolling, LSTM and GRU, by Jakub Kvita, 2016, and inspired by Oinkina
[2] Automatic Image Captioning using Deep Learning (CNN and LSTM), Faizan Shaikh, 2018


How do I create a custom gym environment based on an image? I am trying to create my own gym environment for the A3C algorithm (one implementation is here). The custom environment is a simple login form for any site. I want to create an environment from an image. The idea is to take a screenshot of the web page and create an environment from this screenshot for the A3C algorithm. I know the doc and protocol for creating a custom environment. But I don't understand how to create an environment, exactly, based on a screenshot.
If I do so
self.observation_space = gym.spaces.Box(low=0, high=255, shape=(128, 128, 3), dtype=np.uint8)

I get a new pic.
Here's the algorithm that I am trying to implement (page 39 of the master's thesis Deep Reinforcement Learning in Automated User Interface Testing by Juha Eskonen).

The question is conceptually wrong, because of misunderstanding of area. Explanation: The idea is to replace open ai gym by something different. For example: web-site or computer game. There is no way to create an environment based on image. If you want to use implemented algorithm for open ai gym and want to change environment for your own, could do something like this: https://towardsdatascience.com/creating-a-custom-openai-gym-environment-for-stock-trading-be532be3910e. If need to change env complitely than it is necessary to rewrite your algorithm for your tasks.


The use of classification algorithms nowdays What is the utility today of traditional machine learning algorithms such as classification algorithms with the trend of deep learning? I mean can we still use the classification algorithms for some applications?

Machine learning algorithms lay the foundation for deep learning architectures so they pretty much play a role in which ever deep learning application one can make.

The improvement of these machine learning algorithms coupled with deep learning techniques have lead to the establishment of "strong-narrow AI"


What is the difference between the state transition of an MDP and an action-value? Let's say we have MDP where we have a state transition matrix.

How is this state transition different from action value in reinforcement learning? Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?

Transition Probabilities: Consider that you are at state $s$ and from that state take an action $a$.Then there are some probability you will land up at state $s_{1}'$ or $s_{2}'$ ($s'$ indicate the next states). Those probabilities are called transition probabilities. In this example, the transition matrix is just a 3D array since it depends on your state and action($p(s, a)$).

Action value function $Q_{\pi}(s, a)$: It is the expected total reward you get from state $s$, taking action $a$ and thereafter following the policy, $\pi$.

  Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?

The environment can be stochastic or deterministic. If the environment is stochastic then those transitions are stochastic. If the environment is deterministic then those transitions are deterministic.


Should the range and initial values of weights and biases be adjusted to fit input and output data? As a routine (in typical everyday  tasks) of a data scientist, should they usually decide about weights and biases range and initial values as a function of which data they are planning to insert as an input, and which type of data they expect to get in the output? Or we usually do not deal with such fine-tuning, and let the algorithm to do it?
One could answer that normalizing inputs solves the problem and no need to fit weights and biases, but I guess they depend also on expected output.
To summarize:

is it common to deal with weights and biases in everyday tasks or in most of the cases existing algorithms do it well?

what are the rules of thumb for how to decide about range and initial values of weights and biases?

is it common to deal with weights and biases in everyday tasks or in most of the cases existing algorithms do it well?

No; and it is no coincidence that you will not be able to find any reference to such a practice in any course or tutorial about neural networks. Such a practice would require a whole additional level of (business/SME) know-how in order to meaningfully apply neural networks to real-world problems, and fortunately this is not necessary.
The desired situation is for both weights & biases to remain in a relatively small* range around zero, among other reasons because this avoids the exploding & vanishing gradient problems, which is catastrophic for learning; trying to adjust for the scale of our inputs & outputs by scaling accordingly the model weights & biases is the wrong approach, and never followed.
[*How small? Well, see here and here for what a change in just the standard deviation of a zero-mean weight initialization from 1.0 and 0.1 to 0.01 can do to the model performance]
It is a well-established fact by now that neural nets work with normalized inputs, and, depending on the problem, normalized outputs as well; which answers to your objection:

One could answer that normalizing inputs solves the problem and no need to fit weights and biases, but I guess they depend also on expected output.

It depends indeed, but the solution here is to scale the output(s) as well, and not the weights.
Now, it's true that de-scaling the outputs back to their original range in order to be able to meaningfully compare them with the ground truth (and possibly compute more meaningful metrics, like the error on our true output range, and not the scaled one) is AFAIK seldom mentioned in introductory expositions; but this is indeed the correct thing to do (especially for regression problems, where metrics like MSE are scale-sensitive), instead of trying to manually intervene on the weights range. For details, see own answers in How to interpret MSE in Keras Regressor and ANN regression accuracy and loss stuck.

what are the rules of thumb for how to decide about range and initial values of weights and biases?

Range aside, the general rule, as already implied, is to initialize the biases with zeros and the weights with small random values around zero. Nevertheless, the exact details are an area of active research. Currently used initialization schemes that are already integrated into the relevant frameworks (Tensorflow, Keras, Pytorch etc) are the Glorot (or Xavier) and He initializations (for a nice overview, see Weight Initialization in Neural Networks: A Journey From the Basics to Kaiming).
Beyond these routinely-used approaches that have already reached the practitioner's workbench, and moving closer to the front of active theoretical research, the Lottery Ticket Hypothesis (finding "winning" weight initializations that require minimal training) is an ultra-hot topic lately.


Why do DQNs use linear activations on cartpole? I've been reading a lot of tutorials on DQNs for cartpole. In many of them, they have the funnel layer of the neural net be a linear activation.
Why is this?
Is it just a choice made by the implementer?
Is this Choice specific to cartpole, or do most control task dqns use it?
Thanks.

Q learning predicts the action value, $q(s, a)$ for taking action $a$ in state $s$. The action value is usually the discounted sum of all future rewards. In general it can take any scalar value.

DQN uses a neural network to approximate $q(s, a)$. Although you might use this to select an action (thus think of the problem as a classification), the NN has to perform regression to predict the action values.

It is most common to use a linear final layer, and mean squared error loss in DQN, to match this regression task. So yes, you will find most control DQNs will make the same choice as the cartpole example that you are looking at.


How can I benchmark an application of ant colony optimization? In my research on video games path finding I'm using ant colony optimization, not only to find the shortest path, but also to add some unpredictability and adaptiveness to bots path finding. It works the way as players move in the map, they add some pheromone to the map, so it adds up a probability that bots choose path like players. I have sent the paper, but judges said you need a benchmark and distinguish to previous works. Can you tell me how can I benchmark this work?

I would assume that in this case, they are asking you to compare the performance of your particular algorithm to other similar algorithms used in pathfinding. For example, A*


How should I handle invalid actions in a grid world? I'm building a really simple experiment, where I let an agent move from the bottom-left corner to the upper-right corner of a $3 \times 3$ grid world.
I plan to use DQN to do this. I'm having trouble handling the starting point: what if the Q network's prediction is telling the agent to move downward (or leftward) at the beginning?
Shall I program the environment to immediately give a $-\infty$ reward and end this episode? Will this penalty make the agent "fear" of moving left again in the future, even if moving left is a possible choice?
Any suggestions?

In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge.
It may help if you think through what the actual consequences for making the "wrong" move are in your environment. There are a few self-consistent options:

The move simply cannot be made and count as playing the game as intended. In which case, do not allow the agent to make that choice. You can achieve that by filtering the list of choices that the agent is allowed to make. In DQN that will mean supplying an action mask to the agent based on the state so it does not include the action at the stage it makes a choice. This "available actions" function is usually coded as part of the environment.

The move can be attempted, but results in no change to state (e.g. the agent bumps into a wall). If the goal is to reach a certain state in shortest possible time, then you will typically have 0 reward and a discount factor, or negative reward for each attempted action. Either way, the agent should learn that the move was wasted and avoid it after a few iterations.

The move can be attempted, but results in disaster (e.g. the agent falls off a cliff). This is the case where a large negative reward plus ending the episode is appropriate. However, don't use infinite rewards, positive or negative, because that will cause significant problems with numeric stability. Simply large enough to offset any interim positive rewards associated with that direction should be adequate. For a simple goal-seeking environment with no other positive rewards than reaching the goal, ending the episode early is already enough.

When you don't have a toy environment where you get to decide, then the three basic scenarios above can still help. For instance, in most board games we are not interested in having the agent learn the rules for valid moves when they are already supplied by the environment, so the first scenario applies - only select actions from the valid ones as provided by the environment.


Is there any good reference for double deep Q-learning? I am new in reinforcement learning, but I already know deep Q-learning and Q-learning. Now, I want to learn about double deep Q-learning.

Do you know any good references for double deep Q-learning?

I have read some articles, but some of them don't mention what the loss is and how to calculate it, so many articles are not complete. Also, Sutton and Barto (in their book) don't describe that algorithm either.

Please, help me to learn Double Q-learning.

If you're interested in the theory behind Double Q-learning (not deep!), the reference paper would be Double Q-learning by Hado van Hasselt (2010).

As for Double deep Q-learning (also called DDQN, short for Double Deep Q-networks), the reference paper would be Deep Reinforcement Learning with Double Q-learning by Van Hasselt et al. (2016), as pointed out in
 ddaedalus's answer.

As for how the loss is calculated, it is not explicitly written in the paper. But, you can find it in the Dueling DQN paper, which is a subsequent paper where Van Hasselt is a coauthor. In the appendix, the authors provide the pseudocode for Double DQN. The relevant part for you would be:

  $y_{j}=\left\{\begin{array}{ll}r & \text { if } s^{\prime} \text { is terminal } \\ r+\gamma Q\left(s^{\prime}, a^{\max }\left(s^{\prime} ; \theta\right) ; \theta^{-}\right), & \text {otherwise}\end{array}\right.$

  Do a gradient descent step with loss $ \left\|y_{j}-Q(s, a ; \theta)\right\|^{2}$

Here, $y_j$ is the target, $\theta$ are the parameters of the regular network and $\theta^{-}$ are the target network parameters.

The most important thing to note here is the difference with the DQN target:
$y_{i}^{D Q N}=r+\gamma \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime} ; \theta^{-}\right)$.

In DQN, we evaluate the Q-values based on parameters $\theta^{-}$ and we take the max over actions based on these Q-values parametrized with the same $\theta^{-}$. The problem with this is that it leads to an overestimation bias, especially at the beginning of the training process, where the Q-values estimates are noisy.

In order to address this issue, in double DQN, we instead take the max based on Q-values calculated using $\theta$ and we evaluate the Q-value of $a^{max}$ based on a different set of parameters i.e. $\theta^{-}$.

If you want to learn more about this, by watching a video lecture instead of reading a paper, I'd suggest you take a look at this lecture from UC Berkley's DRL course, where the professor (Sergey Levine) discusses this in detail with examples.


What are the common techniques one could use to deal with collisions in a transposition table? Consider an iterative deepening search using a transposition table. Whenever the transposition table is full, what are common strategies applied to replace entries in the table?
I'm aware of two strategies:

Replace the oldest entry whenever a new one is found.

Replace an entry if the new one has a higher depth.

I'm curious about other replacement approaches.

The term you're looking for is "replacement schemes". As far as I'm aware, the primary reference on this is still Replacement Schemes for Transposition Tables, although it is a fairly old paper from 1994.

I'll very briefly summarize the seven different schemes listed in this paper, but full text of the paper is also freely available and contains more info:

Deep: preserve position for which the deepest subtree below it was searched.
New: Always replace an old entry with the newest entry.
Old: Opposite of New. Paper mentions only including it for the sake of completeness, implies that it's probably not a good scheme.
Big1: similar to Deep, but uses the size (in number of nodes) of subtree searched, rather than just its depth. If the same position (i.e. table entry) occurs multiple times in a subtree, it is only counted once.
BigAll: same as above, but really counts the number of nodes rather than number of positions (so a position that occurs multiple times is counted multiple times).
TwoDeep: In this scheme, the TT is modified to contain two positions per table entry, rather than just one position. You could view it as the table having two "layers". In this scheme, a new position is moved into the first "slot" if it has the deepest subtree (as in the Deep scheme), moving the previous position in the first slot into the second slot. If a new position doesn't have a new deepest subtree, it is instead moved into the second slot.
TwoBig1: Similar to the above, but using Big1-style replacement rather than Deep-style replacement.

If I recall correctly, the Two-layered schemes tend to perform best (of course they do require approximately twice as much memory for the same number of bits per key).


What is the difference between a stationary and a non-stationary policy? In reinforcement learning, there are deterministic and non-deterministic (or stochastic) policies, but there are also stationary and non-stationary policies.

What is the difference between a stationary and a non-stationary policy? How do you formalize both? Which problems (or environments) require a stationary policy as opposed to a non-stationary one (and vice-versa)?

A stationary policy, $\pi_t$, is a policy that does not change over time, that is, $\pi_t = \pi, \forall t \geq 0$, where $\pi$ can either be a function, $\pi: S \rightarrow A$ (a deterministic policy), or a conditional density, $\pi(A \mid S)$ (a stochastic policy). A non-stationary policy is a policy that is not stationary. More precisely, $\pi_i$ may not be equal to $\pi_j$, for $i \neq j \geq 0$, where $i$ and $j$ are thus two different time steps.
There are problems where a stationary optimal policy is guaranteed to exist. For example, in the case of a stochastic (there is a probability density that models the dynamics of the environment, that is, the transition function and the reward function) and discrete-time Markov decision process (MDP) with finite numbers of states and actions, and bounded rewards, where the objective is the long-run average reward, a stationary optimal policy exists. The proof of this fact is in the book Markov Decision Processes: Discrete Stochastic Dynamic Programming (1994), by Martin L. Puterman, which apparently is not freely available on the web.


Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward In the appendix of the Constrained Policy Optimization (CPO) paper (Arxiv), the authors denote the discounted future state distribution $d^\pi$ as:
$$d^\pi(s) = (1-\gamma) \sum_{t=0}^\infty{\gamma^t P(s_t = s \vert \pi)}\tag1$$
and the discounted total reward $J(\pi)$ as:
$$J(\pi) = \frac{1}{1-\gamma} E_{\substack{s\sim d^\pi \\ a \sim \pi \\ s' \sim P}}[R(s,a,s')]\tag2$$
I have two questions regarding these equations.
Question 1
Intuitively, I understand that $d^\pi(s)$ returns the discounted probability of landing on state $s$ when executing policy $\pi$.
I understand that the summation part of $(1)$ results in values that are greater than $1$, and are, therefore, not fit for a probability distribution. But I do not understand why the value that results from this is multiplied by $(1-\gamma)$.
I have read in this question that "$(1−\gamma)$ normalizes all weights introduced by γ so that they are summed to $1$". I have confirmed that this is true, but I don't understand why.
I tested this with a simple example:
Suppose there is are only two states $s_A$ and $s_B$ and the probabilty of landing on $s_A$ is $0.4$ and on $s_B$ is $0.6$, independently of the previous state or action taken (therefore, independently of the policy $\pi$). Also suppose we set the maximum number of time steps $t_{max} = 1000$ (to make the equation easy to compute) and $\gamma = 0.9$.
Then:
$$d^\pi(s_A) = (1-0.9) \sum_{t=0}^{1000} 0.9^t \cdot 0.4 \approx (1-0.9) \cdot 4$$
and
$$d^\pi(s_B) \approx (1-0.9) \cdot 6$$
So indeed if we sum them and multiply by $(1-\gamma)$ we get:
$$(1-0.9)\cdot(4+6) = 1$$
Q: My question is why does multiplying by $(1-\gamma)$ normalize to $1$? And what does $(1-\gamma)$ represent in this context?
Question 2
Similarly, I can't understand the use of $\frac{1}{1-\gamma}$ in $(2)$.
Q: How does multiplying the expected value of the reward function by $\frac{1}{1-\gamma}$ result in the discounted reward, instead of multiplying by $\gamma$? What does $\frac{1}{1-\gamma}$ represent?

Question 1
The taylor expansion of $\frac{1}{1-\gamma}$ at $\gamma= 0$ is as follows
$$\frac{1}{1-\gamma} = 1 + \gamma + \gamma^2 + \dots$$
When you multiply by $1-\gamma$ you get
$$ 1 = (1-\gamma)(1 + \gamma + \gamma^2 + \dots)$$
Which can be equivalently written as
$$1 = (1-\gamma)\sum_\limits{i=0}^{\infty}\gamma^i$$
Hence we can see that by multiplying the coefficients by $(1-\gamma)$ we get a weighted sum of transition probabilities

Question 2
Multiplying by $(1-\lambda)$ is to cancel out the normalisation term which was included in the definition of the discounted distribution of the states. Partially expanding out your expressions for the discounted reward you get this
$$ J(\pi) = \frac{1}{1-\gamma}\sum_\limits{s}E_
{a\sim\pi,s'\sim P}[R(s,a,s')]\cdot\left((1-\gamma)\sum_\limits{t=0}^{\infty}\gamma^tP(s_t=s|\pi)\right)$$
The discounted return form is usually denoted
$$J(\pi) = E[\sum\gamma^iR_i(s,a,s')]$$
Where you notice the discounting terms aren't normalised which motivates cancelling it out


Combine two feature vectors for a correct input of a neural network Let's consider this scenario. I have two conceptually different video datasets, for example a dataset A composed of videos about cats and a dataset B composed of videos about houses. Now, I'm able to extract a feature vectors from both the samples of the datasets A and B, and I know that, each sample in the dataset A is related to one and only one sample in the dataset B and they belong to a specific class (there are only 2 classes).

For example:

Sample x1 AND sample y1 ---> Class 1
Sample x2 AND sample y2 ---> Class 2
Sample x3 AND sample y3 ---> Class 1
and so on...

If I extract the feature vectors from samples in both datasets , which is the best way to combine them in order to give a correct input to the classifier (for example a neural network) ?

feature vector v1 extracted from x1 + feature vector v1' extracted from y1 ---> input for classifier

I ask this because I suspect that neural networks only take one vector as input, while I have to combine two vectors

The easiest way can be the concatenation of the feature vectors to create a single feature vector for each sample.

Assume the first sample is made of the pair $X_1$ and $Y_1$. Let the corresponding feature vectors for $X_1$ and $Y_1$ be $\textbf{v}_1$ and $\textbf{v}_2$, respectively.

$$
\textbf{v}_1 = [f_1, f_2, \ldots , f_n],\\
\textbf{v}_2 = [g_1, g_2, \ldots , g_m].
$$
Then, the first sample's feature can be defined as
$$
\textbf{v} = [f_1, f_2, \ldots , f_n, g_1, g_2, \ldots , g_m].
$$
Eventually, when you pass the latter feature vector to a machine learning model, it will try to capture the dependencies among all of these features, to learn a solution for your task of interest (i.e. classification).


What loss function is appropriate for finding "points of interest" in a array of x,y inputs I am looking into whether a neural network is appropriate to detect "points of interest" (POI) in a set of tuples (say length, and some sensor value). A POI is essentially a quick change in the value which doesn't follow the pattern. So if we have a linear increase in the sensor value and then it suddenly jumps by 200% that would be a POI.

Here is an example of the data I am working with:

[(1,10),(2,11),(3,14),(5,24),(6.5,25), (7,26), (8,45)]

In this example lets say "(3,14)", "(5,24)", and "(8,45)" are points of interest. So I am trying to design a neural network which will detect these.

I have started by creating a Convolution 1D layer with a static input length of 500 elements.

After a couple hidden layers I apply a sigmoid function which provides a list of 0s and 1s as output where 1s signify a POI in the set.

There are a couple of issues with this approach which I am trying to solve.

In a categorical loss function an output of [1,0,0,1,0,0] for example would be seen as completely inaccurate if the expected output is [0,1,0,0,1,0] whereas in reality that is fairly accurate since the predicted POIs are very close to the real POIs.

So what I am trying to do is find a loss function to optimize the neural network.

So far I have tried:

Binary Cross Entropy: I read this is good for classifying where inputs can belong to multiple classes. I tried this out thinking each POI is essentially a "category". But this seems to not work and I assume it's because of what I noted above.
Mean Absolute Error: This seems to have gotten slightly better results but after closer inspection it didn't seem very accurate and would mostly uniformly predict POIs on a set.

I have tried a few others without much luck.

What loss function would be more appropriate for this?

One other output I tried was instead of outputting 0s and 1s it should just return the indexes of the points of interest so say 3, 5 8. Would this be a better output?

You don't have to use machine learning to solve the problem.

Unify the scale of each data input (or each curve), such as normalize to $[0,1]$ (not necessary).
Calculate the slope of each pair of points $\frac{(y2 - y1)}{(x2 - x1)}$.
Set the threshold. Compare the difference between two adjacent slopes, the difference exceeds the threshold are marked as POI.

Isn't that simpler?

If you must solve the problem with CNN, what I can think of is that you first collect (or draw) a bunch of curves, mark the POI in advance, and then feed it to the CNN model.


Can artificial intelligence also make mistakes? The intelligence of the human brain is said to be a strong factor leading to human survival. The human brain functions as an overseer for many functions the organism requires. Robots can employ artificial intelligence software, just as humans employ brains.

When it comes to the human brain, we are prone to make mistakes. However, artificial intelligence is sometimes presented to the public as perfect. Is artificial intelligence really perfect? Can AI also make mistakes?

Since the question was tagged with philosophy, I'll provide a philosophical perspective.

There is no definition of perfection. Just think about the silly example that is always brought up when we talk about autonomous cars: You are about to crash into a group of school kids and the AI's only chance to avoid them is by crashing into an old person. What should the AI do in order to react "perfectly"?

All actions that we take are based on our personal value system, so what is perfect to one person might be completely nonsense to another person. It might be possible to create an AI that always acts perfectly in terms of the value system you fitted it out with, but even in this case there are some caveats like:

reaction time: is the system fast enough to process all available information fast enough in order to take a fast action?
available information: not all information might be available with the system. Do you consider it acting imperfect, if the action of the system turns out to be wrong once you posses all information about the situation, even though some information will never be available to the system (e.g. Heisenberg's principle of uncertainty)


How are generative adversarial networks trained? I am reading about generative adversarial networks (GANs) and I have some doubts regarding it. So far, I understand that in a GAN there are two different types of neural networks: one is generative ($G$) and the other discriminative ($D$). The generative neural network generates some data which the discriminative neural network judges for correctness.
How do the discriminative ($D$) neural nets initially know whether the data produced by $G$ is correct or not? Do we have to train the $D$ first then add it into the GAN with $G$?
Let's consider my trained $D$ net, which can classify a picture with 90% accuracy. If we add this $D$ to a GAN, there is a 10% probability it will classify an image wrong. If we train a GAN with this $D$, then will it also have the same 10% error in classifying an image? If yes, then why do GANs show promising results?

Compare generated and real data

All the results produced by G are always considered "wrong" by definition, even for a very good generator.

You provide the discriminative neural network $D$ with a mix of results generated by the generator network $G$ and real results from an outside source, and then you train it to distinguish if the result was produced by the generator or not - you're not comparing "good" and "bad" results, you're comparing real versus generated results.

This will result in a "mutual evolution" as $D$ will learn to find features that separate real results from generated ones, and $G$ will learn how to generate results that are hard to distinguish from real data.


What is regression layer in a spatial transformer? I came across this line while reading the original paper on Spatial Transformers by Deepmind in the last paragraph of Sec 3.1:

  The localisation network function floc() can take any form, such as a fully-connected network or a convolutional network, but should include a final regression layer to produce the transformation parameters θ.

I understand what regression is, but what is meant by a regression layer?

Basically, it means that the "localization network" should output a set of real valued parameters (typically 6 numbers). The word "regression" doesn't bear any specific meaning.

Any network that relies on the original image as input (directly or indirectly) and outputs 6 numbers, would work. And its last layer would qualify as "regression layer" as long as it is not restricted in the real domain (not normalized, softmaxed, etc)


What is the approach to deduce formal rules based on data? We have data in text format as sentences.
The goal is to detect rules which exist in this set of sentences.

I have a limited set of contextless sentences that fit a pattern and want to find the pattern.
I might not have sentences that don't fit the pattern.

What should be an approach to do that?

If you don't have non-examples of your pattern and don't have some kind of heuristic guide, unfortunately the answer is that you can't. "All sentences" will always be 100% compatible with your examples, and you'll never be able to collect evidence that disconfirms (or even decreases the likelihood) of that hypothesis. Even if you rule out that hypothesis by fiat, the hypothesis that the only non-accepted sentence is

  The answers for “I have a bunch of books and want to infer topics” and “I have a set of contextless sentences that fit a pattern and another set of contextless sentences that don’t fit that pattern and want to find the pattern” are extremely different.

will be impossible to rule out

With a good enough heuristic guide, you might be able to do something, but that would require knowing statistical facts about examples and non-examples and then sampling from the world IID, and just happening to not get any non-examples. That doesn't seem like the situation you're in. If you really have nothing to quantify the non-examples in any way, there's nothing that can be done.


How to train an LSTM to classify based on rare historic event? I want an LSTM to output one of two classes (Y, N), per frame, based on all the input so far.
My original inputs are very long (~100000 samples long, far more than a standard LSTM training can handle due to vanishing gradients).

If the last seen instance out of the tokens (A, B) was A, output Y.
If the last seen instance out of the tokens (A, B) was B, output N.
The very long sequence is guaranteed to start with either A or B.

If the sequence was short, this would be quite easy.
For example, the following top lines and bottom lines correspond to inputs and required outputs:
ABCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
YNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN

ACCCCCCCCCBCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCCACCCCCCCC
YYYYYYYYYYNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNYYYYYYYYYYYYYYYYYYYYYYYYYY

Looks easy enough, just push batches comprised of chunks of the long sequence to the LSTM and have a coffee, right?
However, for my case, the available inputs are (A, B, C), of which (A, B) are extremely rare, meaning I can have batches comprised of 100% C's. The LSTM has no chance then, if not fed with some current state, telling it about the last A or B seen.
Unfortunately, this "state" is really something learned, and I can't just feed it as input AFAIK.
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
????????????????????????????????????????????????????????????????????

I am looking for a standard practice, or other references on how to train an LSTM or other RNN based model to be able to classify based on rare events far in history.
I hope this is clear, if not please ask and I will edit.

Please note that the data is labeled, and labeling can't be generated automatically for this task. The above is just an example for ease of understanding, the reality is more complicated.

How about a Temporal Convolutional Network? It feels like for such a long sequences having the recurrent/memory based approach is not too feasible. But, intuitively, the 1D convolutions should be able to pick out those rare features from your extremely long sequences.
There are also claims that TCNs are comparable to RNNs in performance on common tasks, so there's that.


Overfitted model performs better in test set There are two models for the same task:

model_1: 98% accuracy on training set, 54% accuracy on test set.
model_2: 48% accuracy on training set, 47% accuracy on test set.

From the statistics above we can say that model_1 overfits training set.
Q1: Can we say that model_2 underfits?
Q2: Why model_1 is bad choice if it performs better than model_2 on test set?

Whether a model is over-fitting the data cannot be determined solely from false positive and false negative data on training and test runs.  If the sets are too small, very little can be determined.  If the sample selection method is choosing inequitably with respect to the training model, accuracy information may be misleading.

If the first run was just a matter of luck in selection, that could be determined by making multiple training runs with the first model.

Even if over-fitting is the cause of the drop in testing accuracy for the first model, that does not mean that the over-fitting model isn't none-the-less more fit that the second model.  It doesn't appear to be over-fit from that one training and test run, but may be deficient in other ways.  The second model may under-fit, for instance.


If use the weights from previous iteration of a k-fold cross validation to seed a neural network classifier would I be overfitting? As is done traditionally, I used k-fold cross validation to select and optimize the hyper parameters of my neural network classifier. When it was time to store the final model for future predictions, I discovered that using the weights from the previous k-fold cv iteration to seed the initial weights of the model in subsequent iteration, helps in improving the accuracy (seems obvious). I can use the model from the final iteration to perform future predictions on unseen data.

Would this approach result in overfitting?

(Please note, I am using all available data in this process and I do not have any holdout data for validation.)

To answer directly your question: estimating a model's performance on data previously used for fitting will overestimate performance.

When your dataset is "small" you are faced with a bias-variance dilemma with regards to how much data should be used for train and test sets:

Too much train data, you end up with little test samples and your
performance estimate has high variance.
Too much test data, your train samples do not represent well the
population you are trying to model/target, so your average
performance will be much lower than what could be achieved.

K-fold CV is a compromise in the evaluation of performance of certain procedure. Once you have settled on a certain model or hyperparameters, when moving to production you can:

Choose to take one of the k models you have trained, perhaps using the "one std variation rule of thumb".

Re-train on all of your data and you can expect the resulting model
to be at least as good as you have estimated.
Use all k models to form an ensemble and you can expect the resulting
model to be at least as good as you have estimated.


Text Capturing on the Images I want to capture text and letters on images (png, jpeg, etc.).  Is it Possible Which algorithm/software can I use?

Right now I am using R with the tesseract package but it's not solving my problem: if letters on images are  different colors and different format, sizes in those situations don't work.

I don't have the any training data to compare text on the image with the existing data set. So in those situations is it possible?

What you call is a typical OCR application. There are two things to consider :

1- Localizing the text

2- Recognizing the text

Please note that it is not an easy task to solve the problem globally. However, depending on your working subset, you need to get the data first. There are a lot of open source ocr database that you can find.
Then, you need to add some preprocessing and it depends on which kind of path you want to choose. You can find lots of articles about it.
Then, you need to add a localization network (like Faster R-CNN or any specialized one for OCR), with the combination of recognition network.
This was deep learning approach. It has a learning curve if you have never worked with deep learning frameworks.

Another way that you can take is to get different character image dataset. Resize all of them to same size, flatten to 1-D and train a classifier. Then, you can start to distinguish different image patches if they are character or not (or which character). Then, for inference, you need to crop different sub-regions from input image and feed it to classifier to get a result.
It is more costly in terms of inference time and the success rate will more likely to be much lower than deep learning method but you can learn & implement this one much quicker.


Neural network for data visualization At my work we're currently doing some research into data visualisation for highly inter connected data, basically graphs.

We've been implementing all sorts of different layouts and trying to see which fits best, but, due to the nature of the problem --it's a visual thing - we needed to come up with some automated way to analyse the result so welcome up with a bunch of metrics to analyse our layouts.

So far, the most important metrics have been information density, edge crossings, node overlap and edge length. This gives us some good results and has allowed us to fine tune our layout algorithms.

However, when a new graph is loaded, we noticed that humans still tend to fiddle a lot with the structure of the layout. Moreover, it seems that our metrics do a good job of predicting where a user is likely to mess around. Graph layout is a tough problem, so after some discussion, the idea of just throwing data at a neural network and let it figure it out came up.

None of us are experts, or even experienced in AI.  I'm the one with the most contact with AI methods.  All I've ever done were simple NN models, no convolution, feedback or feedforward or anything of the sorts, but it seems to me this should be doable.

Maybe it's my lack of expertise here but I haven't been able to find any good information on this sort of application for NNs, so I was hoping someone here could point me in the right direction.

What sort of model is best for such a situation? and why? Is this actually possible or would it be super complicated? Has anyone ever tried something like this before?

If it helps, our input data (for v1, I guess) would be two arrays of variable length, one for the nodes and another for the relationships between them and the output data would be an array with the node XY coordinates.

Any help would be greatly appreciated!
Cheers!

Neural networks are used to visualize high dimensional data through the use of autoencoding. It's similar to Principal Component Analysis and is regarded to perform better then PCA. Autoencoding will take your data and convert it to a 2 or 3-dimensional representation. Since you have an array of data you might want to use LSTM. You will have to make sure you reset the LSTM states when you perform the reverse pass. You will probably have trouble interpreting the autoencoded form of the data though, so that will be a task of its own.


Is explainable AI more feasible through symbolic AI or soft computing? Is explainable AI more feasible through symbolic AI or soft computing?

How much each paradigm, symbolic AI and soft computing (or hydrid approaches), adresses explanation and argumentation, where symbolic AI refers e.g. to GOFAI or expert systems and soft computing refers to machine learning or probabilistic methods.

XAI is relevant to "black box" AI (machine learning methods where the decision making rationale is not apparent, only the structure of the system that led to that decision.)

Symbolic AI, GOFAI, and Expert Systems are both explainable and understood, in that the the decision-making process is designed by humans.  (Symbolic AI involves human-readable representations of problems.)

To directly answer, XAI is not only feasible in the latter cases, it is a prerequisite. The difficulty is in making black box decision-making explainable.


Prerequisites for Andrew Ng's Machine Learning Course I am planning to enroll for Andrew Ng's Machine Learning course https://www.coursera.org/learn/machine-learning. I've no background in math. Is it OK if I start the course and learn math as and when required?

This course is focused on machine learning using MATLAB, which is not practical nowadays as it is a programming language used specifically for computing, and cannot display GUI or communicate through the network. The language is powerful but limited in some ways. Nowadays most people use python for machine learning, as it is versatile and can connect to other backend like C++, java, JavaScript easily. The language is also a general language, and unlike MATLAB can do many things not limited to computing.

If you really want to join this course, I would recommend first learning MATLAB language and also learn basic calculus like derivatives. This would greatly help on your learning of the course.

However if you want to here serious about machine learning, I would encourage you to enroll in the Deep Learning specialization also by Andrew Ng on Coursera.
https://www.coursera.org/specializations/deep-learning
This course uses python as the programming language and teaches more modern approaches to deep learning like recurrent neural networks, convolutional neural networks and more. It also talks more about application of neural network. There is also theory, but it also talks about application of a specific algorithm and how it works.

Hope I can help you.


Why use the output of the generator to train the discriminator in a GAN? I've been doing some reading about GANs, and although I've seen several excellent examples of implementations, the descriptions of why those patterns were chosen isn't clear to me in many cases.

At a very high level, the purpose of the discriminator in a GAN is establish a loss function that can be used to train the generator.

ie. Given the random input to the generator, the discriminator should be able to return a probability of the result being a 'real' image.

If the discriminator is perfect the probability will always be zero, and the loss function will have no gradient.

Therefore you iterate:

generate random samples
generate output from the generator
evaluate the output using the discriminator
train the generator
update the discriminator to be more accurate by training it on samples from the real distribution and output from the generator.

The problem, and what I don't understand, is point 5 in the above.

Why do you use the output of the generator?

I absolutely understand that you need to iterate on the accuracy of the discriminator.

To start with it needs to respond with a non-zero value for the effectively random output from the generator, and slowly it needs to converge towards correctly classifying images at 'real' or 'fake'.

In order to achieve this we iterate, training the generator with images from the real distribution, pushing it towards accepting 'real' images.

...and with the images from the generator; but I don't understand why.

Effectively, you have a set of real images (eg. 5000 pictures of faces), that represent a sample from the latent space you want the GAN to converge on (eg. all pictures of faces).

So the argument goes:

As the generator is trained iteratively closer and closer to generating images from the latent space, the discriminator is iteratively trained to recognise from the latent space, as though it had a much larger sample size than the 5000 (or whatever) sample images you started with.

...ok, but that's daft.

The whole point of DNN's is that given a sample you can train it to recognise images from the latent space the samples represent.

I've never seen a DNN where the first step was 'augment your samples with extra procedurally generated fake samples'; the only reason to do this would be if you can only recognise samples in the input set, ie. your network is over-fitted.

So, as a specific example, why can't you incrementally train the discriminator on samples of ('real' * epoch/iterations + 'noise' * 1 - epoch/iterations), where 'noise' is just a random input vector.

Your discriminator will then necessarily converge towards recognising real images, as well as offering a meaningful gradient to the generator.

What benefit does feeding the output of the generator in offer over this?

The main reason that the discriminator is trained concurrently with the generator is to provide (at least in theory) a smooth and gradual learning signal for the generator.

If we trained the discriminator on only the input data, then, assuming our training algorithm converges well, it should quickly converge to a fixed model. The generator can then learn to fool this fixed model, but it will likely still be easy to spot the generator's fakes for a human. For example, the discriminator may have learned that, coincidentally, in the sample you provided, all images of trucks have a fully white pixel in the top left corner. If it learns that pattern, the generator can fool it by generating noise with a white pixel. Once the generator has learned this pattern, all learning stops.

In contrast, suppose that the discriminator is repeatedly re-trained on a mixture of real and generated examples. The discriminator will be forced to learn more complex patterns than "white pixel in upper left", which improves its quality beyond the raw patterns in the sample data.

The converse relationship is also true. If the generator is trained only on the training data, it will also likely pick out only the most obvious patterns. These patterns are likely to create many local minima in the weight space for the network. However, if the error signal from the discriminator is fed to the generator, then the generator must adapt: in effect, we are telling it "making the top left pixel white is not good enough to fool observers. Find more complex patterns".


Understanding a few terms in Andrew Ng's definition of the cost function for linear regression I have completed week 1 of Andrew Ng's course. I understand that the cost function for linear regression is defined as $J (\theta_0, \theta_1) = 1/2m*\sum (h(x)-y)^2$ and the $h$ is defined as $h(x) = \theta_0 + \theta_1(x)$. But I don't understand what $\theta_0$ and $\theta_1$ represent in the equation. Is someone able to explain this?

Linear regression is always associated with an activation function, the weights between layers and the structure of the network. The weights between layers are $\theta_0$ and $\theta_1$. These weights and the input features undergo the dot product operation, which is then the input to the activation function of the next layer's nodes.

An apparently different but the same use of $\theta_0$ and $\theta_1$ is as coefficients to one or more number of terms which themselves are a combination of the input vectors.

Broadly, $\theta_i$ denotes a weight, i.e. how much preference you want to give to a feature.


Is it possible to implement Asimov's Three Laws of Robotics? Would it be possible to put Asimov's three Laws of Robotics into an AI?
The three laws are:

A robot (or, more accurately, an AI) cannot harm a human being, or through inaction allow a human being to be harmed1

A robot must listen to instructions given to it by a human, as long as that does not conflict with the first law.

A robot must protect its own existence, if that does not conflict with the first two laws.

The most challenging part is this section of the first law:

  or through inaction allow a human being to be harmed

Humans manage to injure themselves unintentionally in all kinds of ways all the time. A robot strictly following that law would have to spend all its time saving people from their own clumsiness and would probably never get any useful work done. An AI unable to physically move wouldn't have to run around, but it would still have to think of ways to stop all accidents it could imagine.

Anyway, fully implementing those laws would require very advanced recognition and cognition. (How do you know that industrial machine over there is about to let off a cloud of burning hot steam onto that child who wandered into the factory?) Figuring out whether a human would end up harmed after a given action through some sequence of events becomes an exceptionally challenging problem very quickly.


Neural networks of arbitrary/general topology? Usually neural networks consist from layers, but is there research effort that tries to investigate more general topologies for connections among neurals, e.g. arbitrary directed acyclic graphs (DAGs).

I guess there can be 3 answers to my question:

every imaginable DAG topology can be reduced to the layered DAGs already actively researched, so, there is no sense to seek for more general topologies;
general topologies exist, but there are fundamental restrictions why they are not used, e.g. maybe learning is not converging in them, maybe they generate chaotic osciallations, maybe they generate bifurcations and does not provide stability;
general topologies exist and are promising, but scientists are not ready to work with them, e.g. maybe they have no motivation, standard layered topologies are good enough.

But I have no idea, which answer is the correct one. Reading the answer on https://stackoverflow.com/questions/46569998/calculating-neural-network-with-arbitrary-topology I start to think that answer 1 is the correct one, but there is no reference provided.

If answer 3 is correct, then big revolution can be expected. E.g. layered topologies in many cases reduces learning to the matrix exponentiation and good tools for this are created - TensorFlow software and dedicated processors. But there seems to be no software or tools for general topologies is they have some sense indeed.

The simplistic neural networks that have been given away for free after they prove insufficient by themselves in field use consist solely of two orthogonal dimensions.

Layer width — the number of ordinal numbers or floating point numbers that represent the signal path through any given layer comprise of an array of layer elements
Network depth — the number of layers in the primary signal path, which is the number of activation function arrays or convolution kernels or whatever other elements Usually neural networks consist from layers

However, in large corporations that have AI pipelines, this is not the case.  We are beginning to see more interesting topologies in open source.  We see this in generative systems for images, text, and speech.  We see this in robotic control of robots.  The truth is that these more sophisticated topologies have been in play for years, but were just not appearing in the open source community because they were company confidential.  Enough academic work, releasing of portions of corporate IP, and the accumulation of independent OSS work has occurred to start to see these topologies in GIT repos.

Cyclic Not Acyclic

Artificial network topologies are generally cyclic, not acyclic in terms of their causality or their signal pathways, depending on how you depict them theoretically.  These are three basic examples from among dozens in the literature and in the open source repositories.

Back-propagation represents the introduction of a deliberate cycle in signal paths in a basic multilayer perceptron, making that topology a sequence of layers represented by vertices, connected sequentially by a set of directed edges representing forward propagation, and a set of directed edges in the reverse direction to distribute the corrective error determined at the network output according to the principle of gradient descent.  For efficiency, the corrective signal is distributed recursively backward through the layers to the $N - 1$ matrices of parameters attenuating the signals between $N$ layers.  Back propagation requires the formation of these $N - 1$ cycles for convergence to occur.
In a generative adversarial network (GAN), we have the signal path of each of the two networks feeding the training criteria of the other.  Such a topological arrangement is like negative feedback in a stable control system in that an equilibrium is formed between the generative network and discriminative network. The two directed edges, (a) the one that causally affects G with Ds result, and (b) the one that causally affects D with Gs result, create a cycle on top of the cycles in each of G and D.
In attention based networks being touted as theoretically advantageous over LSMT (which has been dominating over CNNs) has a much more complex topology and more cycles above those in supervisory layer than those in GANs.

Analysis of Answer One of Three

It is true that every directed graph can be realized in an arbitrarily large RNN because they are Turing complete, but that doesn't mean they are a great topology for all finite algorithms.

Turing was aware that his punched tape model was not the best general purpose, high speed computing architecture.  He was not intending to prove anything about computing speed but rather what could be computed.  His Turing machine had a trivial topology deliberately.  He wanted to illustrate his completeness theorem to others and resurrect the forward movement of rationalism after Gödel disturbed it with his two incompleteness theorems.

Similarly, John von Neumann proposed his computing architecture, with a central processing unit (CPU) and unified data and instruction bus, to reduce the number of relays or vacuum tubes, not to maximize parallel algorithm execution.  That topology as a directed graph has the instruction controller and the arithmetic unit in the center and everything else branching out from the data and address bus leading from them.

That a topology can accomplish a task is no longer a justification for persisting in the use of that topology, which is why Intel acquired Nirvana, which deviates from traditional von Neumann architecture, DSP architecture, and the current CUDA core architecture that NVidia GPUs use and offer for artificial network realization through C libraries that can be called via integrated Java and Python adapters.

There is definitely sense to seek for more general topologies, if they are fit for the purpose, just as with Turing's or von Neuman's.

Analysis of Answer Two of Three

General topologies exist, the most economically viable of which is the CUDA cores begun by NVidia, which can be configured for MLPs, CNNs, RNNs, and general 2D and 3D video processing.  They can be configured with or without cycles depending on the characteristics of the parallelism desired.

The realization of topologies unlike the Cartesian arrangements of activation functions in artificial networks or the kernel cells in convolution engines do have barriers to use, but they are not fundamental restrictions. The primary barrier is not one of hardware or software.  It is one of linguistics.  We don't think topologically because we don't talk topologically.  That's what is great about this question's challenge.

FORTRAN began to dominate over LISP during the time when general purpose programming began to emerge in many corporations.  That is not surprising because humans communicate in orthogonal ways.  It is cultural.  When a child scribbles, teachers are indoctrinated to say nice things but respond by drawing a shape.  If the child draws a square, the teacher smiles.  The child is given blocks.  The books are rectangular.  Text is justified into rectangles.

We can see this in building architecture dating back to Stonehenge.  Ninety degree angles are clearly dominant in artificial things, whereas nature doesn't seem to have that bias.

Although directed graphs were easy to implement and traverse in recursive structure and were commonplace in the LISP community.  FORTRAN with its realization of vectors and matrices in one and two dimensional arrays respectively were easier to grasp by those with less theoretical background in data structures.

The result is that even if learning EMMASCRIPT (JavaScript) which has its seed from the LISP community and is not biased toward orthogonal data structures, people tend to proceed from HelloWorld.js to something with a basic loop in it, with an underlying array through which the loop iterates.

There are three wonderfully inquisitive and insightful phrases in answer two of three.

Maybe learning is not converging in them — Interestingly an algorithm cannot learn without a cycle.  Directly applying a formula or converging using a known convergent series of terms does not qualify as learning.  Gradient descent relies entirely on the cyclical nature of a corrective action at the end of each sample processing or batch of them.
Maybe they generate chaotic [oscillations] — This gets into chaos theory and control theory's concept of stability.  They can do so, but so can a basic multilayer perceptron if the learning rate is set to high.
Maybe they generate bifurcations — Now we have fully entered the realm of chaos, which is arguably closely related to creativity.  Mendelbrot proposed the relationship between new forms of order and the apparent chaotic behavior arising from the appropriate level of feedback in a system with signal path components that cannot be modelled with a first degree equation.  Since then, we find that most phenomena in nature are actually strange attractors.  The plot of the training of a network from a continuous feed of consistently distributed data in phase space will reveal ... you guessed it ... a strange attractor.  When purtibations are deliberately injected into a training epoch from a pseudo-random number generator, the specific purpose is a bifurcation, so that the global optimization can be found when the training gets stuck in a local optimization.

Analysis of Answer Three of Three

General topologies exist and are promising and researchers are ready to work with them.  It is enthusiasts that can have a dismissive attitude.  They don't yet understand the demos they've downloaded and painstakenly tweaked to run on their computer, they're about to launch their AI carrier amidst the growing demand from all the media hype, and now someone is introducing something interesting and not yet implemented in code.  The motivational direction is generally to either dismiss or resist the creative proposals.

In this case, Google, CalTech, IBM, MIT, U Toronto, Intel, Tesla, Japan, and a thousand other governments, institutions, corporations, and open source contributors will solve that problem, provided people keep talking about topology and the restrictions inherent in purely Cartesian thinking.

Misunderstanding Topology to Mean Dimensionality or Topography

There has been some confusion in terms.  The SO reference in the question is an example of thinking that changing an array dimension is changing the topology.  If such were so, then there would be no change one could make to the geometry of an AI system that would not be topological.  Topology can only have meaning if there are features that are not topological.  When one draws a layer, they don't need to increase the height of the rectangle representing it if the number of activitations, the width of the layer, is changed from 100 to 120.

I've also seen academic papers that called the texture or roughness of an error surface its topology.  That completely undermines the concept of topology.  They meant to use the term topography.  Unfortunately neither the publisher nor the editor noticed the error.

Software or Tools

Most programming languages support directed graphs in recursive hashmaps.  LISP and its derivatives supported them at a more machine instruction efficient level, and that's still the case.  Object oriented databases and graph libraries exist and are in use.  Google uses them extensively in web indexing and lookup.  FaceBook's API is called the Graph API, because it is a query and insert API into the graph that is FaceBook's user data store.

The explosion is here in global software giants.  There is open source for it.  The revolution that is missing is among those who are not yet educated as to the meaning of topology, the difference between a hierarchy and a network or the role of feedback in any learning system.

Regarding Java and Python there are many barriers to the revolution in thinking, primarily these.

There are now key words in either Java or Python to directly deal with directed graphs other than the idea of a class with references to instances of other classes, which is quite limited.  None of these languages can add edge types with a single simple language construct.
There is no mapping to hardware yet, although Nirvana allegedly developed one, and Intel acquired Nirvana, so that barrier may evaporate soon.
The bias still exists in preschool, kindergarten, and first grade
Hilbert spaces are not generally taught in calculus

Graphviz and other graphing software that auto-generates diagrams from unconstrained directed or bidirectional graph representations have done much to bust through the barriers because the generated images are visible across the web.  It may be through visual representations of graphs that linguistic representations, thought, hardware, and software begin to emerge representing the paradigm shift the question investigates.

It is not that constraints are not useful.  Only some patterns and paradigms produce results, but since the results from the human brain demand attention, and the human brain is

Not at all orthogonal,
Not implemented using Cartesian neural patterns, and
Not topologically a box,

One can all but conclude that those are not particularly well chosen constraints.  Neither is the acyclic criteria.  Nature is cyclic, and intelligence probably requires it in many ways and at many levels.


What is the state-of-the-art algorithm for neural style transfer? I've read the paper A Neural Algorithm of Artistic Style by Gatys et. al. and I find the application of neural style transfer very fun.
I also read that Exploring the structure of a real-time, arbitrary neuralartistic stylization network by Ghiasi et. al. is a more modern approach to NST.
My question is whether the above paper by Ghiasi et. al. is still the state-of-the-art method in NST, or maybe new algorithms perform even more efficiently.
I shall precise that my goal is to deploy some NST algorithm on a web page as a fun project to apply some deep learning and learn about backend-frontend interactions.

So for neural style transfer, using the particular method described in Gatys paper, nobody has done better than using VGG net. This is seemingly due to VGGs inherent stability and inability to learn non-robust features of images. More on this here: https://reiinakano.com/2019/06/21/robust-neural-style-transfer.html
That being said, GANs have had huge success in the field of style transfer, getting much better results than the neural style transfer method described in the paper you mentioned. Cyclegan is one of the best in this respect: https://machinelearningmastery.com/what-is-cyclegan/


What is the derivative of the Leaky ReLU activation function? I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the derivative of the Leaky ReLU is.

Here is the C# code for the leaky RELU function which I got from this site:

private double leaky_relu(double x)
{
    if (x >= 0)
        return x;
    else
        return x / 20;
}

Derivative gives the rate of change in $y$ for a small change in $x$ or the slope of a function at point $x$.

In the above function,

y = x      for x >= 0,     i.e. y/x = 1
y = x/20   for x < 0,      i.e.  y/x = 1/20

The following function returns the derivative of leaky ReLU as explained

private double leaky_relu_derivative(double x)
{
    if (x >= 0)
        return 1;
    else
        return 1.0 / 20;
}


How is G(z) related to x in GAN proof? In the proofs for the original GAN paper, it is written:

$$∫_x p_{data}(x) \log D(x)dx+∫_zp(z)\log(1−D(G(z)))dz
=∫_xp_{data}(x)\log D(x)+p_G(x) \log(1−D(x))dx$$

I've seen some explanations asserting that the following equality is the key to understanding:

$$E_{z∼p_z(z)}log(1−D(G(z)))=E_{x∼p_G(x)}log(1−D(x))$$

which is a consequence of the LOTUS theorem and $x_g = g(z)$. Why is $x_g = g(z)$?

It's not supposed to be derived from some equation. That is the basic premise under which GANs work. The output of the Generator $G(z)$ is fed as an input $x_g$ to the discriminator.


How should I design this LSTM network to perform stock prediction? I'm trying to develop a stock predictor.

I'm using LSTM but I am unsure about the structure of the Neural Network. For example, I'm assuming that the Neural Network is a many-to-one since we have many inputs (i.e Open, Close etc) and one output (stock price).

By misunderstanding is coming with how to construct the nodes. For example, what input goes into the "Cell" (or node)? I.e does say 60 timestep mean 60 days of 'Open Price' are fed into the Neural Network at t and then 60 days of 'Close' into t + 1 until we use all input to produce an output?

If someone could explain the process of how LSTM are used with stock predictions that would be appreciated.

I can't say for all cases, but I can certainly give you an example of how an LSTM could be used to predict stock prices.

An LSTM is temporal, meaning you can feed in one input, get an output and the network will remember this interaction (if it's important), which will effect the outcome of future predictions. As such, it is reasonable to feed in all the data for the day (Open, Close etc), obtain an output for the predicted stock price of the following day (or a prediction for all the inputs you feed in, the output can be as large or small as you want), then repeat for as long as you want, each time only feeding in the data for that particular day as the network will remember previous days.

In your example, 60 timesteps would mean doing a forward pass of the LSTM 60 times, for 60 days. Each of these forward passes will produce an output, which you can compare to the next actual stock price for verification, until you reach the current day, where the prediction is for the future.


What are the advantages of having self-driving cars? What are the advantages of having self-driving cars?

We will be able to have more cars in the traffic at the same time, but won't it also make more people choose to use the cars, so both the traffic and the public health will actually become worse?

Are we really interested in this?

One of the main arguments for self-driving cars is that presumably they'll get better and better at driving as the technology progresses, they have no temporal attention deficits or aggressive urges or drug habits and sense their environment 360°, all the while communicating with the other cars, which all together basically amounts to LESS DEAD PEOPLE.

We are really interested in this.

It is also unclear whether most people will actually own cars in 30 years. Maybe there'll be a net of mini busses with flexible routes which take you from door to door on demand. That would reduce traffic quite a bit and there would also be less incentive to drive 200 m to get cigarettes or something. Self-driving cars would allow us to use the car as a resource a lot more efficiently, because suddenly we can relocate empty cars without paying a driver.


Balancing learning and structure in a neural network In general, how does one balance the two opposing forces of allowing a layer of a neural network to adapt/learn to its training data, while also forcing the neural network to represent some known structure (e.g., representing a family of functions)? The neural network might find the optimal weights, but those weights might no longer make the layer represent the function I originally intended.

For example, suppose I want to create a convolutional layer in the middle of my neural network that is a low-pass filter. In the context of the entire network, however, the layer might cease to be a low-pass filter at the end of training because the backpropagation algorithm found a better optimum. How do I allow the weights to be as optimal as possible, while still maintaining the low-pass characteristics I originally wanted?

Thanks in advance for any help and suggestions. General tips or pointing to specific literature would be much appreciated.

Extending @mirror2image's comment, if you have a certain metric that allows you to measure how close the intended layer is to a low pass filter (something that compares its output with what a low pass filter would have produced, for example), the simplest way to achieve what you want would be to add a term in your loss function that calculates the value of this metric. This way, each time you do a training step, the network now is not only made to output the correct predictions but is also forced to do so while also keeping that specific layer's behavior as close to a low-pass filter as possible.
This is the most common way of tweaking the behavior of neural networks and is often encountered in many research papers.


Why is Lisp such a good language for AI? I've heard before from computer scientists and from researchers in the area of AI that that Lisp is a good language for research and development in artificial intelligence. Does this still apply, with the proliferation of neural networks and deep learning? What was their reasoning for this? What languages are current deep-learning systems currently built in?

First, I guess that you mean Common Lisp (which is a standard language specification, see its HyperSpec) with efficient implementations (à la SBCL). But some recent implementations of Scheme could also be relevant (with good implementations such as Bigloo or Chicken/Scheme). Both Common Lisp and Scheme (and even Clojure) are from the same Lisp family. And as a scripting language driving big data or machine learning applications, Guile might be a useful replacement to Python and is also a Lisp dialect. BTW, I do recommend reading SICP, an excellent introduction to programming using Scheme.

Then, Common Lisp (and other dialects of Lisp) is great for symbolic AI. However, many recent machine learning libraries are coded in more mainstream languages, for example TensorFlow is coded in C++ & Python. Deep learning libraries are mostly coded in C++ or Python or C (and sometimes using OpenCL or Cuda for GPU computing parts).

Common Lisp is great for symbolic artificial intelligence because:

it has very good implementations (e.g. SBCL, which compiles to machine code every expression given to the REPL)
it is homoiconic, so it is easy to deal with programs as data, in particular it is easy to generate [sub-]programs, that is use meta-programming techniques.
it has a Read-Eval-Print Loop to ease interactive programming
it provides a very powerful macro machinery (essentially, you define your own domain specific sublanguage for your problem), much more powerful than in other languages like C.
it mandates a garbage collector (even code can be garbage collected)
it provides many container abstract data types, and can easily handle symbols.
you can code both high-level (dynamically typed) and low-level (more or less startically typed) code, thru appropriate annotations.

However most machine learning & neural network libraries are not coded in CL. Notice that neither neural network nor deep learning is in the symbolic artificial intelligence field. See also this question.

Several symbolic AI systems like Eurisko or CyC have been developed in CL (actually, in some DSL built above CL).

Notice that the programming language might not be very important. In the Artificial General Intelligence research topic, some people work on the idea of a AI system which would generate all its own code (so are designing it with a bootstrapping approach). Then, the code which is generated by such a system can even be generated in low level programming languages like C. See J.Pitrat's blog, which has inspired the RefPerSys project.