Why my Fully Connected Neural Network outputs the same prediction? I have a relatively small data set comprised of $3300$ data points where each data point is a $13$ dimensional vector where the $12$ first dimensions depict a "category" by taking the form of $[0,...,1,...,0]$ where $1$ is in the $i-th$ position for the $i-th$ category and the last dimension is an observation of a continuous variable, so typically one data point would be $[1,...,0,70.05]$.
I'm not trying to have something extremely accurate so I went with a Fully Connected Network with two hidden layers each comprising two neurons, the activation functions are ReLus, one neuron at the output layer because I'm trying to predict one value, and I didn't put any activation function for it. The optimizer is ADAM, the loss is the MSE while the metric is the RMSE.
I get this learning curve below:

Eventhough at the beginning the validation loss is lesser than the training loss (which I don't understand), I think at the end it show no sign of overfitting.
What I don't understand is why my Neural Network predicting the same value as long as the $13-th$ dimension takes values greater than $5$ and that value is $0.9747201$. If the $13-th$ dimension takes for example $4.9$ then the prediction would be $1.0005863$. I thought that it has something to do with the ReLu but even when I switched to Sigmoid, I have this "saturation" effect. The value is different but I still get the same value when I pass a certain threshold.
EDIT: I'd also like to add that I get this issue even with normalizing the 13th dimension (substracting the mean and dividing by the standard deviation).
I'd like to add that all the values in my training and validation set are at least greater than $50$ if that may help.

two hidden layers each comprising two neurons

From your description it looks like that you only have 6 parameters for your inner layer (2x2 weight matrix + 2 biases). The whole network should be easy to interpret: you've got two 13-dimensional weight vectors $\vec{w}_1,\vec{w}_2$ that are dot-multiplied with the inputs, plus two biases $b$ and activation $\sigma$:
$$ l_1 = \sigma\left(\vec{w}_1\vec{x} + b_1\right)$$
$$ l_2 = \sigma\left(\vec{w}_2\vec{x} + b_2\right)$$
Then these two values are multiplied by 2x2 matrix + biases, then activation and linear combination.
I'd look at how $l_1$ and $l_2$ are distributed. The fact that the outputs don't change is most likely due to the first layer getting saturated somehow. Look at the 13th dimension of $\vec{w}_i$ - it is likely to be large compared to other dimensions.
First thing I'd try - standardizing your input 13th dimension, so it is distributed closer to $[0,1]$ ( or $[-1,1]$ ) range.


Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions? Neural networks are commonly used for classification tasks, in fact from this post it seems like that's where they shine brightest.

However, when we want to classify using neural networks, we often have the output layer to take values in $[0,1]$; typically, by taking the last layer to be the sigmoid function $x \mapsto \frac{e^x}{e^x +1}$.

Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions? Is there an analogue to the universal approximation theorem for this case?

As far as I know, the sigmoid is often used as the activation function of the output layer mainly because it is a convenient way of producing an output $p \in [0, 1]$, which can be interpreted as a probability, although that can be misleading or even wrong (if you interpret it as an uncertainty too).
You may require the output of the neural network to be a probability, for example, if you use a cross-entropy loss function, although you could in principle produce only $0$s or $1$s. The probability $p$ can then be used to decide the class (or label) of the input. For example, if $p > \alpha$, then you purposedly decide that the input belongs to class $1$, otherwise, it belongs to class $0$. The parameter $\alpha$ is called the classification (or decision) threshold. The choice of this threshold can actually depend on the problem and it is one of the reasons people use the AUC metric, i.e. to avoid choosing this classification threshold.

Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions? Is there an analogue to the universal approximation theorem for this case?

The most famous universal approximation theorem for neural networks assumes that the activation functions of the units of the only hidden layer are sigmoids, but it does not assume that the output of the network will be squashed to the range $[0, 1]$. To be more precise, the UAT (theorem 2 of Approximation by Superpositions of a Sigmoidal Function, 1989, by G. Cybenko) states

Let $\sigma$ be any continuous sigmoidal function. Then finite sums of the
$$G(x) = \sum_{j=1}^N \alpha_j \sigma (y_j^T x + \theta_j)$$
are dense in $C(I_n)$.
In other words, given any $f \in C(I_n)$ and $\epsilon > 0$, there is a sum, $G(x)$, of the above form, for which
$$|G(x) - f(x)| < \epsilon $$

Here, $f$ is the continuous function that you want to approximate, $G(x)$ is a linear combination of the outputs of $N$ (which should be arbitrarily big) units of the only hidden layer, $I_n$ denotes the $n$-dimensional unit cube, $[0, 1]^n$, $C(I_n)$ denotes the space of continuous functions on $I_n$, $x \in I_n$ (so the assumption is that the input to the neural network is an element of $[0, 1]^n$, i.e. a vector $x \in \mathbb{R}^n$, whose entries are between $0$ and $1$) and $y_j$ and $\theta_j$ are respectively the weights and bias of the $j$ unit. The assumption that $f$ is a real-valued function means that $f$ can take any value on $\mathbb{R}$ (i.e. $f: [0, 1]^n \rightarrow \mathbb{R}$). You should note that $G(x)$ is the output of the neural network, which is a combination (where the coefficients are $\alpha_j$) of the outputs of the units in the only hidden layer, so there's no restriction on the output of $G(x)$, unless you restrict $\alpha_i$ (but, in this theorem, there's no restriction on the values $\alpha_j$ can take).
Of course, if you restrict the output of the neural networks to the range $[0, 1]$, you cannot approximate all continuous functions of the form $f: [0, 1]^n \rightarrow \mathbb{R}$ (because not all of these functions will have the codomain $[0, 1]$)! However, the sigmoid has an inverse function, i.e. the logit, so you can reverse the output of such a neural network. So, in this sense (i.e. by reversing the output of the sigmoid), a neural network with a sigmoid as the activation function of the output layer can potentially approximate any continuous function too.
The UAT above only states the existence of $G(x)$ (i.e. it's an existence theorem). It doesn't tell you how you can find $G(x)$. So, if you use a sigmoid as the activation function of the output layer or not is a little bit orthogonal to the universality of neural networks.


Are there any public real-life code examples of ML applications in Python? Problems I often face at work usually differ from tutorial or book-like examples so I end up with a code that works but it's not elegant and takes too much time to write.

I wanted to ask you if there are some publicly accesible examples or repositories of Python codes that deal with machine learning development and application process but were created in a real company or organisation to develop their real-life products or services?

EDIT: What I do not think about are libraries or packages repositories such as tensorflow. I would like to see some codes of projects that for example use tensorflow to create some other product or service.

The best way is probably to Google it with "[org name] tensorflow github" and look what you get.

For instance I found:

Microsoft

Nvidia

Intel


Finding the optimal combination of inputs which return maximal output I am currently working on a problem and now got stuck to implement one of it's steps. This is a simple attempt to explain what I am currently facing, which is something that I am aiming to implement in my regression simulation in python.

Let's say that I fit a non-linear model to my data. Now, I want to find the combination of inputs within a specified range that returns the the highest outcome. When I am using a quadratic function or only a few inputs, this task is quite simple. However, the problem comes when trying to apply the same logic for more complex models. Supposing that I have 9 variables as inputs, I will have to test all possible combinations and that would be computationally unfeasible by doing it with meshgrid if you want to cover a range with a several intervals in between.

So, here it comes my question, is there such a way to avoid having to go through this computationally costly process in order to achieve the combinations of inputs defined in a given range that return the optimal output?

If your model is gradient-based, such as a neural network, then may also be able to use gradient methods to drive virtual inputs:

Freeze all network weights to the trained version
Define a loss function that decribes how you want the output - or any internal measure - to behave. E.g. to maximise the output, the loss function can simply be the negative of the output, assuming you will perform gradient descent later. Some libraries will also support gradient ascent to maximise a function.
Define your input as a variable that can be optimised and instantiate an optimiser (details will vary depending on your library)
Start with a random or best-guess input, and iterate normal training routine (feed forward then backpropagate) to get better and better inputs

This is basically how Deep Dream and Style Transfer algorithms work - the detail that is different is definition of the loss functions. It is also a way to make adversarial attacks against known models, for example taking a picture of a car, modifying it such that a classifier returns that it is an ostrich (whilst it still looks like a car to a human).

It is not guaranteed to find the absolute best inputs for a given range, but should find good approximations of local minima or maxima far faster than a meshgrid when there are many dimensions. You could combine the idea with a coarse meshgrid or random search for start points for a better chance of finding the best results within input range constraints.

You should bear in mind that the ideal inputs you discover will only be as accurate as your trained model will let them be. If you discover maximising inputs in the model, that use inputs that are far away from any training examples, then in reality those inputs might not get anything like the result that the model predicts. Statistical models with many degrees of freedom are typically ok at interpolating between data points, but very bad at extrapolating beyond data that has been observed.


How to compute the number of centroids for K-means clustering algorithm given minimal distance? I need to cluster my points into unknown number of clusters, given the minimal Euclidean distance R between the two clusters. Any two clusters that are closer than this minimal distance should be merged and treated as one.

I could implement a loop starting from the two clusters and going up until I observe the pair of clusters that are closer to each other than my minimal distance. The upper boundary of the loop is the number of points we need to cluster.

Are there any well known algorithms and approaches estimate the approximate number of centroids from the set of points and required minimal distance between centroids?

I am currently using FAISS under Python, but with the right idea I could also implement in C myself.

Yes, the silhouette method (which is implemented in sklearn as silhouette_score) is commonly used to assess the quality of clusters produced by any clustering algorithm (including $k$-means or any hierarchical clustering algorithm). Roughly, you can compute the silhouette value for different $k$, then you would pick the $k$ with the highest silhouette value.


How do I select the (number of) negative cases, if I'm given a set of positive cases? We were given a list of labeled data (around 100) of known positive cases, i.e. people that have a certain disease, i.e. all these people are labeled with the same class (disease). We also have a much larger amount of data that we can label as negative cases (all patients that are not on the known positive patient's list).
I know who the positives are, but how do I select negative cases to create a labeled dataset of both positives and negatives, on which to first train a neural network, and then test it?
This is a common problem in the medical field, where doctors have lists of patients that are positive, but, in our case, we were not given a specific list of negative cases.
I argued for picking a number that represents the true prevalence of the disease (around 1-2%). However, I was told that this isn't necessary and to do a 50:50 split of positives to negatives. It seems that doing it this way will not generalize outside our test and train datasets.
What would you do in this case?

Short answer
To select the proper dataset to construct, you should first figure out a metric to use to compare, and then select the dataset construction that gives the better metric. There is no single best metric, it depends on the task and your interpretation on what type of error is more important.
If you believe it is important that errors should not be normalized across class, then use the overall accuracy, and keep your dataset distribution same as the natural distribution (so 1-2% positive cases).
If you believe it is important that errors should be normalized across class, then use PR-AUC or ROC-AUC, and re-balance your dataset so that the samples a little more closer to 1:1. The exact ratio will only be determined after testing and comparing the PR-AUC or ROC-AUC metrics.
How to select the best metric?
Two popular metrics are ROC-AUC and PR-AUC. ROC curves (Receiver Operating Characteristics) plot the true positive rate vs false positive rate, while PR curves (Precision and Recall) plot the precision vs recall. AUC stands for "area under curve", because you can achieve any single point in the curve by specifying the classifier threshold, so the sum of all points (i.e. the entire area under the curve) is the most general way of comparing if one model is doing better than another.
Although both ROC curves and PR curves equalize class imbalance at some capacity, PR curves are more sensitive to class imbalance. The paper The Relationship Between Precision-Recall and ROC Curves concludes that if the PR-AUC is good then the ROC-AUC will always also be good, but not the other way around. The difference is due to the fact that if the dataset has huge class imbalance, a false positive hurts PR curves significantly more than ROC curves.
On the other hand, total accuracy does not normalize class imbalance at all, so therefore favors the majority class.
As a result:

if you do not care about normalizing the measures of class imbalance, choose total accuracy, which will optimize for the most # of correct cases (regardless of class)
if you want to normalize your metric across class imbalance, and normalizing false positive errors across classes is at all important to you, choose PR-AUC
if you want to normalize your metric across class imbalance, and don't care about normalizing false positive errors, PR-AUC or ROC-AUC may both be good for you

If it helps, for most imbalance problems, people usually go for PR curves.
By the way, (this paper) studies class imbalance in neural networks by optimizing the ROC curves, and show that you should definitely have equal numbers of positive and negative examples. So if you want the best performance in terms of ROC-AUC, you should do the 50:50 split. I haven't read any similar study that optimizes for PR-AUC, but my intuition tells me that it will have the same conclusion (you should do 50:50 split to optimize for PR-AUC as well).


Semantic Segmentation how to upsampling Many of the architectures that do semantic segmentation like SegNet, DilatedNet (Yu and Koltun), DeepLab, etc. do not work on high resolution images. For such benchmarks like Cityscapes, what is a standard/practical approach for such methods to perform on the benchmark?

I've tried to look into the paper, but I couldn't find such details. There's an article mentioning that they output at 1/8 of input images than do interpolation (usually 2, 4 or 8 times) from their results, but the article does not specify which upsampling techniques are the most reasonable one.

You can always train your network with higher resolution images. There is nothing preventing you to do that if you don't have any restriction about inference time.

Also, the paper is actually mentioning how to upsample. Check for the phrase "Upsample using deconvolutional layers".

The most common ways to upsample are either using the deconvolutional layer or just by simple resize methods (like image resize).


What is the reason for taking tuples as vectors rather than points? Across the literature of artificial intelligence, especially machine learning, it is normal to treat the tuples of datasets as vectors.
Although there is a convention to treat them as data points. Treating them as vectors is also considerable.
It is easy to understand the tuples of datasets as points over space. But what is the purpose of treating them as vectors?

They are equivalent. When we consider a particular instance as a vector, we are not literally imagining it as an arrow with it's head at the point coordinates and tail at the origin. It's just when you are working with a tuple of numbers in a mathematical context, it is conventional to call it a vector. This language follows into machine learning which is usually based on associated linear algebra.


How to design a fitness function for the 8-queens problem? In evolutionary computation and, in particular, in the context of genetic algorithms, there is the concept of a fitness function. The better a state, the greater the value of the fitness function for that state.
What would be a good fitness function for the 8-queens problem?

Here you can find an example of how to apply genetic algorithms to solve the 8-queens problem.
The proposed fitness function is based on the chessboard arrangement, and in particular, it is inversely proportional to the number of clashes amongst attacking positions of queens; thus, a high fitness value implies a low number of clashes.


Is there any board game where a human can still beat an AI? Significant AI vs human board game matches include:

chess: Deep Blue vs Kasparov in 1996,
Go: DeepMind AlphaGo vs Lee Sedol in 2016,

which demonstrated that AI challenged and defeated professional players.

Are there known board games left where a human can still win against an AI? I mean based on the final outcome of authoritative famous matches, where there is still same board game where AI cannot beat a world champion of that game.

Not all games (or even board games) are computationally algorithmic. Even the least skilled player is likely to trounce the hottest pattern-matching algorithm in a game of Pictionary (for example).

If you want to say that the movement of pieces upon successful completion of a task is only ancelary to the object of the game, than your answer will be largely self-selecting. A sufficiently sophisticated algorithm will brute force a computational problem better than human intuition… eventually.


What are advantages of using meta-heuristic algorithms on optimization problems? What are the advantages and disadvantages of using meta-heuristic algorithms on optimization problems? Simply, why do we use meta-heuristic algorithms, like PSO, over traditional mathematical techniques, such as linear, non-linear and dynamic programming?

I actually have a good understanding of meta-heuristic algorithms and I know how they work. For example, one advantage of this kind of algorithms is that they can find an optimal solution in a reasonable time.

However, my lack of knowledge about other methods and techniques brought this question to my mind.

Meta-heuristics are particularly suited for combinatorial optimization problems, given that, although they are not usually guaranteed to find the optimal global solution, they can often find a sufficiently good solution in a decent amount of time. So, they are an alternative to exhaustive search, which would take exponential time. For example, ant colony optimization algorithms have been used to approximately (or exactly, in the case of small or medium-size instances) solve the travelling salesman problem, whose decision version is an NP-complete problem (which means that, unless P=NP, there is no polynomial-time solution to solve it).

Meta-heuristics can also be easily applied to many problems, given that they are not problem-specific. For example, in the case of genetic algorithms, you just need to encode the possible solutions, but, in principle, you can apply genetic algorithms to a wide range of problems, although they may not always be the best solution to each of these problems. Moreover, as opposed to gradient-based optimisation algorithms, there's no need for the gradient of the objective function. For instance, in the case of genetic algorithms, you just need a way of evaluating the solutions (e.g. the fitness or the novelty).

Meta-heuristics often incorporate some form of randomness in order to escape from local minima. Ant-colony optimization algorithms or simulated annealing are two good examples of this approach.

If you are still interested in meta-heuristics, the book Clever Algorithms: Nature-Inspired Programming Recipes (by Jason Brownlee) is a very good resource for learning about them. There's also a Github repository with the implementation of the algorithms described in this book.


Why does my model not improve when training with mini-batch gradient descent, while it does with Adam? I am currently experimenting with the U-Net. I am doing semantic segmentation on the 2018 Data Science Bowl dataset from Kaggle without any data augmentation.
In my experiments, I am trying different hyper-parameters, like using Adam, mini-batch GD (MBGD), and batch normalization. Interestingly, all models with BN and/or Adam improve, while models without BN and with MBGD do not.
How could this be explained? If it is due to the internal covariate shift, the Adam models without BN should not improve either, right?
In the image below is the binary CE (BCE) train loss of my three models where the basic U-Net is blue, the basic U-Net with BN after every convolution is green, and the basic U-Net with Adam instead of MBGD is orange. The learning rate used in all models is 0.0001. I have also used other learning rates with worse results.

Well, some time ago I also faced the same issue in the semantic segmentation task. Batch normalization is expected to improve convergence, because the normalization of activations prevents the explosion of the gradients magnitude and leads to more steady convergence.
Adam is an adaptive optimizer with momentum and division by the weighted sum of gradients on previous iterations squared. https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c.
The loss surfaces of the neural networks is a difficult and poorly understood topic in the present. I suppose, that the poor convergence of SGD is caused by the roughness of loss surface, where the gradient makes big leaps, and jumps over the mimima. The adaptive learing strategy of the Adam, on the other hand, allows to dive into the valleys.


What are the best programming languages for developing evolutionary algorithms? What are the "best" Turing complete programming languages which can be used for developing evolutionary algorithm-based AI programs?

"Best" should be based on pros and cons of performance and easiness for machine learning.

Most machine learning applications today are built on tensors, matrices, probabilistic / Bayesian inference, neural networks, etc. But those can all be built with any modern programming language (all the useful ones are Turing complete). And the best performing language for any of those will generally be assembly / machine code.

Python is famous for machine learning, but that may be due to adoption of Python in academia and NumPy, SciPy, etc. Python isn't very performant, but most of the machine libraries leverage native code, so they're fairly performant.

Julia is a new language that is gunning for a lead position in the data science space, which machine learning builds on. It is allegedly very performant over number crunching domains.

Java has a decent developer ecosystem, and is fairly performant, but the highest performing libraries (including those that leverage GPU) tend to call out to native code via JNI. See DeepLearning4J.

I personally like Clojure - a modern Lisp running on the Java JVM. There's a new deep learning project called Cortex built on Clojure and some fast native libraries, including GPU acceleration.

I think Clojure provides a great balance of being able to easily wrap performant libraries with highly expressive, succinct and simple programming idioms.


Value iteration algorithm from pseudo-code to C++ I am having a difficult time translating this pseudocode into functional C++ code.

At line 10: The value function is represented as V[s], which has a bracket notation like arrays. Is this a separate method or just a function of the value with a given state? Why is S inside the brackets? Is this supposed to be an array with as many elements as S?
At line 12: Vk would be the element in index k inside of array V?
At line 16: I'm interpreting this as the start of a do-while loop that ends at line 20.
Line 19: I'm finding the action that maximizes the sum, for all states, of the equation following the sigma?
Line 20: I'm interpreting this a the end of the do-while. But what is this condition? Am I checking if there is an s such that this condition applies? So Would I would have to loop between all states and stop if any state satisfies the condition? (Basically a loop with a break, instead of a while)

At line 10: The value function is represented as V[s], which has a bracket notation like arrays. Is this a separate method or just a function of the value with a given state? Why is S inside the brackets? Is this supposed to be an array with as many elements as S?

This is just a notation that the value function is a mapping between S and the real numbers. When implementing you would want to store V(s) and π(s) as either arrays or some kind of hashmap like unordered_map (in which case your states must be hashable). Here we also have to assume that this container will fit in the memory, otherwise the value function has to be approximated with function approximation which is not covered explicitly by this pseudo-code.

  At line 12: Vk would be the element in index k inside of array V?

No. This line says that you would need to store multiple value functions during the algorithm, basically a list of functions (or think of it as an array of functions or a two-dimensional array, where k goes along one dimension and s goes along the other dimension) and if you look at the loop following it, it shows you how you need to keep appending new value functions to the end of this list. However notice (in line 19) that you only need the value function from the previous iteration to calculate your new value function, which means that you will never need to store more than two value functions (the new one and the previous one).

  At line 16: I'm interpreting this as the start of a do-while loop that ends at line 20.

Yes. This loop calculates your value function.

  Line 19: I'm finding the action that maximizes the sum, for all states, of the equation following the sigma?

Here you have to iterate through all the actions a in state s and calculate the sum according to the formula in the max function. Then you take the highest value and set the new V(s) to be this value.

  Line 20: I'm interpreting this a the end of the do-while. But what is this condition? Am I checking if there is an s such that this condition applies? So Would I would have to loop between all states and stop if any state satisfies the condition? (Basically a loop with a break, instead of a while)

You have to check for all s if the condition satisfies and you may only break the loop if all these values are lower than the threshold θ.


Why not go another layer deeper with Auto-AutoML? So I'm finding AutoML to be pretty interesting but I'm still learning how it all works. I've played with the incredibly broken AutoKeras and got some decent results.

The question is, if you are using a NN to optimize the architecture of another network, why not take it another layer deeper and use another network to find the optimum architecture for your Parent network with a grand-parent network?

The problem doesn't necessarily need to expand exponentially as the grand-parent network could do few-shot training sessions on the parent network which itself is doing few or one-shot training.

Logically it is possible, but you will just end up complicating the entire task.

The aim of AutoML is to provide a drop in solution to the customers. To do this, a trained network decides and generates the model architecture. This is done so that anyone with basic experience is able to integrate the solution into their systems.

Currently, the complicated architectures and networks require experienced data scientist to build, train and deploy. To overcome this bottleneck and make ML accessible to all, AutoML is being developed.

So adding another grand-parent network to optimize the autoML network will just complicate the task in terms to computation time and hyper parameter optimization.

In case we decide to add another network, now the researchers must look at this network and tune it in regards to the inner network and the model both. This means more work and no direct way to understand how the hyper parameters are affecting the final results.


Which is a better form of regularization: lasso (L1) or ridge (L2)? Given a ridge and a lasso regularizer, which one should be chosen for better performance?

An intuitive graphical explanation (intersection of the elliptical contours of the loss function with the region of constraints) would be helpful.

The following graph shows the constraint region (green), along with contours for Residual sum of squares (red ellipse). These are iso-lines signifying that points on an ellipse have the same RSS.

Figure: Lasso (left) and Ridge (right) Constraints  [Source: Elements of Statistical Learning]
As Ridge regression has a circular constraint ($\beta_1^2 + \beta_2^2 <= d$) with no edges, the intersection will not occur on an axis, signifying that the ridge regression parameters will usually be  non-zero.
On the contrary, the Lasso constraint ($|\beta_1| + |\beta_2| <= d$) has corners at each of the axes, and so the ellipse will often intersect the constraint region at an axis. In 2D, such a scenario would result in one of the parameters to become zero whereas in higher dimensions, more of the parameter estimates may simultaneously reach zero.
This is a disadvantage of ridge regression, wherein the least important predictors never get eliminated, resulting in the final model to contain all predictor variables. For Lasso, the L1 penalty forces some parameters to be exactly equal to zero when $\lambda$ is large. This has a dimensionality reduction effect resulting in sparse models.
In cases where the number of predictors are small, L2 could be chosen over L1 as it constraints the coefficient norm retaining all predictor variables.


Pros and Cons of Seq2Seq vs Bidirectional RNN It seems to me that Seq2Seq models and Bidirectional RNNs try to do the same thing. Is that true? Also, when would you recommend one setup over another? Thanks!

Seq2Seq and Bidirectional RNNs are not doing the same thing, at least in there classic form. Seq2Seq models are used to generate a sequence from another sequence, consider for example tranlation task from one language to another. In that sense Seq2Seq is more a family of models not an architecture, while Bidirectionnal RNNs is a neural network architecture and can be used to build several models including Seq2Seq, for example the encodign part of the Seq2Seq can be a Bi-RNN, but they also can be used for other tasks, for example sentence classfication, sentiment analysis...


Significance of depth of a deep neural network How is a feed-forward neural network with few hidden layers and lots of nodes in those hidden layers different from a network with a lot of hidden layers but relatively lesser nodes in those hidden layers?

Note: The following statements are about feed-forward neural networks. If you're interested in something else, please let me know.

There is a paper which proofs that one hidden layer is enough. So theoretically you could represent any network with many layers with a single hidden layer. I don't remember the title, though. Also I don't remember how it scaled (how many additional nodes you need per added layer).

In practice, it is a very different story anyway:

You need to train the network in an acceptable wall-clock time,
able to fit stuff in memory
You have limitations on maximum time for one inference
You need to find the weights by a training algorithm

Anecdotally, I can say wider networks are usually have more problems with overfitting / memory limitation problems while deeper networks have more problems with maximum inference time.

There is a nice paper from Microsoft where they trained the first super deep network. Many papers followed to that one:

  He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

One main claim was that deeper networks learn more complex features. I don't remember how much of a problem overfitting was. Another important insight is that skip connections are critical for really deep networks (> 20 learning layers).


How can I make the kernels non-learnable and set them manually? I'm a newbie in Convolutional Neural Networks. I have found out that kernels in convolutional layers are usually learned while training.
Suppose I have a kernel that is very good to extract the features that I want to extract. In that case, I don't want the kernels to be learnable. So, how can I make the kernels non-learnable and set them manually?
Maybe, in that case, I have to use something different from a CNN.

In most modern neural network frameworks, the update rules for training can be selectively applied to some parameters and not others.

How to do that is dependent on the framework. Some will have the concept of "freezing" a layer, preventing parameters in it being updated. Keras does this for example. Others will do the opposite and expect you to provide a list of trainable parameters - these typically come with helpers that will list all parameters in a neural network, so you would need to add some kind of filter after collecting that data to exclude your pre-trained layer. PyTorch does this (although the linked example is slightly more complex in that it stops calculating gradients too).

If your framework of choice does not allow you to select and isolate layers in the training process, then you still have a couple of options:

You could store a copy of layer parameters that you want to keep and force your learning network to re-load these parameters after each mini-batch. This does depend on you having a method that can selectively set parameters.
If your pre-trained layers are the first ones, immediately next to the input, then instead of including them in your learning network model, you can pre-process all your training data with just the fixed layers (build a model using only those layers), save the output and use that as an alternative input for the learning layers (build a second model with only the learning layers). Later, once training is complete, you can build a combined neural network out of the fixed layers and the learning layers.


How do I design a fitness function that weighs the importance of eating food? Summary:
I am teaching bots to pick food on a playing field. Some food is poisonous and some is good.
Food Details:

Poisonous food subtracts score points and good food adds.
Food points vary based on its size.
There is about 9:1 ratio of poisonous food to good food, so a lot more chances to end up in negative numbers.
Food grows in points overtime.
Food spoils after some predetermined size becoming poisonous.

Fitness Function:
The fitness function I use is simply counting points by the end of iterations. Bots might choose to eat it or skip it.
The Problem:
The problem I am having is that, in the first generation, most bots eat a lot of bad crap and the curious ones end up in negative numbers. So, mostly the ones that make it are the ones that are lazy and didn't eat or didn't head towards the food and most of the time the fittest for first few generations comes out with 0 points and 0 eats of any kind of food.
When trained for a long time, they just end up waiting for the food instead of eating multiple times. Often, while they wait, food goes bad and they just end up going to another food. This way, at the end of the iteration, I have some winners, but they are nowhere near the potential they could have been at.
Question:
I somehow need to weigh the importance of eating food. I want them to eventually learn to eat.
So I thought of this:
brain.score += foodValue * numTimesTheyAteSoFar

But this blows up the score too much and now the food quality is not respected and they just gulp on anything slightly above 0.

A human analogy can help you here (a variance).
Initialize all the agents with an initial value $x$; we will call this energyUnits. I Will talk later more about this.
Now, add some value, as an incentive, whenever the agent eats good food, to the energyUnits. You need to add a function that will keep decrementing the value of the agent's energyUnits, as humans degrade energy (calories) with time. We will call this function normalDegrade. This is the core part of the solution for your problem.
Now, for the bad (or poisonous) food you can be more creative with. You can simply subtract a given value whenever an agent eats poisonous food. Or you can extend your normalDegrade function with a very high downward slope. In this case, the energy units (value) of the agent will fall very rapidly. This will force the agent to look for good food to survive.
Since the ratio of food is 9:1 with poisonous, you need to initialize the value of $x$ (energyUnits) very high. You need to do some trial and error to find the right fit for you here.
Also, I am assuming that the agent is being removed from the population whenever the value of $x$ is zero or some negative value (which depends). This is important, as it makes sure that the algorithm is not wasting time in processing bad agents.
Because of this, another problem arises of the population coming to extinction. For this, you need to keep generating new agents for which any of the genetic algorithms will do. A new population with better parents of the already present generation will keep the population fit and efficient.
A good fitness function is a core to solving any problem of this kind, and sometimes it is hard to find. You might need to do some trial and error with different values to look for the right fit.


Can dropout layers not influence LSTM training? I am working on a project that requires time-series prediction (regression) and I use LSTM network with first 1D conv layer in Keras/TF-gpu as follows:

model = Sequential()
model.add(Conv1D(filters=60, activation='relu', input_shape=(x_train.shape[1], len(features_used)), kernel_size=5, padding='causal', strides=1))
model.add(CuDNNLSTM(units=128, return_sequences=True))
model.add(CuDNNLSTM(units=128))
model.add(Dense(units=1))

As an effect my model is clearly overfitting:

So I decided to add dropout layers, first I added layers with 0.1, 0.3 and finally 0.5 rate:

model = Sequential()
model.add(Dropout(0.5))
model.add(Conv1D(filters=60, activation='relu', input_shape=(x_train.shape[1], len(features_used)), kernel_size=5, padding='causal', strides=1))
model.add(Dropout(0.5))
model.add(CuDNNLSTM(units=128, return_sequences=True))
model.add(Dropout(0.5))
model.add(CuDNNLSTM(units=128))
model.add(Dense(units=1))

However I think that it has no effect on the network learning process, even though 0.5 is quite large dropout rate:

Is this possible that dropout has little/no effect on a training process of LSTM  or maybe I do something wrong here?

[EDIT] Adding plots of my TS, general and zoomed in view.

I also want to add that the time of training increases just a bit (i.e. from 1540 to 1620 seconds) when I add the dropout layers.

A couple of points:

Have you firstly scaled your data, e.g. using MinMaxScaler? This could be one reason why your loss readings remain high.
Additionally, consider that while Dropout can be useful for reducing overfitting, it is not necessarily a panacea.

Let's take an example of using LSTM to forecast fluctuations in weekly hotel cancellations.

Model without Dropout

# Generate LSTM network
model = tf.keras.Sequential()
model.add(LSTM(4, input_shape=(1, previous)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
history=model.fit(X_train, Y_train, validation_split=0.2, epochs=20, batch_size=1, verbose=2)

Over 20 epochs, the model achieves a validation loss of 0.0267 without Dropout.

Model with Dropout

# Generate LSTM network
model = tf.keras.Sequential()
model.add(LSTM(4, input_shape=(1, previous)))
model.add(Dropout(0.5))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
history=model.fit(X_train, Y_train, validation_split=0.2, epochs=20, batch_size=1, verbose=2)

However, validation loss is slightly higher with Dropout at 0.0428.

Make sure you have specified the loss function correctly. If you are forecasting a time series, then you are most likely working with interval data. Therefore, mean_squared_error is an appropriate loss function as one is trying to estimate the deviation between the predicted and actual values.

As a counterexample, binary_crossentropy would not be suitable as the time series is not a classification set. However, misspecifying the loss function is a common error. Therefore, you also want to make sure you are using the appropriate loss function and then work from there.


How should we regularize an LSTM model? There are five parameters from an LSTM layer for regularization if I am correct.
To deal with overfitting, I would start with

reducing the layers
reducing the hidden units
Applying dropout or regularizers.

There are kernel_regularizer, recurrent_regularizer, bias_regularizer, activity_regularizer, dropout and recurrent_dropout.
They have their definitions on the Keras's website, but can anyone share more experiences on how to reduce overfitting?
And how are these five parameters used? For example, which parameters are most frequently used and what kind of value should be input? ?

One LSTM layer should be enough unless you have lots of data. The same thing goes for the number of nodes in the layer. Start small first so 5 to 10 nodes and increment it until the performance is reasonable.
Once you have a model working you can apply regularization if you think it will improve performance by reducing overfitting of the training data. You can check this by looking at the learning curves or compring the error on the validation and test sets.
In my experiments I've used the L1 and L2 regularizers along with dropout. These can all be mixed together in fact using both L1 and L2 at the same time is called the ElasticNet.
I tend to apply the regularizers on the kernel_regularizer because this affects the weights for the inputs. Basically feature selection.
The value for the L1 and L2 can start with the default (for tensorflow) of 0.01 and change it as you see fit or read what other research papers have done.
Dropout can start at 0.1 then increment it until there is no performance gain. This is basically a percentage so 0.1 would remove about 10% of your nodes.
Finding the best regularizer is the same as any other hyperparameter optimization which is mostly trial and error.


Is AlphaZero an example of an AGI? From DeepMind's research paper on arxiv.org:

  In this paper, we apply a similar but fully generic algorithm, which
  we call AlphaZero, to the games of chess and shogi as well as Go,
  without any additional domain knowledge except the rules of the game,
  demonstrating that a general-purpose reinforcement learning algorithm
  can achieve, tabula rasa, superhuman performance across many
  challenging domains.

Does this mean AlphaZero is an example of AGI (Artificial General Intelligence)?

Good question!

AlphaZero, though a major milestone, is most definitely not an AGI :)

AlphaGo, though strong at the game of Go, is narrowly strong ("strong-narrow AI"), defined as strength in a single problem or type of problem (such as Go and other non-chance, perfect information games.)

AGI, at minimum, must be about as strong as humans in all problems worked on or solved by humans.

AGI is often associated with superintelligence, defined as intelligence that surpasses human levels.

AGI does not necessarily imply super-intelligence, in the sense that we'd consider an android that can perform all human activities with the same capability as humans as an Artificial General Intelligence.

But technically, AlphaGo is a narrow superintelligence in that it exceeds all human performance in a single problem.


Are there any applications of reinforcement learning other than games? Is there a way to teach reinforcement learning in applications other than games?

The only examples I can find on the Internet are of game agents. I understand that VNC's control the input to the games via the reinforcement network. Is it possible to set this up with say a CAD software?

You will see a lot of game examples in reinforcement learning literature, because game environments can often be coded efficiently, and run fast on a single computer that can then contain the environment and the agent. For classic games, such as backgammon, checkers, chess, go, then there are human experts that we can compare results with. Certain games or simplified game-like environments are commonly used to compare different approaches, much like MNIST handwritten digits are used for comparing supervised learning approaches.

  Is there a way to teach reinforcement learning in applications other than games?

Yes. Informally you could apply reinforcement learning approaches whenever you can frame a problem as an agent acting within an environment where it can be informed of the state and a goal-influencing reward value. More formally, reinforcement learning theory is based upon solutions to Markov Decision Processes, so if you can fit your problem description to a MDP then the various techniques used in RL - such as Q-learning, SARSA, REINFORCE - can be applied. This fit to theory does not need to be perfect for the resulting system to work, for instance you can often treat unknown or imperfectly observed state as effectively random to the agent, and consider this part of a stochastic environment.

Here are some examples of possible uses for reinforcement learning outside of recreational games:

Control logic for motorised robot, such as learning to flip pancakes and other examples. Here the environment measurements are made by physical sensors on the robot. The rewards are given for completing a goal, but may also be adjusted for smoothness, economic use of energy etc. The agent chooses low-level actions such as motor torque or relay position. In theory there can be nested agents where higher level ones choose the goals for the lower-level ones - e.g. the robot might decide at a high level between doing one of three tasks that require moving to different locations, and at a lower level might be decisions on how to control motors to move the robot to its chosen goal.
Self-driving cars. Although a lot of focus on sensor interpretation - seeing road markings, pedestrians etc, a control system is required in order to select accelerator, brake and steering.
Automated financial trading. Perhaps a game to some, there are clear real-world consequences. The reward signal is simple enough though, and RL can be adjusted to prefer long or short term gains.

  is it possible to set this up with say a CAD software?

In theory yes, but I do not know what might be available to do this in practice. Also you need one or more goals in mind that you code into the agent (as reward values that it can observe) before giving it a virtual mouse and setting a task to draw something. Computer games come with a reward scheme built in as their scoring system, and provide frequent feedback, so an agent can gain knowledge of good vs bad decisions quickly. You would need to replace this scoring component with something that represents your goals for the CAD-based system.

CAD does not have anything suitable built-in, although CAD tools with simulations, such as various physics engines or finite element analysis, could allow you to score designs based on a simulated physical measure. Other possibilities include analysis of strain, non-wasteful use of material, whatever metrics the CAD/CAM system can provide for a partial or completed design. The tricky part is constraining a design to its goal or purpose and either arranging for that to be rewarded, or building the constraints into the environment; giving an RL agent full unconstrained control of CAD process and rewarding on lowest strain will likely result in something very uninteresting such as a small cube.


is a "word prediction" problem, applicable using HMMs? I'm learning HMMs and decided to model a problem for learning purposes. I came to this idea of word predicting by letters.
here is the model :

while typing, the word is typed letter by letter, so we can consider them as series of observations.
let's say we have just 4 words in our database:

Tap
Trip
Trap
Trigger

and we want to predict the word after 1,2 or 3 written letters.

we have to define states and HMM parameters (state transitions, emissions and priors).

our [hidden ?] states would be :

[ ][ ][ ][ ][ ][ ][ ] : no observations. I chose 7 [ ] because of the longest word
T [ ][ ][ ][ ][ ][ ]
T A [ ][ ][ ][ ][ ]
T R [ ][ ][ ][ ][ ]
… .

and we have to learn the transition probabilities of each state pairs (the pairs which come after each other like T,TR and TA. but not  for TA and TR).

our prior probabilities are 1/3 because we have 3 words. but we may change it by learning which word is used more frequently.

Now, I have these questions :

is HMM suitable for this kind of problem ?
are my assumptions (about states and prior probabilities) correct ?
the states get out of hand when the word count increases. making the model very complex. is that deduction true?
how the emission probabilities are defined in this model ?
what do I miss in case of parameters or definitions ?

I'm a new contributor, be nice to me. regards :)

I think a HMM is overkill for this problem. You kind of have 'hidden' states, but they are very limited and dependent on the full sequence of previous states, which you probably want to avoid to make best use of the HMM's features. It also, as you rightly say, leads to a proliferation in states: each dictionary item adds as many states as it has letters to the model.

The way I would approach this it to use a trie which contains your dictionary, and then you traverse the trie as the user types characters. At each point you have a sub-trie which gives you the possible completions of the word. You could even (if available) augment this with probabilities to guess the most likely word (though this does not take into account the previous word in the sentence, which might be more useful).

If you want to learn about HMMs, one possible application would be a weather forecast (I thought this was an example Lawrence Rabiner uses in his tutorial, but it's actually on the HMM Wikipedia page). Or, if you want to work with texts, use parts-of-speech tagging. You want to have something where each observation can belong to several possible states (eg light can belong to adj, noun, and verb). Here you would have the words as observations and the part-of-speech tags as states, which keeps the model at a reasonably small size (and is thus easier to train).


How to generate the original image from feature set? We all know that using CNN, or even simpler functions, like CLD or EHD, we can generate a set of features out of images.

Is there any ways or approaches that given a set of features, we can somehow generate a corase version of the original image that was given as input? Maybe a gray-scale version with visible objects inside? If so, what features do we need?

The model (that I know of) which most resembles your description is the auto-encoder, which is trained to learn a compact representation (a vector) of the input, which can later be used to reconstruct the original input. In a certain way, this compact representation (implicitly) encodes the most important features of the input. In particular, you may be looking for denoising auto-encoders.


Does it matter if it's a bot or a human generating text? Doesn't it come down to the content? It was noted today that automated text generation is advancing at a rapid pace, potentially accelerating.

As bots become more and more capable of passing turing tests, especially in single iterations, such as social media posts or news blurbs, I have to ask:

Does it matter where a text originates, if the content is strong?

Strength here is used in the sense of meaning.  To elucidate my argument I'll present an example.  (It helps to know the Library of Babel, an infinite memory array where every possible combination of characters exists.)

  An algorithm is set up to produce aphorisms. The overwhelming majority of the output is gibberish, but among the junk is an incredibly profound observation emerges that changes the way people think about a subject or issue.

Where the bot just spams social media, the aphorism in question is identified because it recieves a high number of reposts by humans, who, in this scenario, provide the mechanism for finding the needle (the profound aphorism) in the haystack (the junk output).

Does the value of the insight depend on the cognitive quality of the generator, in the sense of having to understand the statement?

A real world example would be Game 2, Move 37 in the AlphaGo vs. Lee Sedol match.

sometimes voice pitch also matters(and system designers forget about that) - if system is badly designed then you will feel that it is a bot, if design is great than you don't feel it's a bot at all


What is the exact structure within the nodes of a hidden layer? I've been reading on neural networks, but for me, seems like the easiest way for me to learn is seeing some code.  I am curious about what is the exact structure within a node of a hidden layer and the areas of customization?
From what I have gathered, each node receives some inputs (either features of your original data or outputs from the previous layer).  The first step is, we form a linear combination of the inputs with coefficients in the node.  Here's my first few questions:

For a NODE within a layer, if we have p inputs, we would multiply a 1xp coefficient matrix with a px1 input vector, correct?
In practice, for that LAYER, we would multiply a mxp coefficient matrix by a px1 input vector, where m is the number of nodes within that layer?
If we have a bias term, then it would be a mx(p+1) coefficient matrix and a (p+1)x1 input vector?
In terms of customization, for most out of the box NNs, do we choose to have a bias term or not?  Or for the most part, it automatically includes one.
In terms of customization, this combination of inputs is always linear?  There are no other ways to combine the inputs (such as including polynomials, interaction terms, etc...)

After we have generated a scalar via the linear combination, the second step is we feed that value to our "activation function" which is most often a non-linear function like ReLU.  Here obviously the customizable aspect is the choice of "activation" function.

For each node within the same hidden layer, are the activation functions usually the same, or differ among each node?

Let's go part-by-part.

For a NODE within a layer, if we have p inputs, we would multiply a 1xp coefficient matrix with a px1 input vector, correct?

In practice, for that LAYER, we would multiply a mxp coefficient matrix by a px1 input vector, where m is the number of nodes within that layer?

These two are correct. No issues here. The only differences may be in implementation, the logic here is sound.

If we have a bias term, then it would be a mx(p+1) coefficient matrix and a (p+1)x1 input vector?

This is correct. Intuitively, it is analogous to "offest" term in the equation of a line, y = mx + c. c here plays the same role as the bias.

In terms of customization, for most out of the box NNs, do we choose to have a bias term or not? Or for the most part, it automatically includes one.

Most out-of-the box NN include a bias term by default, since they do provide flexibility to the network, no matter how small. Practically, their utility varies with architecture and application. For example, Feed Forward networks have a good boost in performance with biases when used for regression. CNNs used for classification usually aren't affected much if biases are removed.

In terms of customization, this combination of inputs is always linear? There are no other ways to combine the inputs (such as including polynomials, interaction terms, etc...)

Non-linear combinations are definately possible. But using such combinations basically kills the purpose of ANNs - to have simple stacked operations approximate a complex function. Non-linear combinations may help in rare cases, but more often than not, will probably lead to underfit or overfit models. By convention, any non-linearity is introduced into the model on top of the linear combination, as the activation function.

For each node within the same hidden layer, are the activation functions usually the same, or differ among each node?

Conventionally, all nodes in the same layer have the same activation. This makes vectorization easy, and ultimately doesn't really affect preformance. Instead of varying activations of individual neurons, complex architectures use parallel layers having different activations but with the same inputs. Usually, all stacked layers in models share the same activations, except maybe the last layer and in some cases, the penultimate layer.


Is there something like OpenCV for voice recognition and NLP? As you know, OpenCV is a great open-source library for image recognition and related tasks.
Is there a similar library for voice recognition and natural language processing (NPP)?
I know espeak for TTS, also pocketsphinx for voice recognition. Are there better options?

I don't know about voice recognition, but, for NLP, I think that Gensim could be what you are looking for.
Gensim is an NLP package that contains efficient implementations of many well-known functionalities for the tasks of topic modeling, such as tf–idf, Latent Dirichlet allocation, Latent semantic analysis, etc.


Splitting audio consisting of male / female speaker into segments To create language flashcards I would like to split an audio course into many single audio clips. They're basically a man and a woman speaking after each other. The intervals aren't regular so it's not possible to split it after time intervals. Furthermore silence detection is not possible since some sentences also include pauses. I have already tried diarization using LIUM but the timings were completely wrong. Additionally the audio course includes a transcript which would certainly be machine-readable consisting of the English sentence and the Japanese sentence as well as its Romaji version (Japanese words using English letters).

I'm not experienced in AI, so I'm looking for a solution which isn't to difficult (like constructing and training my own neural network).
I have some programming experience, so a mathematical approach would be fine.

Links:

audio course: http://www.japaneseaudiolessons.com/download-japanese-lessons/

LIUM: http://lium3.univ-lemans.fr/diarization/doku.php/welcome

Edit: It's not clear what exactly you're trying to accomplish...  My answer assumes you wanted to split the man and the woman's audio, but re-reading your question make me think otherwise.

Note:  Don't expect this to be a perfect answer.  I'm not an expert in the field, just an interested student.  I can't comment on this site yet, so I'm submitting this as an answer.

This sounds less like a problem suited for machine learning and more of a... statistical analysis(terminology?) problem.  Why? You have an extremely small sample space to train on, and the natural cost function would require you to go through the entire recording and manually classify the track anyway.

Your problem is almost well suited for an Independent Component Analysis approach, except for the fact that you only have one audio source.  ICA requires one source for each feature you want to extract.  It might be a good start though to familiarize yourself with the techniques and terminology.  See the Wikipedia page for ICA, an article called "ICA for dummies" by Arnaud Delorme, and this paper: ICA: a tutorial.  I would link the first two, but apparently I need more site rep.

I also found a paper which seems to perfectly match your problem, from Mitsubishi Electrical Research Labs: Separation of Mixed Sources by Independent Subspace Analysis.  It's a bit a of a heavy read, to say the least, but that might be your best shot.  I'll quote the abstract, emphasis mine:

  We propose the method of independent subspace analysis (ISA) for separating individual audio
  sources from a single-channel mixture. ISA is based on independent component analysis (ICA)
  but relaxes the constraint that requires at least as many mixture observation signals as sources. A
  second extension to ICA is the use of dynamic components to represent non-stationary signals.
  Sources are tracked by similarity of dynamic components over small time steps. We propose
  a method for grouping components by partitioning a matrix of independent component crossentropies
  that we call an ixegram. The ixegram measures the mutual similarities of components
  in an audio segment and clustering the ixegram yields the source subspaces and time trajectories.
  To demonstrate the techniques we give examples of ISA applied to separation of musical and
  speech sources from single-channel mixtures.

Good luck!


What kind of search method is A*? What kind of search method is A*? Explain to me with an example.

A* is an informed search algorithm. A* informed because it is based on the use of a heuristic function, which estimates the distance of each node to the goal, that is, the heuristic function provides information about the distance from any node to the goal node. The heuristic function can e.g. be the Euclidean distance (in the case this can be defined).

More precisely, at each step, A* needs to select the next node to explore. It chooses the node $n$ with the smallest value of $$f(n) = g(n) + h(n),$$ where $g(n)$ is the actual distance from the starting node to node $n$ and $h(n)$ is the heuristic function that estimates the distance from node $n$ to the goal node.

For example, suppose that we want to find the shortest path from Paris to Madrid. If you are already in Madrid, then you can estimate that the distance from Madrid to Madrid is zero, so $h(\text{Madrid}) = 0$. However, if you're still in Paris, what is the estimate of the distance from Paris to Madrid? We can e.g. pick up a map of the world and choose $h(\text{Paris})$ to be the length of the straight segment that goes from Paris to Madrid. Similarly, we can do this for other intermediate cities. There are other ways of estimating this distance, but this is an understandable one, given that people are usually familiar with the Euclidean distance. (This example is a little bit misleading because the Earth is not flat and so the Euclidean geometry does not really apply, but, for simplicity, you can ignore this).

For info regarding the completeness and optimality of A*, have a look at https://ai.stackexchange.com/a/8907/2444.


Detecting Keypoint of 3D model, and distance between them I am very new to AI, I have a set of 3D human models that I would like to train the algorithm to identify wrist, upper arm, lower arms, etc, and distance between them.

From my understanding, this is a regression problem. But with my very limited knowledge, most tutorial online showing me cat and dog classification problem.

Do you have any clue for me to research next? There are some paper saying to convert the 3D model to image, and use convolutional neural network for training.

p/s: Please don't downvote me, I am too young and too lost in this field.

It depends on the format of your 3D model dataset , if your dataset is made of cad models you could voxelize your dataset and train a convolutional neural network on it , but it could be very time consuming to train 3D convnets . instead you could use a multi view 2D CNN https://arxiv.org/abs/1505.00880?context=cs


Is the next state drawn from the joint distribution of the previous state and action? Suppose $G_t$, the discounted return at time $t$ is defined as: $$ G_t \triangleq R_t+\gamma R_{t+1}+\gamma^{2}R_{t+2} + \cdots = \sum_{j=1}^{\infty} \gamma^{k}R_{t+k}$$

where $R_t$ is the reward at time $t$ and $0 < \gamma < 1$ is a discount factor. Let the state-value function $v(s)$ be defined as: $$v_{\pi}(s) \triangleq \mathbb{E}[G_t|S_{t}=s]$$

In other words, it is the expected discounted return given that we start in state $s$ with some policy $\pi$. Then  $$v_{\pi}(s) = \mathbb{E}_{\pi}[R_t+\gamma G_{t+1}|S_{t}=s]$$

$$ = \sum_{a} \pi(a|s) \sum_{s',r} p(r,s'|s,a)[r+\ \gamma v_{\pi}(s')]$$

  Question 1. Are the states $s'$ drawn from a from a joint probability distribution $P_{sa}$? In other words, if you are in an
  initial state $s$, take an action $\pi(s)$, then $s'$ is the random
  state you would end up in according to the probability distribution
  $P_{sa}$?

Also let $q_{\pi}(s,a)$, the action-value function be defined as: $$q_{\pi}(s,a) \triangleq \mathbb{E}_{\pi}[G_t|S_t = s, A_t = a]$$

$$=\sum_{s',r} p(r,s'|s,a)[r+\ \gamma v_{\pi}(s')]$$

  Question 2. What are the advantages of looking at $q_{\pi}(s,a)$ versus $v_{\pi}(s)$?

Question 1. Are the states $s'$ drawn from a from a joint probability distribution $P_{sa}$? In other words, if you are in an
  initial state $s$, take an action $\pi(s)$, then $s'$ is the random
  state you would end up in according to the probability distribution
  $P_{sa}$?

This is tricky, because you don't show a definition of $P_{sa}$. My first thought was that you meant the transition matrix $P_{ss'}^a$, but that doesn't fit with the phrase joint probability distribution.

If you really mean joint probability distribution then the answer is generally "no", because $P_{sa}$ should be the probability of observing state $s$ and action $a$ when taking a random sampled time step:

$$P_{sa} = \pi(a|s)\rho_{\pi}(s)$$

where $\rho_{\pi}(s)$ is the distribution of states under policy $\pi$. Note this makes no reference to $s'$ at all.

However, there are ways that this could relate to the distribution of $s'$. Probably the most direct relationship would be when there is a deterministic environment, thus knowing $s$ and $a$ would determine $s'$. If, in addition to that, each $s'$ could only be reached from a single $(s,a)$ combination, then knowing $P_{sa}$ would also give you knowledge of $P_{s'}$ - this is not the same thing as the question is asking though.

If you did mean the transition matrix $P_{ss'}^a$ instead in the question, then the answer is yes, because

$$P_{ss'}^a = \sum_r p(r,s'|s,a)$$

  Question 2. What are the advantages of looking at $q_{\pi}(s,a)$ versus $v_{\pi}(s)$?

The main advantage is that you can derive a policy more easily from $q_{\pi}$:

$$\pi'(s) = \text{argmax}_a q_{\pi}(s,a)$$

Compare with deriving a policy using $v_{\pi}$:

$$\pi'(s) = \text{argmax}_a \sum_{s',r} p(r, s'|s,a)(r + \gamma v_{\pi}(s'))$$

Note that these policies are not necessarily the same as $\pi$ on which the $q$ or $v$ values are evaluated. In fact this is a common situation whilst searching for an optimal policy, and it is possible to show that $\pi'(s)$ will result in same or higher returns as $\pi(s)$ across all states . . . the proof of this is called the policy improvement theorem.

The important thing about the first equation using $q$ is that it does not involve using the MDP model $p(r, s'|s,a)$ directly. This is the basis of model-free RL. Whilst the version using $v$ is more complex (taking more computation) and requires that you know $p(r,s'|s,a)$.

The main disadvantage of looking at $q_{\pi}(s,a)$ is that it has a larger dimension, it maps $S \times A \rightarrow \mathbb{R}$, as opposed to using $v_{\pi}(s)$ which maps $S \rightarrow \mathbb{R}$. So it can take longer to get good approximations of $q$ compared to $v$.


What is the difference between vanilla policy gradient and advantage actor-critic? What is the difference between vanilla policy gradient (VPG) with a baseline as value function and advantage actor-critic (A2C)?
By vanilla policy gradient I am specifically referring to spinning up's explanation of VPG.

The difference between Vanilla Policy Gradient (VPG) with a baseline as Value function and Advantage Actor Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA:

The value estimates used in updates for VPG are based on full sampled returns, calculated at the end of episodes.

The value estimates used in updates for A2C are based on temporal difference bootstrapped from e.g. a single step difference, and the Bellman function.

This leads to following practical differences:

A2C can learn during an episode which can lead to faster refinements in policy than with VPG.

A2C can learn in continuing environments, whilst VPG cannot.

A2C relies on initially biased value estimates, so can take more tuning to find hyperparameters for the agent that allow for stable learning. Whilst VPG typically has higher variance and can require more samples to achieve the same degree of learning.


In which scenario would you want to have two adjacent pooling layers? In which scenario, when assembling a CNN, would you want to have two adjacent pooling layers, without a convolutional layer in between?

Assuming you're not referring to any particular type of pooling operation, it's possible that you could have, for example, a mean pool followed by a max or min pool. What this could do is combine the idea of reducing the dimensionality of your data from a holistic perspective with the mean pool, and then choosing the best of your averages with your max pool.


Are Bayesian networks important to learn in 2018? I study AI by myself with the book "Artificial Intelligence: A Modern Approach". I've just finished the chapters about the Bayesian network and probabilities, and I found them very interesting. Now, I want to implement different algorithms and test them in different cases and environments.

Is it worth it to spend time on these techniques?

*AI, A Modern Approach," was given that title to break from previously narrow approaches to duplicating desirable qualities of human thinking.

Although Bayesian networks require somewhat resource intensive computational elements, the importance of Bayesian inference and probability are still of paramount importance in that some of the highest scientific thinking require mastery of them.  Furthermore, developing in silicon dies (or possibly graphene nanites) the machinery to perform elementary probability computations in massively parallel architectures may arise over the next few decades.  The use of video DSP circuits to implement ANNs is a notable segue into this kind of development.

I would not dismiss the techniques you just read about.  If your intention is to capitalize on the recent crazes, you may enter the river of wannabees chasing every current trend, implement many systems that other people will rewrite later, and have a meaningless yet profitable career.  I would recommend following your inquisitiveness instead.


Which edges of this tree will be pruned by Alpha-beta pruning? So I know that 'h' and 'f' will be pruned, but I'm not sure about 'k' and 'l'.

When we visit 'j', technically there is no need for us to visit 'k' and 'l' because there are 2 options:

one or two of them might be higher than 8 ('j')
both of them less than 8

But no matter what, the decision of the max(root) will not change, the max will choose the right side no matter what 'k' and 'l' are, because the right side will either be 8 or 9, which is still higher than 4 (returned value from left side)

so will alpha beta prune 'k' and 'l' or not? if not, then it means alpha beta is not "optimal" overall right? considering it will not prune all the unnecessary paths.

If you prune k and L then you could miss the optimal solution. Assume L=9, if you prune L then the value of the tree is 8. If you don't prune L then the value of the tree is 9. Now I will try and address what I think your actual question is

  But no matter what, the decision of the max(root) will not change, the max will choose the right side no matter what 'k' and 'l' are, because the right side will either be 8 or 9, which is still higher than 4 (returned value from left side).

From this sentence it seems like you don't care about the value of the tree, you only want to find the optimal first move based on alpha beta. You are correct in saying that the correct first move will always contain the right most child of the root, but oftentimes that is not the only information we want. Sometimes we want to know the value of the tree as well or what the complete correct path is, but if we had pruned k and L we would not know these.

Edit: I have changed all 'L's to uppercase, because lower case 'L' looks to much like uppercase 'i'


Looking to build, compile, and/or find dataset for serial-parallelized code examples I'm looking to perform two tasks:

Train a classifier to classify code as serial or parallel
Train a generative algorithm to generate parallel code from serial

For the first task a simple scraper can scrape random C and C++ code from git, however for the second step I would need a decently large source of examples of serial to parallel code. Any ideas or pointers for existing or creating this type of dataset would be greatly appreciated.

A generic way to start with under such circumstances is to try find an "oracle".

Serial to parallel converters exist for quite some times, and some are open source (e.g. PIPS). The idea is to get serial code from step (1), use the "oracle" to produce parallel code, and that's it: Each conversion makes en entry in the dataset.

Ensuring the quality of the dataset is critical here. A script generating the dataset shoud ensure (1) the serial code compiles and runs properlly, (2) the parallelized code compiles and runs properly too, (3) serial and parallel programs produce the same result, (4) some metrics state objectively how the parallel version does against the serial version, and (5) keep track of the actual hardware configuration.

Point (4) is critical to the quality of such a dataset. Parallel programs are not always faster than a serial version: A 1000-iteration loop dumped over 1000 workers on a 8-core CPU may not do so well compared to 8 workers---we need to check what the converter is doing. And point (5) ensures we know under which conditions the data is valid.

Using several "oracles" would be even better, to bring diversity, and to hopefully get the learning algorithm discover the best conversion tradeoffs---perhaps better than what the carefully hand-crafted converters are able to do on their own.


Why are Neural Networks considered Artificially Intelligent? Why in every aspect are we now considering Neural Networks as an Artificially Intelligent entity/program?

Why in every aspect we are now considering artificial intelligence as a neural network?

"We" aren't. It is generally due to reporting by media sources that simplify science and technology news.

The definition of AI is somewhat fluid, and also contentious at times, but in research and scientific circles it has not changed to the degree that AI=NN.

What has happened is that research into neural networks has produced some real advances in the last decade. These advances have taken research-only issues such as very basic computer vision, and made them good enough to turn into technology products that can be used in the real world on commodity hardware.

These are game-changing technology advances, and they use neural networks internally. Research and development using neural networks is still turning out new and improved ideas, so has become a very popular area to learn.

A lot of research using neural networks is also research into AI. Aspects such as computer vision, natural language processing, control of autonomous agents are generally considered parts of AI. This has been simplified in reporting, and used by hyped-up marketing campaigns to label pretty much any product with a NN in it as "Artificial Intelligence". When often it is more correctly statistics or Data Science. Data Science is another term which has been somewhat abused by media and technology companies - the main difference between use of AI and Data Science is that Data Science was a new term, so did not clash with pre-existing uses of it.

The rest of AI as a subject and area of study has not gone away. Some of it may well use neural networks as part of a toolkit to build or study things. But not all of it, and even with the use of NNs, the AI part is not necessarily the neural network.


Why does a Lipschitz continuous discriminator in GANs assure statistical boundedness? I have been reading the paper which introduced spectral normalization in GANs.

At some point the paper mentions the following:

  The machine learning community has been pointing out recently that the
  function space from which the discriminators are selected crucially
  affects the performance of GANs. A number of works (Uehara et al.,
  2016; Qi, 2017; Gulrajani et al., 2017) advocate the importance of
  Lipschitz continuity in assuring the boundedness of statistics.

What does it mean that the Lipschitz continuity assures the boundedness of statistics and why does that happen?

To put it simply GANs suffer from a problem of uneven learning rate. Imagine the learning rate of a pitcher and hitter if the pitcher gets to a point where they can throw much better than the hitter can hit then the hitter may fall into a 'training pit' as to be unable to ever learn how to hit from the pitcher.

This follows a continues relationship in between the two learning rates where if the pitcher is becoming a much better pitcher at a faster rate they could become too good and make learning impossible for the hitter. So the rate must be 'slowed down' as to ensure the pitcher doesn't ruin the hitter.

If the cone of the Lipschitz continuity function of either function is outrunning/outpacing the other than the learning for the one who is in front must be slowed down so the other catches up.

Two runners trying to push each other athletically is another example. If one outpaces the other an injury may occur in the one lagging behind while trying to keep pace this happens, when the adversarial network becomes too good at generating training material that the behind network is not ready to learn with.

The GAN will do best when the learning rates are adjusted to slow down the fast learner artificially.

The statistics will not be bounded correctly if the learning rates are not kept in check similarly to how step size needs to be right to find local minimum and maximum. If the learning is not artificial augmented so both keep relatively same pace getting stuck at local minimum and maximum of the solution space will occur.


What is the current research in artificial intelligence in the field of data compression? What is the current research in artificial intelligence and machine learning in the field of data compression?
I have done my research on the PAQ series of compressors, some of which use neural networks for context mixing.

It is already combined.

Adaptive entropy techniques are already used in most of the best compression encoders.  This is true for file encoders, video encoders, and audio encoders.  We use it in the solar lab to optimize sample rates in data acquisition.

In fact, pattern recognition and compression are very tightly coupled if you consider autoencoders and other feature extraction schemes and compare them mathematically with what compression does.  See Data Compression – A Generic Principle of Pattern Recognition?, Gunther Heidemann, Helge Ritter, VISIGRAPP 2008

Zlib and lz4 have more like hyper-parametric learning, however they don't persist what they learn.  This work is interesting: Adaptive On-the-Fly Compression, Chandra Krintz, Sezgin Sucu, IEEE Parallel and Distributed Systems, v17 n1, January 2006.

**Suggested Project:

Create a theoretical framework and software POC that learns correlations between these two sets.

Quickly ascertainable features of documents or audio or video streams (i.e file path components, media titles, date, file type, and first N bytes)
The existing parameters that existing open source compression software learns during its existing pattern recognition algorithms

Persisting those correlations between compression invocations may considerably improve file transfer, kernel operations (since lz4 is now native in kernels like LINUX), and media streaming.

How much effort is made to persist features extracted (pattern recognition) between frames in media streaming is worth investigating too.


Is AI already being used in the drug industry to combat the COVID-19? We all have heard about how beneficial AI can be in health. There are plenty of papers and research about confronting against diseases, like cancer. However in 2020 with COVID-19 be one of the most serious health problems that has caused thousands of deaths worldwide,

Is AI already being used in the drug industry to combat the COVID-19? If yes, can you, please, provide a reference?

A global race is underway to discover a vaccine, drug, or combination of treatments that can disrupt the SARS-CoV-2 virus.

The problem is, there are more than a billion such molecules. A researcher would conceivably want to test each one against the two dozen or so proteins in SARS-CoV-2 to see their effects. Such a project could use every wet lab in the world and still not be completed for centuries.

Computer modelling is a common approach used by academic researchers and pharmaceutical companies as a preliminary, filtering step in drug discovery. However, in this case, even every supercomputer on Earth could not test those billion molecules in a reasonable amount of time.

Folding@home is a distributed computing project run by Stanford University. The aim of the project is to examine how proteins fold and it does this using spare computing power. however, there is a lot of research in progress that are harnessing the potential of Artificial intelligence to develop the potential treatment to combat the COVID-19.

Check this recent article By Tyler Orton in biv focus on how artificial intelligence is used to accelerate the process of Drug Discovery:
Drug research turns to artificial intelligence in COVID-19 fight

Here are the list some companies that are using AI-driven approach for Drug discovery

BlackThorn Therapeutics
xscientia
Insilico Medicine
Insitro
Notable Labs
Standigm
Recursion Pharmaceuticals

The Hong Kong-based company Insilico Medicine, a developer of a comprehensive drug discovery and biomarker development platform GENTRL, and a pioneer in the application of generative adversarial networks (GANs) to drug discovery.

Insilico Medicine, Publishes a paper in September last year titled,
Deep learning enables rapid identification of potent DDR1 kinase inhibitors," in a most reputed journal Nature Biotechnology. The paper describes a timed challenge, where the new artificial intelligence system called Generative Tensorial Reinforcement Learning (GENTRL) designed six novel inhibitors of DDR1, a kinase target implicated in fibrosis and other diseases, in 21 days.

Four compounds were active in biochemical assays, and two were validated in cell-based assays. One lead candidate was tested and demonstrated favourable pharmacokinetics in mice.


How to understand 'losses' in Spacy's custom NER training engine? From the tid-bits, I understand of neural networks (NN), the Loss function is the difference between predicted output and expected output of the NN. I am following this tutorial, the losses are included at line #81 in the nlp.update() function.
I am getting losses in the range 300-100. How to interpret them? What should be the ideal output of this losses variable? I went through Spacy's documentation, but nothing much is written there about losses. Also, please let me know the links to relevant theories to understand this in general.

A critical goal of training a neural network is to minimize the loss.  Loss is not explained for spaCy because it is a general concept for machine learning and deep learning.  Loss is not specific to spaCy and although there are some finer details I don't believe that is your inquiry.
In general, to understand loss functions, I recommend the following resources:

Loss and Loss Functions for Training Deep Learning Neural Networks by Jason Brownlee.  Jason writes some great articles and tutorials and I highly suggest reading his other articles and books.
Introduction to Loss Functions
Loss Functions in Machine Learning for Beginners by John Kaller

If you like videos watch:

Loss Functions Explained by Siraj Raval.  Siraj has many great videos on machine learning.


Why MLP momentum term must be in the range 0-1? Why momentum factor greater than 1 is a bad idea?

What are the mathematical conclusions?

If gradient descent is like walking down a slope, momentum would be the literal momentum of the agent traversing the hyperplane.

Under that analogy then, momentum factor would be analogous to the friction coefficient, with 1 being max friction and 0 being no friction.

You should be able to see why there can't be friction beyond that range: if friction = 1 it would be identical to having no friction; if friction <= 0 then by conservation of energy gradient descent will not find a local minima; if friction > 1 then gradient descent would be moving backwards.


Can one use an Artificial Neural Network to determine the size of an object in a photograph? My question relates to but doesn't duplicate a question that has been asked here.

I've Googled a lot for an answer to the question: Can you find the dimensions of an object in a photo if you don't know the distance between the lens and the object, and there are no "scales" in the image?

The overwhelming answer to this has been "no". This is, from my understanding, due to the fact that, in order to solve this problem with this equation,

$$Distance\ to\ object(mm) = \frac{f(mm) * real\ height(mm) * image\ height(pixels)}{object\ height(pixels) * sensor\ height(mm)}   $$

you will need to know either the "real height" or the "distance to object". It's the age old issue of "two unknowns, one equation". That's unsolvable. A way around this is to place an object in the photo with a known dimension in the same plane as the unknown object, find the distance to this object and use that distance to calculate the size of the unknown (this relates to answer from the question I linked above). This is an equivalent of putting a ruler in the photo and it's a fine way to solve this problem easily.

This is where my question remains unanswered. What if there is no ruler? What if you want to find a way to solve the unsolvable problem? Can we train an Artificial Neural Network to approximate the value of the real height without the value of the object distance or use of a scale? Is there a way to leverage the unexpected solutions we can get from AI to solve a problem that is seemingly unsolvable?

Here is an example to solidify the nature of my question:

I would like to make an application where someone can pull out their phone, take a photo of a hail stone against the ground at a distance of ~1-3 ft, and have the application give them the hail stone dimensions. My project leader wants to make the application accessible, which means he doesn't want to force users to carry around a quarter or a special object of known dimensions to use as a scale.

In order to avoid the use of a scale, would it be possible to use all of the EXIF meta-data from these photos to train a neural network to approximate the size of the hail stone within a reasonable error tolerance? For some reason, I have it in my head that if there are enough relevant variables, we can design an ANN that can pick out some pattern to this problem that we humans are just unable to identify. Does anyone know if this is possible? If so, is there a deep learning model that can best suit this problem? If not, please put me out of my misery and tell me why it it's impossible.

Can one use an Artificial Neural Network to determine the size of an object in a photograph?

Yes: Learning Depth from Single Monocular Images

In the end, depth is just one special form of size.

Of course, you need something partially known, e.g. another car. You don't need to know the exact size of the car, but you know which size cars in general have. If you have an image without any reference, it is impossible.


Text classification of non-equal length texts, should I pad left or right? Text classification of equal length texts works without padding, but in reality, practically, texts never have the same length.
For example, spam filtering on blog article:
thanks for sharing    [3 tokens] --> 0 (Not spam)
this article is great [4 tokens] --> 0 (Not spam)
here's <URL>          [2 tokens] --> 1 (Spam)

Should I pad the texts on the right:
thanks for     sharing --
this   article is      great
here's URL     --      --

Or, pad on the left:
--   thanks  for    sharing
this article is     great
--   --      here's URL

What are the pros and cons of either pad left or right?

For any model that does not take a time series approach like an RNN does, the padding shouldn't make a difference.
I prefer padding right simply because there also might be text you need to cut-off. Then padding is more intuitive as you either cut-off a text if it's too long or pad a text when it's too short.
Either way, when a model is trained a certain way, it shouldn't make a difference so long the testing is also padded the way it was presented in training.


What's the difference between a static AI and a dynamic AI? I recently watched a YouTube video (sorry, can't remember the link) where (a very talented) someone created what they called a "static AI".

Somewhere in the video they said something along the lines of:

  "this is a static AI, it's very simple and not dynamic at all"

What does this mean? What's the difference between a static AI and a dynamic AI?

My sense is they're probably talking about the difference between symbolic AI (aka GOFAI)  and learning algorithms.

GOFAI typically uses heuristics, which are static—essentially fixed rules governing decision making.

Statistical AI, aka Machine Learning, is dynamic and can analyze and evaluate the environment to form its own decision rules.

Sometimes a trained NN that is no longer learning is said to have created it's own heuristics.


Is the Q value updated at every episode? I trying to understand the Bellman equation for updating the Q table values. The concept of initially updating the value is clear to me. What is unclear is the subsequent updates to the value. Is the value replaced with each episode?  It doesn't seem like this would learn from the past.  Maybe average the value from the previous episode with the existing value?

Not specifically from the book. I'm using the equation

$$V(s) = \max_a(R(s, a) + \gamma V(s')),$$

where $\gamma$ is the learning rate. $0.9$ will encourage exploration, and $0.99$ will encourage exploitation.  I'm working with a simple $3 \times 4$ matrix from YouTube

I think you are a bit confused about what is the update function and the target.

The equation you have there, and what is done in the video is the estimation of the true value of a certain state. In Temporal-Difference algorithms this is called the TD-Target.

The reason for your confusion might be that in the video he starts from the end state and goes backwards using that formula to get the final value of each state. But that is not how you update the values, that is where you want to get to at the end of iterating through the states.

The update formula may have several forms depending on the algorithm. For TD(0), which is  a simple 1-step look ahead off-policy where what is being evaluated is the state (as in your case), the update function is:

$$
V(s) = (1 - \alpha) * V(s) + \alpha * (R(s,a) + \gamma V(s')),
$$
where alpha is the learning rate. What alpha will do is balance how much of your current estimate you want to change. You keep $1 - \alpha$ of the original value and add $\alpha$ times the td-target, which uses the reward for the current state plus the discounted estimate of the value of the next state. Normal values for alpha can be 0.1 to 0.3, for example.

The estimate will slowly converge into the real value of the state which is given by your equation:
$$
V(s) = \max_a(R(s, a) + \gamma V(s')).
$$

Also, the $\gamma$ is actually the discount associated with future states, as it is said in the video you referenced. It basically says how much importance you give to future states rewards. If $\gamma = 0$, then you only care about the reward in your current state to evaluate it (this is not what is used). On the other extreme if $\gamma = 1$ you will give as much value for a reward received in a state 5 steps ahead as you will to the current state. If you use some intermediate value you will give some importance to future rewards, but not as much as for the present one. The decay on the reward received on a state $n steps$ in the future is given by $\gamma^n$.

Another thing that I would correct is that the exploration - exploitation balance is not in any way related to $\gamma$. It is normally balanced by some policy, for example $\epsilon - greedy$. This one for example says that a certain % of the actions you take are random, which in turn makes you explore less valued states.


What are the best known gradient-free training methods for deep learning? As I know, the current state of the art methods for training deep learning networks are variants of gradient descent or stochastic gradient descent.

What are the best known gradient-free training methods for deep learning (mostly in visual tasks context)?

There are several different algorithms that can be used for gradient free neural network training. Some of these algorithms include particle swarm optimization, genetic algorithms, simulated annealing, and several others. Almost any optimization algorithm can be used to train a neural network. Here is an overview of some of the algorithms I listed:

Particle Swarm optimization - I would say that this is one of the better optimization algorithms to train neural networks other than back propagation. I am currently using it and have achieved quite good results.
Genetic Algorithms - I have tried to use genetic algorithms to train neural networks in the past and I was not able to get it to work. However, I was using deep neural networks with almost a million parameters and the performance was not that good.
Simulated annealing - simulated annealing is based off of metals cooling. I have seen simulated annealing work fairly well but maybe not as well as particle swarm optimization.
Derivatives of genetic algorithms - derivatives of genetic algorithms such as NEAT have been shown to work pretty well. I have not personally used them extensively but some of the things that people have used them for are pretty cool.


How is parallelism implemented in RL algorithms like PPO? There are multiple ways to implement parallelism in reinforcement learning. One is to use parallel workers running in their own environments to collect data in parallel, instead of using replay memory buffers (this is how A3C works, for example).

However, there are methods, like PPO, that use batch training on purpose. How is parallelism usually implemented for algorithms that still use batch training?

Are gradients accumulated over parallel workers and the combined? Is there another way? What are the benefits of doing parallelism one way over another?

OpenAI have a post on that: https://openai.com/blog/openai-five/

They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed for batches of 4096 observations which are then averaged.

PPO is actually designed to allow this kind of parallelisation as it uses trajectory segments with a fixed size of $T$ to collect data, e.g. 60 seconds for OpenAI Five, where $T$ is supposed to be "much less than the episode length" (p.5 of PPO paper).


Will BERT embedding be always same for a given document when used as a feature extractor When we use BERT embeddings for a classification task, would we get different embeddings every time we pass the same text through the BERT architecture? If yes, is it the right way to use the embeddings as features? Ideally, while using any feature extraction technique, features values should be consistent. How do I handle this if we want BERT to be used as a feature extractor?

BERT is deterministic. There is no variation unless you parse your tokens differently in succeeding runs. Here is the original paper the model architecture is based off of Transformer Paper. Note that in every layer, the only operations used for the most part are matrix multiplications, concatenations, basic ops, and layer normalizations, all of which are deterministic.