Are these two TRPO objective functions equivalent? In the TRPO paper, the objective to maximize is (equation 14)
$$
\mathbb{E}_{s\sim\rho_{\theta_\text{old}},a\sim q}\left[\frac{\pi_\theta(a|s)}{q(a|s)} Q_{\theta_\text{old}}(s,a) \right]
$$

which involves an expectation over states sampled with some density $\rho$, itself defined as
$$
\rho_\pi(s) = P(s_0 = s)+\gamma P(s_1=s) + \gamma^2 P(s_2=s) + \dots
$$

This seems to suggest that later timesteps should be sampled less often than earlier timesteps, or equivalently sampling states uniformly in trajectories but adding an importance sampling term $\gamma^t$.

However, the usual implementations simply use batches made of truncated or concatenated trajectories, without any reference to the location of the timesteps in the trajectory.

This is similar to what can be seen in the PPO paper, which transforms the above objective into (equation 3)
$$
\mathbb{E}_t \left[ \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_\text{old}}(a_t|s_t)} \hat A_t \right]
$$

It seems that something is missing in going from $\mathbb{E}_{s\sim \rho}$ to $\mathbb{E}_t$ in the discounted setting.
Are they really equivalent?

As you point out, they are not equivalent. I guess you could store the time index for each state visited, but there are two problems with this.

First, if you sample states according to their time index, sampling from the replay memory will become more cumbersome and probably much slower (you'd have to sample the time index and then a specific state with that time index). This is definitely something undesirable. If you choose to add the importance sampling term, then you will make really small most of the terms if $\gamma$ is not so close to 1.

Second, while the original objective is very nice to obtain theoretical results, you may ask yourself if that objective is really what you want to maximize. Do you really care more about the performance at the beginning of the trajectory than at the end?

While I have no rigorous proofs, I like to think that a better definition for the expected discounted reward is a time average:

$$\eta(\pi) = \lim_{N\to\infty}\frac{1}{N}\sum_{k=1}^N\mathbb E_{s_k,a_k,s_{k+1},\cdots}\left[\sum_{t=k}^\infty\gamma^{t-k}r(s_t)\right].$$

Since each term of the average satisfies the original proofs, the only difference with the equation 14 is that the density is $\tilde\rho_{\theta_{old}}(s) = \lim_{N\to\infty}\frac{1}{N}\sum_{k=1}^N\rho_{k,\theta_{old}}(s)$, where $$\rho_{k,\theta_{old}} = P(s_k = s)+\gamma P(s_{k+1}=s) + \gamma^2 P(s_{k+2}=s) + \dots.$$
You can notice that for a large $N$ you basically give the same importance to all of the time-steps. So, the usual implementations actually optimize some complicated function similar to the one I defined, where you give a similar weight to all samples, irrespective of their corresponding time-step.


Is it possible to prove that the target policy is better than the behavioural policy based on learned Q values? I have retrospective data for a sort of "behaviour policy" which I will use to train a deep q network to learn a target greedy policy. After learning the Q values for this target policy, can we make the conclusion that because the Q value for the target policy, $Q(s,\pi_e(s))$ is higher than the Q values for the behaviour policy, $Q(s,\pi_b(s))$ at all states encountered, where $\pi_e$ is the policy output by deep Q-learning and $\pi_b$ is the behaviour policy, then this target policy has better performance than the behaviour policy?

I know the proper way is to run the policy and do an empirical comparison of some sort. However, that is not possible in my case.

No, mainly because these are all stochastic approximations and may not represent the true values.

Almost nothing good can be said about NN approximations to value and Q functions(at least according to a professor I have had).


How to define a reward function in POMDPs? How do I define a reward function for my POMDP model?

In the literature, it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this way you have to do define a reward for every possible action-state combination. I think that the examples in the literature might not be practical in reality, but only for the purpose of explanation.

There is no major difference here between a POMDP and MDP. When setting reward values, you are generally trying to give the minimal information to the agent that when the sum of rewards is maximised, it solves the problem that you are posing.

  In literature it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this way you have do define a reward for every possible Action-State combination.

Some defined value of reward has to be returned after all state, action pairs taken in the environment. The value could be $0$ of course.

Reward can depend on, or be a function of, current state, action, next state and a random factor. In a POMDP it may also be from any unobserved factor in the environment (you might know this in a simulation because you have created the environment and are choosing not to share the data with the agent).

In practice, the reward often does not have to depend on all the possible factors. In addition, the relationship between the factors and the possible reward can be very simple or sparse.

Classic examples you may find in the literature include:

Reward in a game can be simply $+1$ for winning, or $-1$ for losing, granted at the end. All other rewards are $0$
If your goal is to reach a certain state, such as exit from a maze, in minimum time, then a fixed reward of $-1$ per time step is enough to express the need to minimize the total number of steps.
For a goal of maintaining stability and avoiding a failure in e.g. CartPole, then a reward of +1 per time step without failure suffices.

All of these examples express the reward function as a simple condition plus one or two numbers. The key thing is that they allow you to calculate a suitable reward on each and every time step. They are all strictly reward functions - the general case can cover very complex functions if you wish, that will depend entirely on the goals you want to set the agent to learn/solve.


How can I derive the rotation matrix from the axis-angle rotation vector? Given an axis-angle rotation vector $\Theta = (2,2,0)$, after finding the unit vector $k=(1/\sqrt{2}, 1/\sqrt{2}, 0)$ and angle $\theta = 2\sqrt{2}$ representing the same rotation, I need to derive the rotation matrix $R$ representing the same rotation and to show, that the matrix is orthonormal. How can I do that?

The rotation matrix $R_k(\theta)$ associated with a given unit-length vector $k$ and angle $\theta$ is given by the following formula

$$\small{R_k(\theta) = \begin{bmatrix}
\cos \theta +k_x^2 \left(1-\cos \theta\right) & k_x k_y \left(1-\cos \theta\right) - k_z \sin \theta & k_x k_z \left(1-\cos \theta\right) + k_y \sin \theta \\
k_y k_x \left(1-\cos \theta\right) + k_z \sin \theta & \cos \theta + k_y^2\left(1-\cos \theta\right) & k_y k_z \left(1-\cos \theta\right) - k_x \sin \theta \\
k_z k_x \left(1-\cos \theta\right) - k_y \sin \theta & k_z k_y \left(1-\cos \theta\right) + k_x \sin \theta & \cos \theta + k_z^2\left(1-\cos \theta\right)
\end{bmatrix}}$$

So, to find your specific rotation matrix, you just need to substitute the values of your $k$ and $\theta$ in the above matrix.

The derivation of this matrix can be found in section 9.2 Rotation Matrix Derivation of the PhD thesis Modelling CPV (2015), by Ian R. Cole. The basic idea of the derivation follows the following steps

Rotate the given axis $k$ and the point $p$ (that you want to rotate) such that the axis $k$ lies in one of the coordinate planes: xy, yz or zx
Rotate the given axis $k$ and the point $p$ (that you want to rotate) such that the axis $k$ is aligned with one of the two coordinate axes for that particular coordinate plane: $x$, $y$ or $z$
Use one of the fundamental rotation matrix to rotate the point $p$ depending on the coordinate axis with which the rotation axis is aligned
Reverse rotate the axis-point pair such that it attains the final configuration as that was in step 2 (that is, you have to undo step 2)
Reverse rotate the axis-point pair which was done in step 1 (that is, you have to undo step 1)

To show that a matrix is orthonormal, you need to show that it is orthogonal (each row is independent of any other row and each column is independent of any other column) and that the length of each row (and column) is 1. Equivalently, a square matrix $Q$ is orthogonal if and only if

$$
Q^TQ = QQ^T = I
$$

where $Q^T$ is the transpose of $Q$ and $I$ is the identity matrix. If you are really stuck, have a look at this proof https://math.stackexchange.com/a/537248/168764 (and this other answer https://math.stackexchange.com/a/156742/168764), but, at this point, it should just be a matter of replacing $R_k(\theta)$ in the equation above and check that the equation holds.


I want to determine how similar a given song is to Queen's songs. Am I headed in the right direction? I've asked this question before (@ Reddit) and people suggested CNNs on a mel spectrogram more than anything else. This is great.
But I'm sort of stuck at: label some music data as "queen" and "not queen" and have this be the training set. Like, download 300 songs, 70 queen (that's all they have) and 230 not queen, create their mel spectrograms using some python package that can do that.
First of all, is 300 songs even enough?
I only have a basic understanding of what I'm doing. I need some help

You are heading in the right direction to make an audio-based classifier. This is not quite the same as providing a similarity metric between two pieces of audio, but it may do as a first attempt. You could use the "probability that this audio is a Queen song" as a proxy for similarity.

First of all, is 300 songs even enough?

Nowhere near enough to train a classifier from scratch. In that case you might be aiming for 10,000 samples or even up to a million, depending on how sophisticated you want your classifier to be.
However, you don't necessrily need to find that many training examples. Instead, if you can find a pre-trained audio classifier for music that is compatible with the libraries you are working with, and use transfer learning, you may get decent results. This works by replacing the last few layers of a neural nework trained with your own ones and training the network on your data only modifying those new changed layers.
From your comments:

similar in voice, like Gnarles Barkley is more similar to Alicia Keys than System of a Down

The tone of the vocals is one part of many things that can vary between music recordings. A classifier with only a few examples to work from will not be able to isolate just the elements you are interested in. Also, without examples that are explicitly labelled in a way that "more similar" is meaningful in the way that you want, you will not have control over whether the neural network identifies Brian May's guitar, or 1970s studio equipment as the most easy to identify element compared to Freddie Mercury's vocals.
These things may limit the usefulness or accuracy of your first attempt, but I would not suggest you consider them immediately. You basic idea should produce something that has an interesting behaviour when given different inputs. Just be realistic that you will not get state-of-the-art results on your first attempts at the project.


Why is the actor-critic algorithm limited to using on-policy data? Why is the actor-critic algorithm limited to using on-policy data? Or can we use the actor-critic algorithm with off-policy data?

It's because, in the actor-critic algorithm, the objective function is an expectation under the $\tau$ of the policy. If we want to use off-policy data, we have to resort to importance sampling relative to the other policy.


For which of the following reasons can Hill Climbing get stuck? One-point Hill Climbing can get stuck due to which of the following phenomenon?

Local Maxima
Inadmissible heuristic
Ridges
Plateau

This question really looks like a homework problem, in part because it is too vague (what does it mean to 'get stuck' exactly?).

Hill climbing stops when it reaches a local maximum.

Hill climbing is an uninformed search algorithm, so it does not make use of a heuristic.

Hill climbing may or may not stop on a ridge, depending on the implementation. Some common implementations move randomly when equal (but not better) moves are available. Others stop in this situation.

Hill climbing may or may not stop on a plateau, depending on how the implementation handles the case of ties, like with ridges. Whether you think it 'gets stuck' depends a lot on your definition. Implementations that stop when encountering a tie will stop. Implementations that don't stop will eventually exit the plateau through random movements, but it could take infinite time if your movements in the gradient-free region are unlucky.


How to give each segment a new colour in image segmentation? I'm currently working on a tumour detection project using Dicom images. As I'm a beginner, I don't know how to segment each part in the image and give each segment a new different colour. I'm using the Watershed segmentation algorithm in Matlab, which only segments the part of interest (the ones that have tumour).

Deep learning based image segmentation is basically pixel-wise image classification.

So instead of predicting a $C$ dimensional vector $\vec{x}$ where $C$ corresponds to the number of class labels, you predict a tensor $x\in\mathbb{R}^{h\times w \times C}$ of the same height and width dimensions as your input image, and with $C$ channels which correspond to the different classes.

The paper "U-Net: Convolutional Networks for Biomedical Image Segmentation" provides a good overview of the current methods.


Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)? I'm looking at some baseline implementations of RL agents on the Pendulum environment. My guess was to use a relatively small neural net (~100 parameters).
I'm comparing my solution with some baselines, e.g. the top entry on the Pendulum leaderboard. The models for these solutions are typically huge, i.e. ~120k parameters. What's more, they use very large replay buffers as well, like ~1M transitions. Such model sizes seem warranted for Atari-like environments, but for something as small as the Pendulum, this seems like complete overkill to me.
Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?

Actually, I just started inspecting the entries further down in the leaderboard list, and there are in fact more modest architectures, e.g. this one, which uses 3 hidden layers with 8 units each.


How can an AI train itself if no one is telling it if its answer is correct or wrong? I am a programmer but not in the field of AI. A question constantly confuses me is that how can an AI be trained if we human beings are not telling it its calculation is correct?

For example, news usually said something like "company A has a large human face database so that it can train its facial recognition program more efficiently". What the piece of news doesn't mention is whether a human engineer needs to tell the AI program each of the program's recognition result is accurate or not.

Are there any engineers who are constantly telling an AI what it produced it correct or wrong? If no, how can an AI determine if the result it produces is correct or wrong?

By "company A has a large human face database so that it can train its facial recognition program more efficiently" the article probably means that there is a training dataset $S$ of the form

$$
S = \{ (\mathbf{x}_1, y_1), \dots,(\mathbf{x}_N, y_N) \}
$$

where $\mathbf{x}_i$ is an image of the face of the $i$th human and $y_i$ (which is often called a label, class or target) is e.g. the name of the $i$th human. So, the programmer provides a supervisory signal (the label) for the AI to learn. The programmer also specifies the function that determines the error the AI program is making, based on the answer of the AI model and $y_i$.

This way of learning is called supervised learning (SL). However, there are other ways of training an AI. For example, there is unsupervised learning (UL), where the AI needs to find patterns in the data by aggregating objects based on some similarity measure, which is specified by the programmer. There's also reinforcement learning (RL), where the programmer specifies only certain reinforcement signals, that is, the programmer tells the AI which moves or results are "good" and which ones are "bad" to achieve its goal, by giving to the AI, respectively, a positive or negative reward. You can also combine these three approaches and there are other variations.

  Are there any engineers who are constantly telling an AI what it produced it correct or wrong?

Yes, in the case of SL. In the case of RL, the programmer also needs to provide the reinforcement signal, but it doesn't need to explicitly tell the AI which action it needs to take. In UL, the programmer needs to specify the way the AI needs to aggregate the objects, so, in this case, the programmer is also involved in the learning process.


What does it mean when a model "statistically outperforms" another? I was reading this paper where they are stating the following:

We also use the T-Test to test the significance of GMAN in 1 hour ahead prediction compared to Graph WaveNet. The p-value is less than 0.01, which demonstrates that GMAN statistically outperforms Graph WaveNet.

What does "Model A statistically outperforms B" mean in this context? And how should the p-value threshold be selected?

Most model-fitting is stochastic, so you get different parameters every time you train, and you usually can't say that one algorithm will always give you a better-performing model.
However, since you can retrain many times to get a distribution of models, you can use a statistical test like the T-Test to say "algorithm A usually produces a better model than algorithm B," which is what they mean by "statistically outperforms."
p-value is usually set by consensus in the field. The higher the p-value, the less confidence you have that there's a statistical difference between the distribution of values being compared. 0.1 might be normal in a field where data is very expensive to collect (like risky, long-term studies of humans), but in machine learning, it's usually easy enough to retrain a model that 0.01 is common, and demonstrates very high confidence. To know more about selecting and interpreting the values, I recommend Wikipedia's page on statistical significance.


Which linear algebra book should I read to understand vectorized operations? I am reading Goodfellow's book about neural networks, but I am stuck in the mathematical calculus of the back-propagation algorithm. I understood the principle, and some Youtube videos explaining this algorithm shown step-by-step, but now I would like to understand the matrix calculus (so not basic calculus!), that is, calculus with matrices and vectors, but especially everything related to the derivatives with respect to a matrix or a vector, and so on.
Which math book could you advise me to read?
I specify I studied 2 years after the bachelor in math school (in French: mathématiques supérieures et spéciales), but did not practice for years.

If you already have two years of a bachelor's of mathematics, I recommend part I of the book that you're mentioning. That part of the book reviews the main mathematics used in the optimization of neural nets (in part 1), and then actually goes through the various models in detail in the later parts. The review is done at a level that is suitable for someone who has already studied these topics, but needs a refresher.
The book Matrix Differential Calculus with Applications in Statistics and Econometrics covers more advanced topics, which might also be what you are looking for. There is also the related Wikipedia article.


Is anybody still researching GOFAI? A lot of textbooks and introductory lectures typically split AI into connectionism and GOFAI (Good Old Fashioned AI). From a purely technical perspective, it seems that connectionism has grown into machine learning and data science, while nobody talks about GOFAI, Symbolic AI, or Expert Systems at all.
Is anyone of note still working on GOFAI?

Oh yeah, definitely.  Just to pick one example, you have Douglas Hofstader's group at Indiana.  I think most of what they do would fall under the rubric of GOFAI (or at least closer to that than the statistical machine learning stuff).

Beyond that, just go to the CORR and browse around the AI category. You'll see plenty of neural networks and probabilistic stuff, but you'll also find the papers by the folks doing symbolic processing / GOFAI as well.


Generate Image with Artificial intelligence I am pretty new to Artificial Intelligence programming, however i do understand the basic concept.
I have an idea in my mind:

Import a JPEG Image,
Convert this Image into a 2D Array (x,y values + r g b values).
Then create a second array with same (xy) values wit rgb all set to 0,0,0.
Now i want to build an AI Layer which will try to lower the error factor between the arrays until they are equal (the rgb values in the second array are equal to the first array (error factor 0) ).
I would prefer to do it in Java. Any suggestions to librarys or example that can help me get started? Thanks for any help.

For recreating an image exactly the same as the original, you can use an autoencoder. This basically use AI Layers to encode the image raw pixel values to a vector of floats, drastically decreasing the representing vector. Afterwards another AI Layer increases the dimensions back to the original image. The method does not required labels, as it only refer to teh image to encode it to a vector of features. For implementing in java, there is not a lot 9f resources. However, you can check this library out: https://deeplearning4j.org/
For the implementation, see this: https://github.com/eclipse/deeplearning4j-examples/blob/master/dl4j-examples/src/main/java/org/deeplearning4j/examples/unsupervised/variational/VariationalAutoEncoderExample.java
For the method, you can see this python tutorial and implement it in java. https://towardsdatascience.com/autoencoders-in-keras-c1f57b9a2fd7

For generating completely new image, you can try GAN(Generative Adverserial Network). This generates completely new images from a random noise image. The noise image is passed through a generator which is a CNN(convolutional neural network) and get a result image. The result image is then feed to  a discriminator(CNN as well) to classify if that image is fake or real. The generator and discriminator compete and slowly gets better. For java implementation, see this: https://github.com/wmeddie/dl4j-gans

Hope I can help you and have a nice day!


How can I systematically learn about the theory of neural networks? I have seen a few articles about neural nets. Mostly they went along these lines: we tried these architectures, these meta parameters, we trained it for $x$ hours on $y$ CPUs, and it gave us these results that are 0.1% better than state of the art.

What I am interested in is whether there exists (at least as a work in progress) a framework, that gives some explanation why is some architecture better than other, what makes one activation function more suitable for image recognition than other, etc. Do you have some tips about where to start? I would prefer something more systematic than Google search (a book, a list of key articles is ideal).

Good question, there is a lot of work in that field. The first part before saying which machine learning algorithm is better and why is defining the problem. Is the problem an optimisation, classification, anomaly detection problem because then you need to use the appropriate machine learning algorithms. Let's assume its an optimisation problem.

Is this problem, dynamic, or static. Is this time series data? So we need to understand the problem.

Each optimisation problem has a landscape or a fitness landscape. Computer science has some nice toy landscapes.

There is a lot of work in determining the nature of the problem landscape. Have a look at K Malan's work.

Once you can identify the problem space and understand it's characteristics then you can start to identify what machine learning functions work well on what kind of landscapes. This is a totally different field of research.

For example some researchers are working on how different evolutionary algorithms handle different landscapes, or neural networks handle different landscapes.

Start by exploring the types of machine learning problems.
Understand the complexity of the problem, followed by classification of machine learning algorithms for specific problem spaces.


Is it possible to implement reinforcement learning using a neural network? I've implemented the reinforcement learning algorithm for an agent to play snappy bird (a shameless cheap ripoff of flappy bird) utilizing a q-table for storing the history for future lookups. It works and eventually achieves perfect convergence after enough training.

Is it possible to implement a neural network to do function approximation in order to accomplish the purpose of the q-table? Obviously, storage is a concern with the q-table, but it doesn't seem to ever train with the neural net alone. Perhaps training the NN on an existing q-table would work, but I would like to not use a q-table at all if possible.

Andrej Karpathy's blog has a tutorial on getting a neural network to learn pong with reinforcement learning. His commentary on the current state of the field is interesting.

He also provides a whole bunch of links (David Silver's course catches my eye). Here is a working link to the lecture videos.

Here are demos of DeepMinds game playing.
Get links to the papers at Andrej Karpathy's blog above
- rat fps

nice demos at 19 minutes into this


Streaming time series data to detect fraud? I'm currently looking in to the possibility of using machine learning to detect fraudulent transactions on our website based on the events that happen for each user.

I'd like to be able to stream events in to it, such as sign up, order placed, inviting another user, etc along with the the time and some how come up with a probability of how likely it is that the person is acting in a fraudulent manner

I'm totally new to machine learning and everything I read goes straight over my head basically :/

Can someone please explain the type of neural network I'd need, how I'd decide how to set it up and how I would go about training it?

Elasticsearch with X-pack is a tool that would come in handy. The suite has an visualization engine, Kibana and logstash that collect, parse and transform the data telemetry.

https://www.elastic.co/products/x-pack/machine-learning

Modeling can take a while to learn, luckily there are tools like Elasticsearch that can help.

Grafana also has tools but does not have a specific AI engine to it.But has a lot of integration with third party fraud detection tools.

https://grafana.com/

Sridhar


How to determine the target value when using ReLU as activation function? Consider the following simple neural network with only one neuron.

The input is $x_1$ and $y_2$, where $-250 < x < 250$ and $-250 < y < 250$
The weights of the only neuron are $w_1$ and $w_1$
The output of the neuron is given by $o = \sigma(x_1w_1 + x_2w_2 + b)$, where $\sigma$ is the ReLU activation function and $b$ the bias.
Thus the cost should be $(o - y)^2$.

When using the sigmoid activation function, the target for each point is usually $0$ or $1$.

But I'm a little confused witch target to use when the activation function is the ReLU, given that it can output numbers greater than 1.

ReLU and sigmoid have different properties (i.e. range), as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but some people may use it for some reason, e.g. regression tasks where the output needs to be positive). ReLU is usually used as the activation function of a hidden layer. However, in your case, you don't have hidden layers.

The sigmoid function is used as the activation function of the output layer when you need to interpret the output of the neural network as a probability, i.e. a number between $0$ and $1$, given that the sigmoid function does exactly this, i.e. it squashes its input to the range $[0, 1]$, i.e. $\text{sigmoid}(x) = p \in [0, 1]$. When do you need the output of the network to be a probability?  For example, if you decide to use the cross-entropy loss function (which is equivalent to the negative log-likelihood), then the output of your network should be a probability. For example, if you need to solve a binary classification task, then the combination of a sigmoid as the activation function of the output layer and the binary cross-entropy as the loss function is probably what you need.

You could also have a classification problem with more than 2 classes (multi-class classification problem). In that case, you probably need to use a softmax as the activation function of your network combined with a cross-entropy loss function.

See this question How to choose cross-entropy loss in TensorFlow? on Stack Overflow for more info about different cross-entropy functions.

By the way, in general, the targets don't necessarily need to be restricted to be 0 or 1. For example, if you are solving a regression task, your target may just be any number. However, in that case, you may need another loss function (which is often the mean squared error).


How can I perform multivariable regression with neural networks? I want to use a neural network to perform a multivariable regression, where my dataset contains multiple features, but I can't for the life of me figure it out. Every kind of tutorial on the internet seems to be either for a single feature without information on how to upgrade it to multiple, or results in a yes or a no when I need numeric predictions (that is, it uses neural networks for classification).

Can someone please recommend some kind of resource I can use to learn this?

Have a look at sklearn's sklearn.neural_network.MLPRegressor class, which uses a multi-layer neural network to do regression. You first need to define the object MLPRegressor, for example, by specifying the value of the parameter hidden_layer_sizes, which determines the number of layers and the number of neurons per layer, then you should call the method fit on this created object and pass to it your data matrix $X \in \mathbb{R}^{n \times m}$, where $n$ is the number of samples and $m$ is the number of features.


What are the differences between Bytenet and Wavenet? I recently read Bytenet and Wavenet and I was curious why the first model is not as popular as the second. From my understanding, Bytenet can be seen as a seq2seq model where the encoder and the decoder are similar to Wavenet. Following the trends from NLP where seq2seq models seem to perform better, I find it strange that I couldn't find any paper that compares the two. Are there any drawbacks of Bytenet over Wavenet other than the computation time?

My conclusion is the same as yours that there doesn't seem to be any published comparison of the two models.  ByteNet is computationally expensive and requires a lot of parameters.  WaveNet improves on ByteNet's efficiency, as you mentioned, and I believe that is the main difference.


Use deep learning to rank video scenes I'm new to machine learning and especially, deep learning. Given a video (and it's subtitle), I need to generate a 10-second summary out of this video. How can I use ML and DL to produce the most representative summary out of this video? More specifically, given video scenes, what are some ways to select and rank them, and how to do it? Any ideas would be helpful.

It’s seems like quite challenging problem; at least you would need quite a lot of annotated data and computational power.

The approaches/optimizations you could consider:

To make scene change detection and take short piece out of each
To introduce some kind of “novelty” metric and try to maximize
it to get most different parts of video
To convert video to kind of vector with existing solution like
r-cnn and yolo and then process it with recurrent networks.
The task seems to be very close to video capturing/summarization, you
can take inspiration there
Also, the attention approach might be handy, look, for example
self-attention for video,
semantic attention for
video


Relative compute time for each type of layer in a neural network Hello,
I would like to know whether this picture from the paper: Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability valid?

Questions:
1) Does InnerProduct (Fully connected) layer actually take more time to compute in a neural network than Convolution?

2) Is assessing GLOPs/time a good way of estimating performance of different types of layers in a neural network on any hardware? (Conv, FC etc.)

3) Does anyone know where I can find GFLOPs vs compute time for different types of layers across GPUs/CPUs? (I know DeepBench, any other suggestions would be great too)

The graph seems to show the relative compute time of each component of the learning process, independent of the layer, using the typical meaning of layer in the context of artificial networks.

The data is not unbelievable, in that one would expect a matrix-vector multiplication of an entire layer output would take more time than a matrix-vector multiplication of the size of a convolution kernel. Still, these representations are for a specific test condition and measurement methodology.

Most of the theoretical and practical limits of parallel processing are heavily biased by seventy years of sequential processing as the foundation of computer science study and practice. If an understanding of parallel processing is desired, the von Neumann architecture is not the place to look. At the time of that development, computers could either be made of wood, steel, vacuum tubes, or relays. Developing a computer with 65536 parallel data paths was unrealistic to the extreme. The invention of algorithms was a workaround to funnel all processing through a single CPU. We've been backing out of that mind set for the last thirty years, beginning with FPUs, GPUs, multiple buses, computer clusters, and multiple cores. More parallelism in VLSI form is coming, and soon.

The demonstration of parallel processing comes, as many things do, from biology. Look at farming and grain growth under the sun. Look at visual circuitry coming from the eyes of thousands of species. Consider the orbits of millions of planets and their satellites. The artificial parallelism to examine is the transportation systems and the Internet. Parallel processing is efficient when the process was designed as a parallel process rather than trying to make parallel that which was designed as serial. For instance, the human ear does the physical equivalent of a real time windowing FFT without performing a single multiplication operation.

The bottleneck of the floating point multiplication of an attenuation matrix with a vector (parameterizing the mix of signals from one layer to the next) is a key to why NVidia has done well with re-purposing rendering hardware for artificial network training. It is also one of the many reasons this question is interesting: If digital values are mere estimates, why not return to analog for AI?.

When surfing for GFLOPs vs compute time for different types of layers across GPUs and CPUs, using the term 'operation' instead of 'layer' in the search terms may help. Even though elementary operations are lower level than that of the comparison desired, the term layer is delegated strongly to the higher level meaning in the machine learning literature. Reliable data in this space is thin because much of the speed optimization has already been researched and developed, so whatever the visualization shows is useless unless you work on VLSI development at Intel, Samsung, NVidia, Qualcom, TI, or one of the other silicon giants.


Can ConvLSTMs outuput images? I am working on an image to image regression task which requires me to develop a deep learning model that takes in a sequence of 5 images and return another image. The sequence of 5 images and the output images are conceptually and temporally related. In fact, the 5 images in the sequence each correspond to timestep in a simulation and the output, the one I am trying to predict, corresponds eventually to the 6th timestep of that sequence.

For now, I have been training a simple regression-type CNN model which takes in the sequence of 5 images stored in a list and outputs an image corresponding to the next timestep in the simulation. This does work with a small and rather simple dataset (13000 images) but works a bit worse on a more diverse and larger dataset (102000 images).

For this reason, I have been researching a bit now in order to find a better way to carry out this task and I found the idea of ConvLSTMs. However, I have seen these applied to the prediction of feature and the output of a sentence describing that image. What I wanted to know is whether ConvLSTMs can also output images, but more importantly if they can be applied to my case. If not, what other types of deep learning network can be suitable for this task?

Thanks in advance!

If you only need 5 frames to predict the next frame, then I'd recommend a U-Net architecture, wich is basically a CNN encoder/decoder network in which the decoder uses the intermediate features produced in the encoder as well as its own features to produce an output image. Also, in additional to using a conventional L2 loss for the output image, you can always add an additional GAN loss to make the image look more realistic.

If using a longer history of frames can help, then I recommend taking a look at "Recurrent Environment Simulators" and combining it with the ideas above.

Hope it helps!


Is it possible that the model is overfitting when the training and validation accuracy increase? I am aware of similar questions that have been asked, and I have gone through many. I want to bring my case to SE to understand better what my results are.
I am working with a large dataset (around 75million records), but, for the purpose of testing techniques, I am actually using 2M records. I am working towards malicious traffic identification using NetFlow data. After employing some undersampling to have a balanced dataset according to my target variable (benign or attack) I have 1,240,950 of records in the training set and 310,238 in the validation set. Therefore I believe there is a good amount of data to train a Deep neural network properly.
After using Yeo-Yohnsons transform and standardizing the data, I train the network with a very basic model:
def basem():

    model = Sequential()

    model.add(Dense(25, input_dim=38))
    model.add(Activation("relu"))

    model.add(Dense(50))
    model.add(Activation("relu"))

    model.add(Dense(50))
    model.add(Activation("relu"))

    model.add(Dense(25))
    model.add(Activation("relu"))

    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer="adam", metrics=['accuracy'])
    return model

model_base = basem()
model_base._name = 'base'

history_base = model_base.fit(X_train, y_train, batch_size=2048,
                    epochs=15, validation_data=(X_val,y_val), shuffle=True)

This gives me the following plot
It maybe because I am a newbie, but this plot looks too perfect. It is weird to see validation and training accuracy growing together, although I believe this is what we want right? But now I have the feeling it is overfitting. Therefore I use the model and a 5-fold cross validation to understand how well it generalizes. Results, mean accuracy and mean std(%), are:
test acc: 0.9816503485233088
test_prec: 0.9840033637114158
test_f1: 0.9816046990113001
test_recall: 0.9792384866432975
test_roc_auc: 0.9980004347946355

Dev acc: 0.052931962886091546
Dev prec: 0.2854656099314699
Dev f1: 0.057228805478181974
Dev recall: 0.3597811552056071
Dev roc auc: 0.0036456892671197097

If I understand correctly, accuracy is high which is generally good and the standard deviation is very low for each metric, the highest being 0.359% for recall. Does this mean my model generalizes well?
Edit
Adding dropout (0.3) to each layer yields the following:

Now, my validation accuracy is higher than my training. I can't make sense of any of this.

I'll try to answer on more general questions

Is it ok that model performs better on validation, then on train?

It's certainly fine if you use techniques like dropout or data augmentation and the difference is not that big. Because in case of dropout for train you use part of the network, and for validation the whole.

I'm suspicious my model is too good. What could I do?

That a good point, because in my experience too good results sometime are reasoned by the flow in the training. The most common is data leakage, that's mean you give the model a way to fool around in unexpected and unwanted way.
Let me give an example. Lets imagine you try to detect malicious traffic and you have a data about sessions and you split records on train/val. Let's imagine one of the features is IP address and you could have multiple sessions from same IP. But if same IP would be in both train and val for the crooked users, then model could just remember the 'bad' IP and thus getting high score on both train and val.
So, to avoid it you need to think about things like that. If records could be group in some too meaningful way, make sure you put the whole group either in train or val. Another thing, that's it's usually good thing to split it by time as well, i.e. use for train records for 2018 and 2019 and for val for 2020. Thus you not only avoid data leakage, but you make sure your model is robust for future predictions.
More general, try to understand, why you getting such good result with such simple model. Try to squeeze the data even smaller, drop some features to understand, when it would go done.
Finally I would recommend the post by Andrej Karpathy, when he gave lots of really good advices on practical side of training NN. http://karpathy.github.io/2019/04/25/recipe/


Can I train a neural network incrementally given new daily data? I would like to know if it was possible to train a neural network on daily new data. Let me explain this more in detail. Let's say you have daily data from 2010 to 2019. You train your NN on all of it, but, from now on, every day in 2019 you get new data. Is it possible to "append" the training of the NN or do we need to retrain an entire NN with the data from $2010$ to $2019+n$ with $n$ the day for every new day?

I don't know if it is relevant but my work is on binary classification.

Yes, this is possible. Continuously extending your training data is known as incremental learning.

You might also want to take a look at transfer learning, in which you reuse a trained model for a different purpose. This is very useful if you have a smaller dataset.

In your particular case, you could train a NN once using your data from 2010 to 2019 and use it as a base model. Every time you get new data, you can use transfer learning to slightly re-train this model. Based on parameters such as the number of epochs and the learning rate, you can determine how much of an impact this new data will have.


Generalization error: Inputs drawn from distributions I am currently studying Deep Learning by Goodfellow, Bengio, and Courville. In chapter 5.2 Capacity, Overfitting and Underfitting, the authors say the following:

  Typically, when training a machine learning model, we have access to a training set; we can compute some error measure on the training set, called the training error; and we reduce this training error. So far, what we have described is simply an optimization problem. What separates machine learning from optimization is that we want the generalization error, also called the test error, to be low as well. The generalization error is defined as the expected value of the error on a new input. Here the expectation is taken across different possible inputs, drawn from the distribution of inputs we expect the system to encounter in practice.

I found this part unclear:

  Here the expectation is taken across different possible inputs, drawn from the distribution of inputs we expect the system to encounter in practice.

The language used here is confusing me, because it is discussing a "distribution", as in a "probability distribution", but then refers to inputs, which are data gathered from outside of any probability distribution. Based on the limited information my studying of machine learning has taught me so far, my understanding is that the machine learning algorithm (or, rather, some machine learning algorithms) uses training data to implicitly construct some probability distribution, right? So is this what it is referring to here? Is the "distribution of inputs we expect the system to encounter in practice" the so called "test set"? I would greatly appreciate it if people would please take the time to clarify this.

The language used here is confusing me, because it is discussing a "distribution", as in a "probability distribution", but then refers to inputs, which are data gathered from outside of any probability distribution. Based on the limited information my studying of machine learning has taught me so far, my understanding is that the machine learning algorithm (or, rather, some machine learning algorithms) uses training data to implicitly construct some probability distribution, right? So is this what it is referring to here?

They're not referring to probability distributions of training data that ML algorithms (implicitly) construct here. The main point of confusion seems to be where you state this:

  but then refers to inputs, which are data gathered from outside of any probability distribution

Any data / inputs ever collected always originate from some distribution. We will typically not exactly know what that distribution is, we are often not able to provide a clean expression for it, and it might not even be a nice "smooth" distribution, but that doesn't mean it doesn't exist.

If I collect a large number of photographs of $H \times W$ pixels of streets for the purpose of training a self-driving car, then this collection of training data was collected from some distribution. For each of the pixels in the $H \times W$ plane, there exists some probability distribution that tells us how likely it is for such a pixel to have a certain colour under the data collection procedure that was used to generate our data. This is a largely unknown distribution, for which we don't have a nice mathematical expression, but it does exist. I assume that, in this distribution, it's relatively likely for pixels in the centre to be gray (because streets tend to be gray and we collected data by taking photographs of streets). I also guess it's relatively likely for pixels at the top of the images to be blue, because of the sky. Other than that, we can't say much about the distribution, but it does exist.

  Is the "distribution of inputs we expect the system to encounter in practice" the so called "test set"?

Kind of, yeah. Although I suppose the "test set" is mostly a thing in academic settings, where we use a test set to evaluate how well an approach performs on data that it did not observe during training. In the "real world", the distribution of inputs we expect the system to encounter in practice refers to the distribution that generates samples we encounter after "deployment" of the model. For example, this could be the distribution over all images that a self-driving car may encounter when driving anywhere in the world.

Continuing with the self-driving car example, we may get a large generalization error if we only train it on images of streets in one particular city or country, but then afterwards have it drive in many different cities or countries around the world (which may look very different).


Why does not the deepAR model of Amazon require the time series being stationary, as opposed to ARMA model? As what the title said. Does not deepAR require the time series being stationary?

I have have been part of a project where we implemented Amazon Forecast service in production. As per my practical experience, we need not worry about stationarity while applying DeepAR.
As per my understanding as in ARIMA the prediction function is a function around the time series moving average, but in DeepAR we are more relying on the backtest window and leg-horizon length.
DeepAR uses recurrent neural Networks (RNN) and gated recurrent unit (GRU) without any assumption on their probability distribution to learn the sequence for prediction.
For a detailed explanation, you can refer. https://aws.amazon.com/blogs/machine-learning/forecasting-time-series-with-dynamic-deep-learning-on-aws/


Can the same input for a plain neural network be used for a convolutional neural network? Can the same input for a plain neural network be used for CNNs? Or does the input matrix need to be structured in a different way for CNNs compared to regular NNs?

There is no restriction on how you input a data to a NN. You can input it in 1D arrays and do element-wise multiplication using 4-5 loops and imposing certain conditions(which will be slow and hence $nD$ matrix notations are used for a CNN).
Ultimately, the library you are using (TensorFlow, NumPy might convert it into its own convenient dimensions).
The main thing different of a CNN from a normal NN is:

The number of parameters of a CNN in a convolutional layer is less than the number of input features.
$parameters \le features$ (in general it is less than).

Different people have different ways of viewing how the convolutional layer work but the general consensus is that the weights of the convolutional layers of a CNN are like digital filters. It will be a $nD$ filter if input dimension is $nD$. The  output is obtained by superimposing the filter on a certain part of the input and doing element-wise multiplication of the values of filter and the values of the input upon which the filter is superimposed upon. How you implement this particular operation depends on you.

So the answer to your question will be same network cannot be used, but it might be used with modifications (a normal NN is the limiting case of a CNN where $features=parameters$.


Reinforcement Learning (and specifically REINFORCE algorithm) for one-round "games" I'm interested about using Reinforcement Learning in a setting that might seem more suitable for Supervised Learning. There's a dataset $X$ and for each sample $x$ some decision needs to be made. Supervised Learning can't be used since there aren't any algorithms to solve or approximate the problem (so I can't solve it on the dataset) but for a given decision it's very easy to decide how good it is (define a reward).

For example, you can think about the knapsack problem - let's say we have a dataset where each sample $x$ is a list (of let's say size 5) of objects each associated with a weight and a value and we want to decide which objects to choose (of course you can solve the knapsack problem for lists of size 5, but let's imagine that you can't). For each solution the reward is the value of the chosen objects (and if the weight exceeds the allowed weight then the reward is 0 or something). So, we let an agent "play" with each sample $M$ times, where play just means choosing some subset and training with the given value.

For the $i$-th sample the step can be adjusted to be:
$$\theta = \theta + \alpha \nabla_{\theta}log \pi_{\theta}(a|x^i)v$$
for each "game" with "action" $a$ and value $v$.

instead of the original step:
$$\theta = \theta + \alpha \nabla_{\theta}log \pi_{\theta}(a_t|s_t)v_t$$
Essentially, we replace the state with the sample.

The issue with this is that REINFORCE assumes that an action also leads to some new state where here it is not the case. Anyway, do you think something like this could work?

This seems like a multi-armed bandit problem (no states involved here). I had the same problem some times ago and I was advised to sample the output distribution M times, calculate the rewards and then feed them to the agent, this was also explained in this paper Algorithm 1 page 3 (but different problem & different context). I honestly don't know if this will work for your case. You could also take a look at this example.


Showing first layer RGB weights similarly to AlexNet I would like to show the RGB features learned in the first layer of a convolutional neural network similarly to this visualization of the same layer's features from AlexNet:

My learned weights are in the range [-1.1,1.1]. When I use imshow in python or imagesc in Matlab, the weight values are clipped to [0,1], leaving only positive weights intact, everything else black (obviously).
Negative weight values could be informative, so I don't want to clip them. Rescaling the weights to [0,1] works fine for grayscale features, but not for RGB features as it is unclear how negative values of a channel should be visualized. In the above picture 0 furthermore seems to map to the middle of the range (gray).
How are such RGB features visualized so that they look similarly to above AlexNet visualization?
(Sorry for the beginner's question.)

It seems Alex has just used the Matlab function mat2gray, as described here: https://www.mathworks.com/help/vision/ug/image-category-classification-using-deep-learning.html
The visual outcome of the features is very similar. mat2gray will simply scale the weights between 0 and 1 (no clipping).
Leaving the (slightly adapted) example code of Mathworks here for future reference:
layer1weights = mat2gray(layer1weights) ;
layer1weights = imresize(layer1weights,5) ;

figure
montage(layer1weights)
title('First convolutional layer weights')


Why does it make sense to study MDPs with finite state and action spaces? In the standard Markov Decision Process (MDP) formalization of the reinforcement-learning (RL) problem (Sutton & Barto, 1998), a decision maker interacts with an environment consisting of finite state and action spaces.

This is an extract from this paper, although it has nothing to do with the paper's content per se (just a small part of the introduction).

Could someone please explain why it makes sense to study finite state and action spaces?

In the real world, we might not be able to restrict ourselves to a finite number of states and actions! Thinking of humans as RL agents, this really doesn't make sense.

In addition to the reason outlined in the comment, also note that if the state-space and action-space are both finite and of feasible size, tabular methods can be used, and there are some advantages to them (like the existence of convergence guarantees and generally a smaller number of hyperparameters to tune).


Do the order of the features ie channel matter for a 1d convolutional network? Do the test dataset feature order and inference (real world) feature order have to be the same as the training dataset? For example, if features are in the order (a,c,b,e,d) for the training dataset, does that particular order have to match for the inference and test dataset?

Generally, order matters. A (trained) Neural Network (NN) is just a mathematical function trained on taking some given input and producing the corresponding output. So, if you train a certain node on producing large output if (and only if) an animal is present in a picture (for example), but later you give it the numeric evidence for a car being present in the image, it will still produce large output, indicating an animal being present in the image. This is simply because a node doesn't know what it is receiving or supposed to detect. It just follows its standard mathematical procedure.
So, if you train your network on one kind of input data, you must provide the same kind of input data during testing (or at least you cannot simply exchange inputs or outputs of certain nodes without manually correcting for that in the remainder of the network).
In the most simple case, consider whether you could simply swap the inputs to that simple function: $f(x,y) = x^2 + y$. If you swap the inputs, the output will be different. The exactly same applies to NNs.
Addition:
I think this post explains and especially illustrates the intuition nicely.


Detailed explaination of Facenet Model for face recogniton? Can some one explain how Facenet model works in detail and simple words .

Facenet is a Siamese network. It's basic architecture is this:

The input(a face) is fed through a deep convolutional neural network and also a fully connected layer at the end. The fully connected layer at the end output an embedding of the input image which is a predefined size. The embedding can contain feature that human understand or maybe not. The embedding represent the input image, just in a "compressed" form.

To further explain that, let me give an example. Let's say that you have to describe a face. What's will you say? You will probably say something like the face is round, the eyes are blue, it is a female face and more. The neural network is doing what you are doing, describing the face, but using numbers instead of words.

To do a face recognition task, the network take an pre taken image of the list of people to recognize and the unknown new data from the people to be recognised. It then feed both images into the network and get the embeddings. The network then calculate the distance of the two embeddings, using some metric such as squared error or absolute error. In the image it uses the squared error. If the error is below a certain threshold, the face is recognised. If not, it then loops through the other pre taken images in the set of faces of the system and do the task again. The system stores embedding of the pre taken images before hand.

For training the FaceNet, triplet loss is used. Triplet loss has been explained in another of your post, by me. What is the formula used to calculate the accuracy in the FaceNet model?

Basically, the model is trained using the triplet loss as it can train the network to output a similar embedding for the same person and a very different embedded for a different person.

Sometimes, a binary classification end is also used. It removed the need of the triplet loss and outputs a number from 0 to 1 for similarity instead. This removes the triplet loss part.

Hope my answer can help you and have a nice day!


Which marketing-related classification challenges is a feed forward neural network suited to solve? I am trying to think of some marketing-related classification challenges that a feed-forward neural network would be suited for.

Any ideas?

Has a data set been chosen? There's a wide variety of data out there (see this, this, or this).  Analytical possibilities are largely determined by the available data, so choosing your data would probably help to clarify your objectives.


Research into social behavior in Prisoner's Dilemma I've been working on research into reproducing social behavior using multi-agent reinforcement learning. My focus has been on a GridWorld-style game, but I was thinking that maybe a simpler Prisoner's Dilemma game could be a better approach. I tried to find existing research papers in this direction, but couldn't find any, so I'd like to describe what I'm looking for in case anyone here knows of such research.
I'm looking for research into scenarios where multiple RL agents are playing Iterated Prisoner's Dilemma with each other, and social behaviors emerge. Let me specify what I mean by "social behaviors." Most research I've seen into RL/IPD (example) focuses on how to achieve the ideal strategy, and how to get there the fastest, and what common archetypes of strategies emerge. That is all nice and well, but not what I'm interested in.
An agent executing a Tit-for-Tat strategy is giving positive reinforcement to the other player for "good" behavior, and negative reinforcement for "bad" behavior. That is why it wins. My key point here is that this carrot-and-stick method is done individually rather than in groups. I want to see it evolve within a group.
I want to see an entire group of agents evolve to punish and reward other players according to how they behaved with the group. I believe that fascinating group dynamics could be observed in that scenario.
I programmed such a scenario a decade ago, but by writing an algorithm manually, not using deep RL. I want to do it using deep RL, but first I want to know whether there are existing attempts.
Does anyone know whether such research exists?

I've asked this on rl-list and got lots of interesting references to research papers.
So far the most promising one I've seen is this:
Nicolas Anastassacos, Stephen Hailes and Mirco Musolesi. Partner Selection for the Emergence of Cooperation in Multi-Agent Systems using Reinforcement Learning. In AAAI 2020. New York City, NY, USA. February 2020.


Is there a computer vision service for classifying images on a fixed array of images provided by me? I was checking services like Microsoft Azure's Cognitive Services Computer Vision API and Google's Vision API and they are amazing. I was wondering if these services, or any other cloud service for that matter, can recognize an image's content and classify it on a set of fixed categories defined by me, not by the Cognitive Service provider.

For example, I have different products and I will take several pictures of each product. I want to then use the cloud service and upload all the pictures of each product, so that I can then take a picture of one product and the Computer Vision algorithm will tell me which product I am seeing.

Is it possible? Is there a third party solution for this problem? If so, how many pictures do I need to train each product's recognition?

I hope I was clear. Thanks in advance for any light on the topic!

Actually I'm trying the same thing with the Azure Computer Vision API. Although the API is very good in identifying objects, it has problems identifying specific consumer products (in my experience though). For example it can't really distinguish between two pair of shoes, or two pair of watches.

People recommended me using the: Custom Vision Service combined with Custom Decision Service. I haven't really looked in to them both. But maybe it's useful for you.


Advantages of Kullback-Leibler over L1/L2? I've recently encounter different articles that are recommending to use KL instead of L1/L2 norm when trying to minimize a probability distribution. But none of the articles are giving a clear reasoning why ones is better than the other. Could anyone give me a strong argument why KL is suitable for this?

Considering your question was asking why KL-Divergence would be preferred over MSE, RMSE, or L1/L2 Norm between ground truth and predicted, as loss function for predicting a distribution -

KL-divergence is a measure on probability distributions. It essentially captures the information loss between ground truth distribution and predicted.

L2-norm/MSE/RMSE doesn’t do well with probabilities, because of the power operations involved in the calculation of loss. Probabilities, being fractions under 1, are significantly effected by any power operations (Square or Root), and considering we are calculating the squares of differences of probabilities, the values that are summed are abnormally small, essentially barely learning anything as the random initialization itself starts with an abnormally small loss, almost always staying constant.

L1 norm on the other hand, does not have any power operations, making it relatively acceptable.

Loss functions such as Kullback-Leibler-divergence or Jensen-Shannon-Divergence are preffered for probability distributions because of the statistical meaning they hold. KL-Divergence, as mentioned before, is a statistical measure of information loss between distributions, or in other words, assuming $Q$ is the ground truth distribution, KL-Divergence is a measure of how much $P$ deviates from $Q$. Also considering probability distributions, convergence is much stronger in measures of Information Loss such KL-Divergence.

More clarity on the motivation behind Kullback-Leibler can be read here. Hope your query has been answered!


What is the difference between graph semi-supervised learning and normal semi-supervised learning? Whenever I look for papers involving semi-supervised learning, I always find some that talk about graph semi-supervised learning (e.g. A Unified Framework for Data Poisoning Attack to Graph-based Semi-supervised Learning).

What is the difference between graph semi-supervised learning and normal semi-supervised learning?

The authors of your cited paper use the term graph-based semi-supervised learning (G-SSL) to refer to semi-supervised learning techniques which take graph structured data as their input.

Given their main example, the MNIST dataset, is not graph structured, they detail a method for converting the raw Euclidean data $X$ into said form (represented by its adjacency matrix $S$), and then compute the Laplacian $L$ of this graph:

  We consider the graph-based semi-supervised learning (G-SSL) problem. The input include labeled data $X_{l} ∈ \mathbb{R}^{n_{l}×d}$ and unlabeled data $X_{u} \in \mathbb{R}^{n_{u}×d}$, we define the whole features $X = [X_{l}; X_{u}]$. Denoting the labels of $X_{l}$ as $y_{l}$, our goal is to predict the labels of test data $y_{u}$. The learner applies algorithm $A$ to predict $y_{u}$ from available data $\{X_{l}, y_{l}, X_{u}\}$. Here we restrict $A$ to label propagation method, where we first generate a graph with adjacency matrix $S$ from Gaussian kernel:
  $S_{ij} = \exp(−γ\lVert x_i − x_j\rVert ^{2})$, where the subscripts $x_{i(j)}$ represents the $i(j)$-th row of $X$. Then the graph Laplacian is calculated by $L = D − S$, where $D = \text{diag}\{\sum_{k=1}^{n} S_{ik}\}$ is the degree matrix.

A Unified Framework for Data Poisoning Attack to
Graph-based Semi-supervised Learning (2019)

This is consistent with the terminology as used in other literature:

  Semi-supervised learning for node-level classification.
  Given a single network with partial nodes being labeled and others remaining unlabeled, ConvGNNs can learn a robust model that effectively identifies the class labels for the unlabeled nodes [22]. To this end, an end-to-end framework can be built by stacking a couple of graph convolutional layers followed by a softmax layer for multi-class classification.

A Comprehensive Survey on Graph Neural Networks (2019)


How to perform prediction when some features have missing values? Sorry if this is too noob question, I'm just a beginner.
I have a data set with companies' info. There are 2 kinds of features: financial (revenue and so on) and general info (like the number of employees and date of registration)
I have to predict the probability of default. And the data has gaps: about the half of the companies have no financial data at all. But general features are 100% filled.
What is the best practice for such a situation?
Will be great if you can give some example links to read.

You should look into "missing values". This is an entire research field in itself.
First, you need to identify the type of missing values:

They can be missing purely at random.
Whether they are missing or not is itself a useful feature, and should be treated as a class of its own.

(Those two are the best case scenarios.)

Whether they are missing or not depends on the underlying (unknown) value. For example, a thermometer might fail occasionally if the temperatures get too high. In your case, certain types of companies might be more likely to not share their information.
Information might be missing specifically to mislead you, the data analyst. This is the worst possible scenario, and there is not much you can do.

So, what do you do about it? A few typical options:

Throw out all the rows with missing data: we do not have enough information about these companies.
Throw out all the columns with missing data: this field is not reliably measurable and we shouldn't use it.
Try to guess the missing values. This can be done if the amount of missing data is small. Either you train a predictive model based on the non-missing data, or you fill in the median for that type of row, or you fill in the value of the "closest" matching row. This can be dangerous.
Some algorithms are OK with missing data. Check the documentation for your models and algorithms to see how they deal with missing values.


What does the notation $\nabla_\theta \mathcal{L}$ mean? Here's the general algorithm of maximum entropy inverse reinforcement learning.

This uses a gradient descent algorithm. The point that I do not understand is there is only a single gradient value $\nabla_\theta \mathcal{L}$, and it is used to update a vector of parameters. To me, it does not make sense because it is updating all elements of a vector with the same value $\nabla_\theta \mathcal{L}$. Can you explain the logic behind updating a vector with a single gradient?

This is standard backpropagation. The gradient term you see is in fact a vector of partial derivatives where each element is the partial derivative of the log-likelihood with respect to each element of the parameter vector $\theta$. Therefore, it has the same dimensionality as $\theta$. Each element of the parameter vector is then updated with the respective term in the vector of partial derivatives, which are generally not the same.


Can a data compression function be used to make predictions? I've heard that prediction is equivalent to data compression.
Is there a way to take a compression function and use it to create an AI that predicts?

The way some (not all) compression algorithms work is that they encode frequent events in a short code, and rarer events with a longer code. Overall you save more space by encoding the common elements than you need to expend coding the rare ones. One example of this is a Huffman code, which uses a variable length encoding based on the frequency of the items.

You can use a compression algorithm for prediction if if encodes more than one event at a time. For example, word pairs rather than individual words. Each word pair will have a code, and the common word pairs (eg of the) will have shorter codes that the ones which are less common (eg of three). For prediction, select all the word pairs that start with your known sequence (eg of). Now select from that list the pair with the shortest code (which is more common), so in this example of would more likely be followed by the rather than three. After than, repeat the process with the next word, so look for pairs that begin with the.

All you need is the compression 'code book' which is produced during the compression process -- it's essentially a model of the data you compressed. This also works for longer sequences than pairs, of course.

If you want to know more about the topic, I can recommend Managing Gigabytes by Witten, Moffat, and Bell. Great book on compression techniques.


What is the space complexity for training a neural network using back-propagation? Suppose that a simple feedforward neural network (FFNN) contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_l$ nodes in each layer. What is the space complexity to train this FFNN using back-propagation?
I know how to find the space complexity of algorithms. I found an answer here, but here it is said that the space complexity depends on the number of units, but I think it must also depend on the input size.
Can someone help me in finding its worst-case space complexity?

I will not tell you what the exact space complexity of training an FFNN with GD and BP is (because that actually depends on the specific implementation of GD and BP and I don't want to dive into the details of some specific implementation now, maybe later!), but I will guide you towards the specific answer, which you should be able to figure out alone (although it may take some time because you need to understand all the details of the BP algorithm), if you understand this answer.
The space complexity of an algorithm is just the amount of memory that you need to use during the execution of the algorithm. The space complexity, like the time complexity, is typically expressed as a function of the size of the input (or the number of inputs that you have) and, usually, in big-O notation, i.e. in the limiting case. So, $n$ is not the same thing as $\mathcal{O}(n)$, $\Omega(n)$ or $\Theta(n)$. Moreover, you can also express the space/time complexity both in the worst, best, or average case, and this is orthogonal to upper (expressed with $\mathcal{O}$), lower ($\Omega$), or tight ($\Theta$) bounds (check this).
If you use gradient descent (GD) and back-propagation (BP) to train an FFNN, at each training iteration (i.e. a GD update), you need to store all the matrices that represent the parameters (or weights) of the FFNN, as well as the gradients and the learning rate (or other hyper-parameters). Let's denote the vector that contains all parameters of the FFNN as $\theta \in \mathbb{R}^m$, so it has $m$ components. The gradient vector has the same dimensionality as $\theta$, so we need at least to store $2m + 1$ parameters.
Depending on how you implement BP, you may need more memory. For example, if you need to store all the intermediate terms of the partial derivatives, that will require more memory. To compute exactly the amount of required memory, you will have to expand the gradient vector into all their components (which may not be a pleasant experience). As I just said, this only contributes to the space complexity if you need to store these intermediate components, so, ultimately, the space complexity of an algorithm depends on the specific implementation of the algorithm.
Moreover, to be precise, we cannot just say that the space complexity is $2m + 1$ or whatever the amount of memory that you require is (although many careless or ignorant people will just say that), because we are not expressing this complexity as a function of the size of the input in the limiting case (which is usually done when expressing the space complexity of an algorithm), the number of layers or the number of units per layer (and you probably want to express the space complexity as a function of these 3 possible variable hyper-parameters).
If you take a look at this answer, where I describe how to compute the time complexity of the forward pass of an MLP (or FFNN) as a function of the number of inputs and outputs, the number of layers, and units per layers, then you can express the space complexity for training an FFNN in the same way. Given that you are already familiar with how space and time complexities of an algorithm are calculated (and given that this answer is already quite long), I will not repeat the description here.
In any case, to answer one of your questions more directly, yes, the space complexity will depend on the number of inputs that you have, because the number of inputs will determine the number of weights in the first layer, which you need to store in memory. This is true in the case of FFNNs (or MLPs) but note that this would not be true in the case of CNNs (i.e. the number of parameters in the convolutional layers does not depend on the size of the input), and that's why CNNs are often said to be more memory efficient.


Non-Neural Network algorithms for large state space in zero sum games I was reading online that tic-tac-toe has a state space of $3^9 = 19,683$. From my basic understanding, this sounds too large to use tabular Q-learning, as the Q table would be huge. Is this correct?
If that is the case, can you suggest other (non-NN) algorithms I could use to create a TTT bot to play against a human player?

I was reading online that tic-tac-toe has a state space of $3^9 = 19,683$. From my basic understanding, this sounds too large to use tabular Q-learning, as the Q table would be huge. Is this correct?

That is a relatively small number of states that can easily be represented in a table on a modern computer. For a Q table, you would multiply by the number of moves possible in each state, but this is still a small amount of memory. Even with the most naive implementation that tracked impossible states and state/action pairs, using string state representations (so 10 bytes for each state/action key), and using double precision floating point for each action value (another 8 bytes), the full table would be around 3 megabytes in size.
So it is definitely possible to use tabular Q learning here. I have done just that whilst learning about RL - my Q-learning Tic Tac Toe agent is written in Python and available on Github. There are many ways to optimise the required space, e.g. only representing reachable states. Also in games with perfect control (moves directly result in a new state), it is common to use afterstate values instead of action values.

If that is the case, can you suggest other (non-NN) algorithms I could use to create a tic-tac-toe agent to play against a human player?

It is not the case, as explained. However, there are two basic approaches at the top level that are worth learning about for working with game trees:

Forward planning or search algorithms, as mentioned by skillsmuggler in their answer. These use game rules plus a heuristic (measure of likely success) to explore the future states of the game and pick the search result with the best heuristic. Minimax with a heuristic only based on win/lose at the end is a very basic approach, but would cope just fine with Tic Tac Toe.

Other planning algorithms include Negamax (a minor variation of Minimax for zero-sum games) and MCTS (famously used in Alpha Go).

Policy function improvement, which generate policies - maps of current state to actions - and have ways to assess them and select better policies.

Q learning, and the many algorithms of reinforcement learning are in this category, as well as genetic algorithms.
Tic Tac Toe is simple enough that you could hard-code a policy function that mapped any state to the next action. The xkcd webcomic encoded this policy into a diagram.

These two approaches are complementary, in that both can be used together to solve more sophisticated problems. For instance, value-based reinforcement learning algorithms - including Q learning - can be used to provide learned heuristics for a search algorithm. The categories I suggest above are not strict either, in that some algorithms are not clearly one thing or another.


Are there any good alternatives to an LSTM language model for text generation? I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be:

Apply softmax function
Take a weighted random choice to determine the next word

This is working reasonably well for me, but it would be nice to play around with other options. Are there any good alternatives to this?

The current state of the art in natural language generation are all auto-regressive transformer models. Transformers no longer use recurrent neural networks such as LSTM, because the recurrences makes long dependencies messy to calculate. Instead, Transformers only keep the attention layers, and apply attention on all the existing text so far, which can be done in parallel so therefore very fast, while being able to attend to long dependencies (e.g. understanding that "it" refers to "John" from 3 sentences ago). They are also faster to train than LSTMs (on powerful GPUs at least). The downside is more memory requirement, and you need large models and large datasets (LSTMs work better for small models and small datasets). Here is some background info on how they work.
Auto-regressive transformer models only use the decoder for text generation, and removes the encoder. Given an input, they predict the next word.
The most well-known one is GPT (GPT-3 has 175B parameters; GPT-2 has 1.5B parameters, and GPT-1 has 175M parameters) GPT is developed by OpenAI and is a commercial, paid-software if you want to use their official model, but I'm sure with a little digging you can find community-trained models that will perform slightly worse but is at least free to use. GPT is basically a vanilla transformer, but trained on a huge, huge dataset with a huge, huge model to achieve state-of-the-art performance.
Other auto-regressive transformer models include:

CTRL by Salesforce, which uses the novel idea of control codes to guide the style of generation (e.g. to generate Wikipedia article style text or book review style text).
XLNet by Google AI Brain team, which handles longer sequences more accuractely than the others because it re-introduces reccurrence back in the transformer model, allowing it to remember past sequences. Otherwise, vanilla transformers cannot handle dependencies that crosses sequences (note: a sequence is limited by the max length you can feed into the model, bottle-necked by your memory requirement, and can contain many sentences or paragraphs).
Reformer by Google Research, which is a more efficient transformer that significantly reduces the memory requirement while also being faster to compute on long sequences.

If your goal is just to generate English or another commonly researched language, you can use an existing pre-trained language model and avoid doing any training yourself. This saves a lot of time, and there should at least be free community-trained models readily available. Otherwise, for obscure tasks, you'd have to train one yourself, and these state of the art models will take immense resources.


How can I merge two datasets? I want to merge 2 data sets in one, but don't know the right approach to do it. The datasets are similar, the last column is the same - will or not them buy a product. In the first dataset, users who only will buy, in second - only who won't buy.

The 1st dataset contains 500 rows and 2nd 10000 rows. What will be the right approach to merge it? How can I normalize them? And to point for an algorithm that the last column is the main sequence on what it should learn?

Example:

id    income date will_buy

23123 200    10.5 Yes

and second dataset:

id    income date will_buy

2323  100    10.5 No

You can use append function:

final = df1.append(df2, ignore_index=True)

To set the last column as labels, you set them as so by:

labels = np.array(final["will_buy"])

So, when calling the fit method on the model you build, you set labels = labels.


How should I update the weights of a neural network, given the gradient? After watching 3Blue1Brown's tutorial series, and an array of others, I'm attempting to make my own neural network from scratch.

So far, I'm able to calculate the gradient for each of the weights and biases.

Now that I have the gradient, how am I supposed to correct my weight/bias?

Should I:

Add the gradient and the original value?
Multiply the gradient and the original value?
Something else? (Most likely answer)

In addition to this, I've been hearing the term learning rate being tossed around, and how it is used to define the magnitude of the 'step' to descend to minimum cost. I figured this may also play an integral role in reducing the cost.

Consider that you have a loss function, and you want to tune your model (network) to decrease the loss. The main concept is to tune parameters in a direction which decreases the loss and gives you a better model. You can imagine a mountain where you should reach to the lower grounds.

There are 2 questions here.

In which direction to move?
How much should we move in that direction?

1. In which direction to move?

By "move" I mean tuning the parameters and therefore changing the model. If you are familiar with the concept of slope or gradient in mathematics, you should move in the direction where the slope is downward the most. The gradient shows the direction with the most slope upward. So, we should move in the opposite of gradient direction. So, we should minus the gradient from the original value, hence the negative sign in the formula below.

2. How much should we move in that direction?

This is defined by the learning rate, which is a number that is multiplied by the gradient. You can imagine that, if the learning rate is big, you are taking bigger footsteps in that direction when coming down the mountain. Similarly, when the learning rate is low, you are taking small footsteps coming down the mountain.

So, consider you have $\nabla_w$ (the gradient of the parameters $w$, with respect to the loss function $L$). Let the learning rate be $\gamma$ (for example, $\gamma= 0.1$). The formula to update the parameters would be

$$
w_{\text{new}} = w_{\text{old}} - \gamma * \nabla_{w_{\text{old}}}
$$

Note that gradient descent is not the only way to optimize your model. Many other gradient-based approaches exist, for example, stochastic gradient descent, Adam, RMSprop, conjugate gradients, etc. There other methods, like evolutionary methods (genetic algorithm, for example), that use different concepts.

Also, note that the learning rate does not necessarily have to be fixed, so it can be tuned during the training, if you want or need.


What is the purpose of the "gamma" parameter in SVMs? I want to understand what the gamma parameter does in an SVM. According to this page.

Intuitively, the gamma parameter defines how far the influence of a single training example reaches, with low values meaning ‘far’ and high values meaning ‘close’. The gamma parameters can be seen as the inverse of the radius of influence of samples selected by the model as support vectors.

I don't understand this part "of a single training example reaches", does it refer to the training dataset?

I've summarized the key ideas of SVMs. So this is how $\gamma$ is used with a gaussian Kernel:

$$K_{\text{Gauss}}(\mathbf{x}_i, \mathbf{x}_j) = e^{\frac{-\gamma\|\mathbf{x}_i - \mathbf{x}_j\|^2}{2 \sigma^2}}$$

The bigger the $\gamma$, the more "linear" the decision boundary will be. The closer to 0, the more support vectors you have/the more non-linear the boundary is (see interactive example)

You can also find it on udacity: SVM Gamma Parameter

In practice, you can use a grid search or random search to get good values.


Can non-differentiable layer be used in a neural network, if it's not learned? For example, AFAIK pooling layer in CNN is not differentiable, but it can be used because it's not learning. Is it always true?

It is not possible to backpropagate gradients through a layer with non-differentiable functions. However, the pooling layer function is differentiable*, and usually trivially so.

For example:

If an average pooling layer has inputs $z$ and outputs $a$, and each output is average of 4 inputs then $\frac{da}{dz} = 0.25$ (if pooling layers overlap it gets a little more complicated, but you just add things up where they overlap).
A max pooling layer has $\frac{da}{dz} = 1$ for the maximum z, and $\frac{da}{dz} = 0$ for all others.

A pooling layer usually has no learnable parameters, but if you know the gradient of a function at its outputs, you can assign gradient correctly to its inputs using the chain rule. That is essentially all that back propagation is, the chain rule applied to the functions of a neural network.

To answer your question more directly:

  Can non-differentiable layer be used in a neural network, if it's not learned?

No.

There is one exception: If this layer appears directly after the input, then as it has no parameters to learn, and you generally do not care about the gradient of the input data, so you can have a non-differentiable function there. However, this is just the same as transforming your input data in some non-differentiable way, and training the NN with that transformed data instead.

* Technically there are some discontinuities in the gradient of a max function (where any two inputs are equal). However, this is not a problem in practice, as the gradients are well behaved close to these values. When you can safely do this or not is probably the topic of another question.


How can we get a differentiable neural network to count things? Imagine I have images with apples in them. I want to train a neural network which can count the number of apples in each image.
BUT, I don't want to use a detector, then count the number of bounding boxes. I'm interested in knowing if there's a way to bake this sort of logic into a differentiable neural network.
The simplest variation of this problem might be: I have an input vector $x \in \{0, 1\}^N$ and I want to count the number of 1s in it. I can make a single layer neural network, setting all the weights to 1, bias to 0, and linear activation. And my answer pops out. But how would I train the network to do this from scratch? Sure, I could regress to an output in $[0,1]$ and multiply the result by $N$, then the network is differentiable. But did the model really learn how to count? If so, would this behaviour be generalisable to counting multiple types of objects at once? Would it generalise to inputs where there can be any number of said object (like an image can have many apples in it, despite the size of the image)?
I want to know if there's a model which can learn how to count.
Here's another way I'm thinking about it: I can look at an aerial view of pine trees and say "yeah maybe there are 30 trees", but then I can do the task of looking at and identifying each tree individually, incrementing a counter, and making sure not to revisit that tree. The latter is what I consider truly "counting".

Estimating from an observation is a function, but "really counting" is a process. Feed-forward neural networks can learn arbitrary functions from training examples, but they cannot represent (and therefore cannot learn) processes. They can attempt to estimate the results of completing a process as a function, but that is not the same thing as actually performing the process.
To learn arbitrary processes from examples requires a model with some concept of state and evolution over time in addition to any functions that are required. Recurrent neural networks (RNNs) are suitable models for those kinds of learning problems, but so are other AI learning constructs.
This rabbit hole goes deep, and for any model you could build it is possible to ask:

Is the system really performing a counting task similar to how a human might attempt the same task?

Has the sytem really learned to count, or has it been constructed by the developer so that some parts of the process are inevitable?

Has the system learned to count in general within any sub-component, or are the things that it can count tightly coupled across components?

There are other questions you could ask too, depending on your goals for producing such a model.
It is worth noting that for small quantities and certain patterns, that human "counting" does more closely resemble the estimation process, or perhaps is more akin to an NLP problem. For example, consider how you "count" the pips on a die - although you can use a counting process to confirm what you see, typically you do not do so. Instead, some part of your brain is supplying a very accurate answer quickly and unconsciously:

More generally, counting in humans likely consists of multiple related strategies for converting observations into symbolic and/or sensory representations of quantity. Some may be instinct, some are learned conscious behaviours and some appear to be in-between, perhaps originally learned but turned into subconscious skill through repetition.
Ignoring whether or not you are implementing your AI project for counting items in an image wholly within a neural network for now, I think you need the following components:

A state that represents and tracks the start, progress so far and end of the counting process with respect to the input being processed.

An accumulator that represents quantity counted so far.

A detector that can trigger a counting event when it observes something that needs to be counted.

A strategy or planner for processing input so that detection events are separated and only triggered once for each valid event.

The last component can vary a lot, and humans will use a range of different strategies for counting depending on the difficulty of the task. For instance, you might use working memory (a limited resource in humans) to track a few key points in an image to help segment an image into smaller sections as you work across it. Or you might make visual marks on an image to track each object that had already been counted. For a human, counting strategies can be mixed and matched, and switched between sometimes even during the same counting task.
All of these components could be represented in learnable parts of an AI process, but they do not have to be. When you suggest in your question that

I don't want to use a detector, then count the number of bounding boxes.

then you are most likely saying that you don't want to use a fixed hard-coded strategy. You want to somehow create a neural network that can discover at least one strategy for processing the input.
The trouble you will face is that giving a large RNN model (e.g. LSTM) a dataset of examples with input images and output correct counts will likely be too much of a challenge. Discovery of a robust object counting system from scratch will be too hard.
There are a few things that may help you construct something that "really" learns to count. Here are a couple of ideas:

Curriculum learning, where you start training the neural network with some hand-holding examples, and slowly ramp up the complexity. This is analogous to how we teach children to count.

Designed-in modelling for processing strategy. For instance, you could add a virtual fovea and artificial visual saccades and require that the neural network output a sequence of locations within the image that it picks out to run the detector against. This will be a constraint that allows certain types of human-like counting strategy to work, and could simplify the problem to the point that the network has a chance to learn it.

A paper that uses a "fovea" model for object detection: Object detection through search with a foveated visual system


What is the difference between a stochastic and a deterministic policy? In reinforcement learning, there are the concepts of stochastic (or probabilistic) and deterministic policies. What is the difference between them?

A deterministic policy is a function of the form $\pi_{\mathbb{d}}: S \rightarrow A$, that is, a function from the set of states of the environment, $S$, to the set of actions, $A$. The subscript $_{\mathbb{d}}$ only indicates that this is a ${\mathbb{d}}$eterministic policy.
For example, in a grid world, the set of states of the environment, $S$, is composed of each cell of the grid, and the set of actions, $A$, is composed of the actions "left", "right", "up" and "down". Given a state $s \in S$, $\pi(s)$ is, with probability $1$, always the same action (e.g. "up"), unless the policy changes.
A stochastic policy is often represented as a family of conditional probability distributions, $\pi_{\mathbb{s}}(A \mid S)$, from the set of states, $S$, to the set of actions, $A$. A probability distribution is a function that assigns a probability for each event (in this case, the events are actions in certain states) and such that the sum of all the probabilities is $1$.
A stochastic policy is a family and not just one conditional probability distribution because, for a fixed state $s \in S$, $\pi_{\mathbb{s}}(A \mid S = s)$ is a possibly distinct conditional probability distribution. In other words, $\pi_{\mathbb{s}}(A \mid S) = \{ \pi_{\mathbb{s}}(A \mid S = s_1), \dots, \pi_{\mathbb{s}}(A \mid S = s_{|S|})\}$, where $\pi_{\mathbb{s}}(A \mid S = s)$ is a conditional probability distribution over actions given that the state is $s \in S$ and $|S|$ is the size of the set of states of the environment.
Often, in the reinforcement learning context, a stochastic policy is misleadingly denoted by $\pi_{\mathbb{s}}(a \mid s)$, where $a \in A$ and $s \in S$ are respectively a specific action and state, so $\pi_{\mathbb{s}}(a \mid s)$ is just a number and not a conditional probability distribution. A single conditional probability distribution can be denoted by $\pi_{\mathbb{s}}(A \mid S = s)$, for some fixed state $s \in S$. However, $\pi_{\mathbb{s}}(a \mid s)$ can also denote a family of conditional probability distributions, that is, $\pi_{\mathbb{s}}(A \mid S) = \pi_{\mathbb{s}}(a \mid s)$, if $a$ and $s$ are arbitrary. Alternatively, $a$ and $s$ in $\pi_{\mathbb{s}}(a \mid s)$ are just (dummy or input) variables of the function $\pi_{\mathbb{s}}(a \mid s)$ (i.e. p.m.f. or p.d.f.): this is probably the most sensible way of interpreting $\pi_{\mathbb{s}}(a \mid s)$ when you see this notation (see also this answer).
In the particular case of games of chance (e.g. poker), where there are sources of randomness, a deterministic policy might not always be appropriate. For example, in poker, not all information (e.g. the cards of the other players) is available. In those circumstances, the agent might decide to play differently depending on the round (time step). More concretely, the agent could decide to go "all-in" $\frac{2}{3}$ of the times whenever it has a hand with two aces and there are two uncovered aces on the table and decide to just "raise" $\frac{1}{3}$ of the other times.
A deterministic policy can be interpreted as a stochastic policy that gives the probability of $1$ to one of the available actions (and $0$ to the remaining actions), for each state.