Can I combine two classifiers that make different kinds of errors to get a better classifier? I have a dataset with 2,23,586 samples out of which i used 60% for training and 40% for testing. I used 5 classifiers individually, SVM, LR, decision tree, random forest and boosted decision trees. SVM and LR performed well with close to 0.9 accuracy and recall also 0.9 but tree based classifiers reported an accuracy of 0.6. After a careful observation, I found out that SVM and LR did not predict the labels of 20,357 samples identically. So Can I apply voting and resolve this conflict wrt prediction outcome? Can this conflict be due to an imbalanced dataset?

Yes, you can. There are a lot of different techniques, usually called Ensemble Methods.

A better approach might be to use something like AdaBoost along with a cheaper method like the decision trees you looked at. AdaBoost explicitly tries to train classifiers to correctly handle different parts of the data, rather than hoping that different methods turn out to do so by chance.


Assigning probability to output of a ReLU network I'm using a NN created with CNTK's SimpleNetworkBuilder to make choices (specifically in board games). I specified ReLU as the layer type, so outputs can be arbitrary numbers.

When evaluating a custom set of features, getting the "choice" of the function/model is simple: Look for the output signal with the highest value. However, there are times when I wish to introduce some randomness and assign probabilities to each output signal, then select the choice based on each output's probability.

Currently, what I'm doing is manually normalizing all the output using a sigmoid function specified here: https://en.wikipedia.org/wiki/Logistic_function
Then, I multiply them all by a scalar such that the sum total of all outputs is 1.

At this point, I pick a random number 0..1, and see where along the map it falls; that is my selected choice.

What I'd like to know is, is there a better way?

I would just skip the sigmoid function step.

Consider these two scenarios with three choices with the associated values:
0.1,10,100 or 1,100,1000
Given that these scenarios are equivalent in their relativ values, your probabilities should be assigned in the same way. But if you throw in a sigmoid function, the difference between 10 and 100 will be bigger than between 100 and 1000.

Just normalise the values, so that they sum to 1. In that case the choice with a value ten times higher than the value of another choice will be picked ten times as often. Makes sense to me.


Keras : get back labels from a model I have a saved keras model.
How can I get back the labels from the model ?

Because right now, I can use the predict method to get back the probability for a sample to belong to a certain class e.g. class 1, 2, 3.

But how can I know what class 1, 2, 3 correspond to in the model e.g. cat, dog, bird ?

You can use the argmax function to get the class labels. Please refer to this link for more information. Hope this answers your question.


Do models train better if the labelling information is more specific (or dense)? I'm working on a project where there is a limited dataset of videos (about 200). We want to train a model that can detect a single class in the videos. That class can be of multiple different types of shapes (thin wire, a huge area of the screen, etc).
There are three options on how we can label this data:

Image classification (somewhere in the image is this class)
Bounding box (in this area, there is the class)
Semantic segmentation (these pixels are the class)

My assumption is that if the model was trained on semantic segmentation data it would perform slightly better than bounding box data.  I'm also assuming it would perform way better than if the model only learned on image classification data.  Is that correct?

It depends on what is your ultimate goal. If your goal is to simply classify the object in the image, having more complex output won't help. Simpler output representation yields better result. If your goal is to detect the bounding box, output the bounding box. There is no need for a more complex output feature. If you use a segmentation method for bounding box detection, it is more prone to error because of it's excess output features.

Assume you are given a gradr 6 math test. If you do the questions using grade 12 maths knowledge and do it with calculus and stuff to make your calculations seems more complex, will you have a higher mark than doing it the normal way? No! The marks is the same or even less due to higher chance of error in doing complex calculations.

In short, higher complexity on your labels won't help your task if it is a simple task. Hope this would help you and have a nice day!


Is there a family tree for reinforcement learning algorithms? Can anyone point me in the direction of a nice graph that depicts the "family tree", or hierarchy, of RL algorithms (or models)? For example, it splits the learning into TD and Monte Carlo methods, under which is listed all of the algorithms with their respective umbrella terms. Beneath each algorithm is shown modifications to those algorithms, etc. I'm having difficulty picturing where everything lies within the RL landscape.

I highly recommend looking at Reinforcement Learning: An Introduction by Richard Sutton and Andrew Barto.

In it they write:

  Reinforcement learning, like many topics whose names end with “ing,” such as machine
  learning and mountaineering, is simultaneously a problem, a class of solution methods
  that work well on the problem, and the field that studies this problem and its solution
  methods. It is convenient to use a single name for all three things, but at the same time
  essential to keep the three conceptually separate. In particular, the distinction between
  problems and solution methods is very important in reinforcement learning; failing to
  make this distinction is the source of many confusions.

On the solution side of things, rather than a family tree of reinforcement learning, there is a spectrum of different approaches to the problem.

Many of the points within the spectrum above are TD($\lambda$) methods. When the state space is large, one might use a neural network to help generalize across similar states, such as DQN for instance.

It should be noted that there are many applications of reinforcement learning algorithms that do not involve training an agent to perform a task. Instead one might want to evaluate an existing agent or use a general value function to predict an event using an agent's policy.

Nevertheless, many popular problems consist of training an agent to perform a task. These algorithms can be categorized into a "family tree" based either on what values they are updating or on what problem they are tackling. (Source : https://github.com/NervanaSystems/coach)

On the left are Value Based methods which update the value function of a policy and find policies that optimize said value function.
To the right of those are policy gradient methods which optimize for the policies directly without explicitly finding a value function.
To the far right is Imitation Learning whereby one wants to copy a policy given demonstrations of an existing but unacceptable policy.
And in the middle is Direct Future Prediction which is its own weird beast that I found out while finding that picture. Here is the arxiv paper that describes DFP.


Creating videos of AI generated photographs I came across this article today: These faces show how far AI image generation has advanced in just four years. I would never in a million years have guessed that the people on the right (in the first image in the article) were fakes!

Will it be possible to create videos of such AI generated images? What, then, will become of actors and actresses?

I wouldn't really consider what they are doing as AI - they are using a script that intelligently overlaps various images of existing people in order to create a new face.

Animating those images isn't impossible - essentially extrapolation + additional "real" images will be used to know what the face would like from all angles and in all states (happy, sad, intrigued, etc...).


During neural network training, can gradients leak sensitive information in case training data fed is encrypted (homomorphic)? Some algorithms in the literature allow recovering the input data used to train a neural network. This is done using the gradients (updates) of weights, such as in Deep Leakage from Gradients (2019) by Ligeng Zhu et al.
In case the neural network is trained using encrypted (homomorphic) input data, what could be the output of the above algorithm? Will the algorithm recover the data in clear or encrypted (as it was fed encrypted)?

It will recover the encrypted inputs.
The algorithm starts with dummy data and dummy labels, and then iteratively optimizes the dummy gradients to be close as to the original. This makes the dummy data close to the real training data:
$$\mathbf{x}^{\prime *}, \mathbf{y}^{\prime *}=\underset{\mathbf{x}^{\prime}, \mathbf{y}^{\prime}}{\arg \min }\left\|\nabla W^{\prime}-\nabla W\right\|^{2}=\underset{\mathbf{x}^{\prime}, \mathbf{y}^{\prime}}{\arg \min }\left\|\frac{\partial \ell\left(F\left(\mathbf{x}^{\prime}, W\right), \mathbf{y}^{\prime}\right)}{\partial W}-\nabla W\right\|^{2}$$
As the distance is minimized, the algorithm restores the original training data; in the case of encrypted training data - you should get an encrypted input (up to failures when the resulted 'input' isn't close to the original input).


Recommend item from set based on features Forgive what might be a basic question.  I'm just experimenting with ML / AL and I have a small problem set and I'd like to see if it can be solved with ML / AI.  Basically, given a set of objects with multiple features, I'd like to create a process for recommending one automatically to a user.

I'm thinking that some sort of clustering algorithm may be the best approach. However, one main challenge I'm trying to wrap my head around is that I don't know in advance how many distinct clusters will evolve... There may be scenarios where we Feature X is really important, but other scenarios where a user will say Feature Y is important.

Secondly, what is my input set?  For each training sample, I will have 1 selected object, and N-1 unselected objects.  But I don't want to "train" that the unselected objects are "bad" because they could be selected in a future training example.

Finally, I don't have a large training set already, so I would like to use feedback (user input, "This was a bad choice" or "Use this object instead.") from the process to further refine the algorithm.  Is this feasible?

Are there any established patterns for this sort of process?  Thanks in advance.

This problem is usually approached with "Singular Value Decomposition".

Search also for the "Netflix Challenge".


Use of machine learning for analyzing companies enlisted in stock market Can current trends and tools, in the field of machine learning, replicate the complexity of financial market? If yes, then what are the tools available in this domain.

Q. I am trying to build a model to infer results from stock market using the concept to create a graph on the companies enlisted. Can anyone suggest me approaches to do so?

In one way the answer is NO. You can't incorporate all the necessary details in an AI program to correctly predict the financial market, at least not in currently. A particular event is a result of many actions and events from the past. Everything is like a chain.
You might have heard of the ButterFly Effect. It simply states that small things can have very large effects. Like flap of butterfly's wings leading to a hurricane. You can read more about it here.
Now back to your question. I believe you are somewhat familiar with the financial system and stocks. Close your eyes and try to imagine all the data that is being churned into machines. Apart from that, there are many decisions that lead to changes in the financial market directly. Like policies made by the government deals bagged by the company related to imports and exports, death of the main employee (like CEO, founder), and other many things. Apart from that, there are uncountable things that indirectly affect the market. I want you to think about it for a while. Now consider doing this for all the stocks present. After even stock values of one are dependent on the others in one way or another.
This amount of data, in other ways considering all these things, is not possible for current generation computers to process. You might have a hit with quantum computers coming though.
Now what you can do, is try to make a Deep Network (with Reinforcement Learning) that can predict the behavior and shift in the direction of some stocks. For example, whether it will go up for down or try to keep a constant line on the graph.
Selection of Features is always an important key to the success of Machine Learning algorithms. Another problem that stock market-related AI programs face is that Economics is itself an emerging and novice field of science. It's not as mature as physics, chemistry or biology. Thus finding out and deciding over the choice of features is another difficult job.
I found this paper online. Haven't given a look to it but, I believe this might help you a bit.


Understanding Bayesian Optimisation graph I came across the concept of Bayesian Occam Razor in the book Machine Learning: a Probabilistic Perspective. According to the book:

  Another way to understand
  the Bayesian Occam’s razor effect is to note that probabilities must
  sum to one. Hence $\sum_D'  p(D' |m) = 1$, where the sum is over all possible data sets. Complex
  models, which can predict many things, must spread their probability mass thinly, and hence
  will not obtain as large a probability for any given data set as simpler models. This is sometimes called the conservation of probability mass principle.

The figure below is used to explain the concept:

    Image Explanation: On the vertical axis we plot the predictions of 3 possible models: a simple one, $M_1$ ; a medium one, $M_2$ ; and a complex one, $M_3$ . We also indicate the actually observed
    data $D_0$ by a vertical line. Model 1 is too simple and assigns low probability to $D_0$ . Model 3
    also assigns $D_0$ relatively low probability, because it can predict many data sets, and hence it
    spreads its probability quite widely and thinly. Model 2 is “just right”: it predicts the observed data with a reasonable degree of confidence, but does not predict too many other things. Hence model 2 is the most probable model.

What I do not understand is when a complex model is used, it will likely overfit data and hence the plot for a complex model will look like a bell shaped with its peak at $D_0$ while simpler models will more likely have a broader bell shape. But the graph here shows something else entirely. What am I missing here?

The original graph for the aforementioned Bayesian Optimisation is similar to the graph in these slides (slide 18) along with the calculations.

So, according to the tutorial the graph shown should actually have the term $p(D|m)$ on the y-axis, thus making it a generative model.Now the graph starts to make sense, since a model with low complexity cannot produce very complex datasets and will be centred around 0, while very complex models can produce richer datasets which makes them assign probability thinly over all the datatsets (to keep $\sum_{D'}p(D'|m) = 1$).


How to train a LSTM with multidimensional data I am trying to train a LSTM, but I have some problems regarding the data representation and feeding it into the model.

My data is a numpy array of three dimensions: One sample consist of a 2D matrix of size (600,5). 600(timesteps) and 5(features). However, I have 160 samples or files that represent the behavior of a user in multiple days. Altogether, my data has a dimension of (160,600,5).

The label set is an array of 600 elements which describes certain patterns of each 2D matrix. The shape of the output should be (600,1).

My question is how can I train the LSTM to the corresponding label set? What would be the best approach to handle this problem? The idea is that the output should be an array of (600,1) with 3 label inside.

Multiple_outputs {0,1,2}
      Output:    0000000001111111110000022222220000000000000
                 -------------600 samples ------------------

Input: (1, 600, 5)
Output: (600, 1)
Training: (160,600,5)

I look forward for some ideas!

dataset(160,600,5)

X_train, X_test, y_train, y_test = train_test_split(dataset[:,:,0:4], dataset[:,:,4:5],test_size = 0.30)

model = Sequential()
model.add(InputLayer(batch_input_shape = (92,600,5 )))
model.add(Embedding(600, 128))
#model.add(Bidirectional(LSTM(256, return_sequences=True)))
model.add(TimeDistributed(Dense(2)))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer=Adam(0.001),
              metrics=['accuracy'])

model.summary()

model.fit(X_train,y_train, batch_size=92, epochs=40, validation_split=0.2)

I don't see any special characteristic in the problem you're posing. Any LSTM can handle multidimensional inputs (i.e. multiple features). You just need to prepare your data such as they will have shape [batch_size, time_steps, n_features], which is the format required by all main DL libraries (pytorch, keras and tensorflow).

I linked below 2 tutorials that shows how to implement an LSTM for part of speech tagging in Keras and Pytorch. This task is conceptually identical to what you want to achieve: use 2D inputs (i.e. embeddings) to predict the class (i.e. the pos tags) of each element of a sequence (i.e. every single word). For example:

# Input sample, 4 steps, encoded in 2D embeddings matrix
['we', 'had', 'crispy', 'fries']

# Output predictions, 4 labels
['PRP', 'VBD', 'JJ', 'NNS']

In your case the predicted label would be 0, 1 or 2 and you would have to encode your data in a matrix of shape [n_batch, 600, 5]. The only thing you definetly want to pay attention, since you're using temporal data, is to not shuffle your data at all before the training.

Keras tutorial: Build a POS tagger with an LSTM using Keras

Pytorch tutorial: LSTM’s in Pytorch


Is there any specific SW framework, libraries or algorithms (supported by any theory) designed for implementing a practical AGI system? Any (AGI)-KERAS like libraries? Any deep-learning framework to develop AGI applications?
Existing frameworks/algorithms used in NN, NLP, ML, etc are not enough in my opinion. In my opinion any framework has to be based on building blocks from: Cognitive Science, Neuroscience, Mathematics, Artificial Intelligence, Computer Science, psycology, sociology, etc.

Not to my knowledge. The problem is that this is such an enormous task, it cannot really be tackled at once. So the obvious solution is to reduce the scope. In early AI people were using toy domains, whereas nowadays AI systems work more generally (but still perform better if the domain is restricted).
So while (slow) progress is being made putting the individual building blocks together, we're still a long way off building an overarching general system from them.


When is it necessary to manually extract features to feed into the neural network rather than providing raw data? Usually, Neural Networks uses raw data. You do not need to extract features manually. NN's can find & extract good features which is a pattern of an image, signal or any kind of data. When we check layer outputs in a NN, we can see and visualize how NNs extract features.
Do neural networks extract features by themselves every time? When is it necessary to manually extract or engineer features to feed into the neural network rather than providing raw data?
For example, I had a time series sensor data. When I use LSTM & GRU on a raw dataset, I had bad test accuracy but when I extract some features manually I had really good test set accuracy results. I extract Fast Fourier Transform, Cross-correlation features which helped a lot to increase accuracy. "Extraction of features manually" helped to solve my problem.

Feature engineering may be necessary when one cannot achieve acceptable error rate — within a budget or in principle.
NN may be stalling due to information bottleneck: too many pigeons, not enough holes. In that case, custom features may provide slightly better information compression. (Alas, this is not a panacea: some layer(s) may still be too narrow. That's why starting fat and pruning later deems superior, although not always affordable.)
Not a necessity, but a no-brainer nonetheless: having strong insight into underlying processes is a clear call for feature engineering. Let's say, it's apparent that known meaningful transformation can hardly be approximated by (reasonably small) subnetwork: mapping between periodic and non-periodic functions, as an example.
Personally in my check-list, feature stage is coupled with a reminder to search for a domain expert. In some practical sense features are being extracted from people rather than data. Our neural networks are pretty good too!


Which online machine learning technique to use for multi-class classification problem with multiple inputs? I have the following problem. We have $4$ separate discrete inputs, which can take any integer value between $-63$ and $63$. The output is also supposed to be a discrete value between $-63$ and $63$. Another constraint is that the solution should allow for online learning with singular values or mini-batches, as the dataset is too big to load all the training data into memory.

I have tried the following method, but the predictions are not good.

I created an MLP or feedforward network with $4$ inputs and $127$ outputs. The inputs are being fed without normalization. The number of hidden layers is $4$ with $[8,16,32,64]$ units in each (respectively). So, essentially, this treats the problem like a sequence classification problem. For training, we feed the non-normalized input along with a one-hot encoded vector for that specific value as output. The inference is done the same way. Finding the hottest output and returning that as the next number in the sequence.

I suggest using Data Stream algorithms to try on your problems since you are asking for "online learning with singular values or minibatches as the dataset is too big too load all the training data into memory."

MOA is a good choice for these algorithms. Hoeffding Trees is also a good first  choice to try.


How are the observations stored in the RNN that encodes the state? I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like this and this. If I were to use a sequence of raw observations (or features) as an input to RNN for encoding the state of the system, I cannot change the weights of my network in the middle of the episode. Is that correct? Otherwise, the hidden state vectors will be different when the weights are changed.

Does that mean that the use of RNN in RL has to store the entire episode before the weights can be changed?

How does then one take into account the hidden states in RNN for RL? Are there any good tutorials on RNN-RL?

This research question seems to be analyzed in further details here (section 3) - https://openreview.net/pdf?id=r1lyTjAqYX.

Usually, a sequence is taken as a state to be fed into RNN to compute the final hidden state. One can then ask what initial state should the RNN be seeded with? This paper analyzes three methods with respect to the seed -

zero initialization: When the RNN is initialized with the zero state
burn-in: when the sequence is prepended by some preceding observations for RNN to learn a good initial state
storing the initial hidden state: When the hidden state at the beginning of the sequence is stored


How we are calculating average reward ($r(\pi)$) if the policy changes over time? In the average reward setting the quality of a policy is defined as:
$$ r(\pi) = \lim_{h\to\infty}\frac{1}{h} \sum_{j=1}^{h}E[R_j] $$
When we reach the steady state distribution we can write the above equation as follows:
$$ r(\pi) = \lim_{t\to\infty}E[R_t | A \sim \pi] $$
We can use the incremental update method to find $r(\pi)$:
$$ r(\pi) = \frac{1}{t} \sum_{j=1}^{t} R_j = \bar R_{t-1} + \beta (R_t - \bar R_{t-1})$$
where $ \bar R_{t-1}$ is the estimate of the average reward $r(\pi)$ at timestep $t-1$.
We use this incremental update rule in the SARSA algorithm:

Now, In this above algorithm, we can see that the policy will change with respect to time. But to calculate the $r(\pi)$, the agent should follow the policy $\pi$ for a long period of time. Then how we are using $r(\pi)$ if the policy changes with respect to time?

You are correct: to evaluate a policy, we need to fix it.

We can temporarily fix it, just to evaluate it over a number of test cases. For a fair comparison, we should fix the start states and random seeds used for the transitions.
We can wait until convergence / until we are satisfied. The resulting policy would be what we implement in the "true", trained agent. This is important when exploration might be harmful in the "real world" domain where the agent will be operating.
We can also measure average reward of the "non-stationary" policy, and assume that, once the agent is doing well, this should be close enough to evaluating the fixed policy. This is not ideal, but on the other hand it is trivial to implement, and is often used to track the learning process. If you have a life-long learning agent, this might be the best you can do.


What are the applications of ant colony optimization algorithms? I'm interested in ant colony optimization algorithms and bee algorithms,but I'm confused what are the applications of these algorithms

Can you suggest me some examples of applications can I work on?

The first ant colony optimisation algorithm was introduced by Marco Dorigo in the report Positive Feedback as a Search Strategy (1991) and his PhD thesis Optimization, Learning and Natural Algorithms (1992). He's still one of the leading figures in the field of swarm intelligence (having also written or co-written several papers and books). Another important person that contributed to ACO algorithms is Luca Gambardella (co-director of IDSIA).

There are several ACO algorithms. They are all based on the way real ants behave, that is, by leaving a substance called "pheromone" on the ground in order to communicate. More specifically, the amount of pheromone is associated with value (e.g. food): more pheromone means more value. (It should now be clear the reason behind the queues real ants form).

A list of ACO algorithms can be found at http://iridia.ulb.ac.be/~mdorigo/ACO/publications.html. For reproducibility, here's a (non-exhaustive) list:

Ant System
Elitist Ant System
Ant-Q
Ant Colony System
Max-Min Ant System
Rank-based Ant System
ANTS
Hyper Cube - ACO

ACO algorithms have been applied to combinatorial and NP-complete (e.g. the travelling salesman problem) problems. ACO algorithms are thus a collection of meta-heuristic and probabilistic algorithms (in the same family of simulated annealing) to tackle often considered intractable problems. The related Wikipedia article contains a more exhaustive section dedicated to the applications of these algorithms. ACO algorithms are often combined with local search algorithms (like the 2-opt or 3-opt).

I would suggest you to start with the travelling salesman problem, which was the first application of these algorithms. You can have a look at the reference implementations at http://iridia.ulb.ac.be/~mdorigo/ACO/aco-code/public-software.html, where you can also find software to solve specific tasks (not just the TSP, such as maximum clique problems).


Does leaky relu help learning if the final output needs negative values? In an MLP with ReLU activation functions after each hidden layer (except the final),

Let's say the final layer should output positive and negative values.

With ReLU intermediary activations, this is still possible because the final layer, despite taking in positive inputs only, can combine them to be negative.

However, would using leaky ReLU allow faster convergence? Because you can pass in negative values as input to the final layer instead of waiting till the final layer to make things negative

In short, yes Leaky Relu helps in faster convergence if your output requires both positive and negative values. But the catch is that you need to tune the negative slope of Leaky Relu which is a hyperparameter to get better accuracy.


How can a single sample represent the expectation in gradient temporal difference learning? I was reading the gradient temporal difference learning version 2(GTD2) from rich Sutton's book page-246. At some point, he expressed the whole expectation using a single sample from the environment. But how a single sample can represent the whole expectation.

I marked this point in this image.

In the, presumably final, printed version the last two equal signs are approximations. This is just because over a large amount of weight updates where you have been sampling the expectation will be approximated by Monte Carlo.


Why is the fraction of time spent in state $s$, $\mu(s)$, not in the update rule of the parameters? I am reading "Reinforcement Learning: An Introduction (2nd edition)" authored by Sutton and Barto. In Section 9, On-policy prediction with approximation, it first gives the mean squared value error objective function in (9.1):
$\bar{VE}(\boldsymbol{w}) = \sum_{s \in S} \mu(s)[v_{\pi}(s) - \hat{v}(s,\boldsymbol{w})]^2$. (9.1)
$\boldsymbol{w}$ is a vector of the parameterized function $\hat{v}(s,\boldsymbol{w})$ that approximates the value function $v_{\pi}(s)$. $\mu(s)$ is the fraction of time spent in $s$, which measures the "importance" of state $s$ in $\bar{VE}(\boldsymbol{w})$.
In (9.4), it states an update rule of $\boldsymbol{w}$ by gradient descent:
$\boldsymbol{w}_{t+1} = \boldsymbol{w} -\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$. (9.4)
I have two questions regarding (9.4).

Why $\mu(s)$ is not in (9.4)?
Why is it the "minus" instead of "+" in (9.4)? In other words, why is it $\boldsymbol{w} -\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$ instead of $\boldsymbol{w} +\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$?

$\mu(s)$ is not in equation (9.4) because we are assuming that the examples by which we update our parameter $w$, i.e. the frequency of which we will observe the states during online training, is the same. That is, it is a constant with respect to $w$ and since we are differentiating it can be somewhat disregarded as a constant of proportionality -- it essentially can be 'absorbed' by $\alpha$.

The minus is there because we are performing gradient descent. For more information on this, see e.g. the wikipedia page


How do the achievements met in the gaming field (ex. AlphaGo Zero) impact other fields of application? How can we use the ability of AlphaGo Zero computer, to do something in any other life important related field? Is it possible to make something important besides having created something so smart that can play mind games way better than humans?

Yes it's created something important. Until Alpha(Go) Zero all (or almost all) of Deep Learning approach to Reinforcement Learning was based on Time Difference loss function. The weakness of Time Difference loss function was that it was essentially training on itself, that is data produced by the same method was used as part of regression target. That was producing the problem of "extrapolation error" - solution would blow up, or oscillate wildly. There was attempts to mitigate that problem (n-steps algorithms), but they weren't helping much. Alpha Zero combined Deep Network with tree search instead (Monte Carlo Tree Search). Tree search algorithm produced wide and long fields of high precision data (value function), with network influence on value much diminished. That way network was training mostly not on itself, but on data produced by tree, and tree search itself was accelerated greatly by network (using it as heuristics). The whole happens to be much more then sum of parts.

This approach is not limited to broad game or RL theory. It may work for any problem for which high-precision simulator could be built. Essentially if problem allow Monte Carlo Tree Search, or other tree search which could be augmented with heuristic, Alpha Zero approach would probably work on it. Of cause Alpha Zero approach is computationally expensive, so it's not always efficient to apply it.


When training an object detection network for one class, should I include empty images in the dataset? I fine tuned MobileNetSSD for object detection using a dataset with just one class (~4000 images). All the training images include at least one bounding box related to that class (no empty images). By following the example with the VOC dataset, the labelmap includes two classes, the background and my custom class. However, as I mentioned, there are no annotations related to the background and I  am not sure if there should be any.

Now my fine tuned network performs very well when objects belonging to my class are present, however there are some false detections with very high confidence when the class is not present. Can this be related to the fact that I don't have empty images in my training set?

It is correct that to eliminate the false positives, the training must include images that reflect true negative cases. The objective of training is not this.

  Find the objects of class $O$ in scene $X$.

It is this.

  Find any objects of class $O$ in scene $X$.

Training is effective when sample $S$ is drawn according to the same distribution as will be used for drawing at test time or during actual use.

It is not necessary to annotate the background if the loss function is chosen so that finding nothing when nothing there is indicated as a zero loss and a proper distribution of those cases exist in the training sample.


Using AI to interpret the nature a specific input and use the correct model How would one go about building an AI that is capable to look at any kind of input and then identify what is the nature of this data?

For example, an AI that is able to do image classification, NLP and react to some other sensors. Is it possible to build an AI that will be able to identify what kind of data it is seeing such that it can send the data to the correct model for it to be treated, similarly to how the human brain knows to send visual information to the visual cortex and auditory information elsewhere?

In a simple scenario, I think we can get very good performance by having a cascaded image classifier. For example 2 layers, the first layer identifies if the image contains a dog and a cat. The next layer, has two different CNNs, one trained to identify the breed of dog and the other one for cats. That way once we identify that we have a dog, the image can be sent to the correct CNN. A CNN that is trained specifically to detect the breed, thus being much more robust that a more generalized CNN. Kind of like a professional in the field. First, the human identifies that he is looking at a dog, then he consults a professional to ask him the breed.

I would like to extend this idea to be able to identify various kinds of data sources that do not resemble each other at all. Various input. Are there any models that can do this?

Because of all the inputs you could give are obviously a list of number. It will only require some CNNs and fully-conncted, that classify what kind of data is it and then pass the data to the right stack of layers. The only tiny problem is that all the the things you pass into the network must always be a matrix(to handle also images) with a certain size. So even if you  pass only a tiny chunk of data you must transform it into a  fixed-size matrix and fill all the empty places with 0s, and that could bring to an accuracy lack.

But to solve that you could also try to slide over your data a tiny CNN-RNN to handle different sizes .


Predicting a day's data I have a dataset containing timestamp and temperature. For each day, I have 1440 values viz., I have data for every minute of that day(60minutes * 24hrs = 1440).

The Dataset looks like this:

As an initial step, I gathered day1 data to predict day2 data. I have tried AR, ARIMA, SARIMAX models but I didn't find any positive results. I think this is multivariate since the time and the temperature values changes with respect to date. I need guidance to choose the ML model that will suit for my dataset and it should be able to predict next day/ next month

Just for clarification: your description (1 sample per minute) does not match the example data (far fewer data points which is understandable, but also two data points in one minute which contradicts the initial assertion.) If your actual measurements are like that you should first work on the sampling process to get reliable data.

For creating predictions, you need to have a reasonable model of the observed process. If you're measuring environmental temperatures, you will basically have three causes of variation:

A day/night cycle
A seasonal (summer/winter) cycle
Local weather fluctuation

From only one day of samples, the only thing you can reasonably predict is that the next day will look mostly the same. If you collect more data over a year, you will be able to extract a seasonal cycle and estimate the deviations caused by local weather. "You" means either you as a researcher or any machine learning system that you program. Without sufficient historical data it is impossible to make good predictions (and even with sufficient data it's hard.)


How exactly can ReLUs approximate non-linear and curved functions? Currently, the most commonly used activation functions are ReLUs. So I answered this question What is the purpose of an activation function in neural networks? and, while writing the answer, it struck me, how exactly can ReLUs approximate a non-linear function?
By pure mathematical definition, sure, it's a non-linear function due to the sharp bend, but, if we confine ourselves to the positive or the negative portion of the $x$-axis only, then it's linear in those regions. Let's say we take the whole $x$-axis also, then also it's kinda linear (not in a strict mathematical sense), in the sense that it cannot satisfactorily approximate curved functions, like a sine wave ($0 \rightarrow 90$) with a single node hidden layer as is possible by a sigmoid activation function.
So, what is the intuition behind the fact that ReLUs are used in NNs, giving satisfactory performance? Are non-linear functions, like the sigmoid and the tanh, thrown in the middle of the NN sometimes?
I am not asking for the purpose of ReLUs, even though they are kind of linear.

As per @Eka's comment, the ReLu derives its capability from discontinuity acting in the deep layers of the NN. Does this mean that ReLUs are good, as long as we use it in deep NNs and not a shallow NN?

The outputs of a ReLU network are always "linear" and discontinuous. They can approximate curves, but it could take a lot of ReLU units. However, at the same time, their outputs will often be interpreted as a continuous, curved output.
Imagine you trained a neural network that takes $x^3$ and outputs $|x^3|$ (which is similar to a parabola). This is easy for the ReLU function to do perfectly. In this case, the output is curved.
But it is not actually curved. The inputs here are 'linearly' related to the outputs. All the neural network does is it takes the input and returns the absolute value of the input. It performs a 'linear', non-curved function. You can only see that the output is non-linear when you graph it against the original $x$-values (the $x$ in $x^3$).
So, when we plot the output on a graph and it looks curved, it's usually because we associated different x-values with the input, and then plotted the output as the $y$-coordinate in relation to those $x$-values.
Okay, so you want to know how you would smoothly model $\sin(x)$ using ReLU. The trick is that you don't want to put $x$ as the input. Instead put something curved in relation to $x$ as the input, like $x^3$. So, the input is $x^3$ and the output is $\sin(x)$. The reason why this would work is that it isn't computing the sine of the input - it's computing sine of the cube root of the input. It could never smoothly compute the sine of the input itself. To graph the output $\sin(x)$, put the original $x$ as the $x$ coordinate (don't put the input) and put the output as the $y$ coordinate.


How do autonomous robotic vacuum cleaners perceive the environment for navigation? How does a domestic autonomous robotic vacuum cleaner -  such as a Roomba - know when it's working cleaned area (aka virtual map), and how does it plan to travel to the areas which hasn't been explored yet?

Does it use some kind of A* algorithm?

An agent perceives the environment through sensors and act according to the incoming percepts (agent's perceptual input at any instant). An autonomous vacuum cleaner can be as simple as

  (blocki, clean) --> Move to blocki+1
  (blocki, dirty) --> Clean

This is just a general description, actual one is more complicated. Or the bot can have a memory where it stores all its previous decision and incorporate those while taking new ones.
This can be helpful if the bot wants to remember where an obstacle (like wall, in this case bot don't want to go and check the presence of wall each and every single time it is turned on) is, or where it is more probable to find dirt. If the bot is not remembering its history then it will be scanning the whole house over and over again, sensing the same obstacle every time and going across them.
Bot which keeps no log of its history will take the same procedure again and again, making the same mistakes again and again. This is not an efficient way and a waste of its energy (or battery).
Normally today bots have ordinary sensors which can only sense the dirt and obstacle. This limits the number of tasks a bot can perform. If a bot has decent camera as a sensor, and some algorithms of Image Processing are dumped into it, then it increases the tasks it can perform. Like detecting the stairs and cleaning different floors. Normally stairs will be considered obstacle and bot will just go around them. In case, when camera sensor is provided, stairs are potentially a path to be taken.
A* algorithm is not necessarily used in case when the bot is not remembering the map of the house (or room). A normal robot which just scans the room and cleans it, will not be needing, as it don't know it's destination. Its only goal is to clean if it finds something dirty. But a bot which knows the map of the room and where there is a high probability of finding dirt, the A* algorithm can be used.


How should I implement the backward pass through a flatten layer of a CNN? I am making a NN library without any other external NN library, so I am implementing all layers, including the flatten layer, and algorithms (forward and backward pass) from scratch. I know the forward implementation of the flatten layer, but is the backward just reshaping it or not? If yes, can I just call a simple NumPy's reshape function to reshape it?

Yes, a simple reshape would do the trick. A flattening layer is just a tool for reshaping data/activations to make them compatible with other layers/functions. The flattening layer doesn't change the activations themselves, so there is no special backpropagation handling needed other than changing back the shape.


What are the minimum computing resources needed to train a machine learning algorithm? For a school project, I would like to investigate a paper on either reinforcement learning or computer vision. I am particularly interested in DQN, RNNs, CNNs or LSTMs. I would eventually like to implement any of these. However, I also need to take into account the computing resources required to train and analyse any of these algorithms. I understand that, in computer vision, the data sets can be quite large, but I am not so sure regarding the resources needed to implement and train a typical state-of-the-art RL algorithm (like DQN).

Would a "standard PC" be able to run any of these algorithms decently to achieve some sort of analysis/results?

It might be hard to implement deep reinforcement learning algorithms, especially considering your previous experience and the computing resources you have. They require almost the same (even more) GPU power. Deep reinforcement learning algorithms use deep neural networks for learning the optimal policy. Even if you are given appropriate resources, it would be tough to replicate the results of the paper, if you are a novice.


How do 3 channels affect a network when detecting human skin (CNN)? Yeah I know, best title ever. Anyway,

I want to make a neural network which is fed with frames coming from an usb camera.
Don't wanna be so specific, so I'm just gonna say that the network's goal is to classify human hand gestures, therefore I need to make sure it can effectively learn how the hand moves around.

My problem is that I've no idea about what happens when having 3 channels instead of 1, I only know that (for 3 channels) it does 3 separate convolution operations with the same kernel, resulting actually in 3 separate layers.
How do this 3 channels affect the network? Does it learn from the movement 3 parallel times, then it mixes toghether this 3 "separate movements"? Do I need to make it single channel to help him detect the hand?

PS: the text is problably confusing, but that's because I'm confused to, that's why I'm asking.

3 channels will give you more information about the color of the object and surroundings which might be important in certain cases. For example if you want to classify blue cars and red cars then color of the object is very important. Since your problem is to classify hand gestures then color might not be that relevant. You're not very interested in the color of the hand you only care about it's position so there is a good chance that grayscale images might be enough. You should try with grayscale first and if that works good, if not, try with 3 channels.


Neural network returns similar output I was following Daniel Shiffman's tutorials on how to write your own neural network from scratch. I specifically looked into his videos and the code he provided in here. I rewrote his code in Python, however, 3 out of 4 of my outputs are the same. The neural network has two input nodes, one hidden layer with two nodes and one output node. Can anyone help me to find my mistake? Here is my full code.

import random

nn = NeuralNetwork(2,2,1)
inputs  = np.array([[0, 0], [1, 0], [0, 1], [1, 1]])
targets = np.array([[0], [1], [1], [0]])
zipped = zip(inputs, targets)
list_zipped = list(zipped)

for _ in range(9000):
    x, y = random.choice(list_zipped)
    nn.train(x, y)

output = [nn.feedforward(i) for i in inputs]

for i in output:
   print("Output ", i)

#Output  [ 0.1229546]  when it should be around 0
#Output  [ 0.6519492]  ~1
#Output  [ 0.65180228] ~1
#Output  [ 0.66269853] ~0

EDIT_1: I tried debugging my code by choosing all weights and bias' values to 0.5. I did this in both my code and Daniel's. This obviously ended up showing me all outputs with the same value.

After that I increased my weights and bias' values variety from [0 , 1) to [-1, 1). By running this a few times, I would sometimes get the correct output:

[ 0.93749991] # should be ~1
[ 0.93314793] # ~1
[ 0.07001175] # ~0
[ 0.06576194] # ~0

If I ran nn.train() 100 000 times, I get the correct output 2/3 times.
Is this the issue of gradient descent, where it converges to the local minima?

Local minima.
You have the exact same issue of this question. If you randomize your initial weights, you'll see sometimes you get the correct results, and others you won't. It's because when the weights are initialized with a certain range of values, they will converge to a local minima which you cannot escape with a low learning rate.
A simple solution is to increase the size of your hidden layer, which will make the network more robust to such issues.

When you have only 2 dimensions, a local minima exists. When you have more dimensions, this minima gets harder and harder to reach, as its likelihood decreases. Intuitively, you have a lot more dimensions through which you can improve than if you only had 2 dimensions.
The problem still exists, even with 1000 neurons you could find a specific set of weights which was a local minimum. However, it just becomes so much less likely.


How do I derive the gradient with respect to the parameters of the softmax policy? The gradient of the softmax eligibility trace is given by the following:

\begin{align}
\nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\
&= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a')
\end{align}

How is this equation derived?

The following relation is true:

\begin{align}
\nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \frac{\nabla_{\theta} \pi_{\theta}(a|s)}{\pi_{\theta}(a|s)} \tag{1}\label{1}
\end{align}

Thus, the following relation must also be true:
\begin{align}
\frac{\nabla_{\theta} \pi_{\theta}(a|s)}{\pi_{\theta}(a|s)}  &=\phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a')
\end{align}

Mathematically, why would this be the case? Probably, you just need to answer my question above because \ref{1} is true and it's just the rule to differentiate a logarithm.

Softmax policy $\pi_\theta(s,a)$ is defined as $\frac{\exp{(\phi(s,a)^T \theta})}{\Sigma \exp{(\phi(s,a) ^T \theta) }}$, where the summation is over the action space.
Taking log, this becomes
$$
\log \pi_\theta(s,a) = log(e^{\phi(s,a) ^T \theta}) - log({\Sigma e^{\phi(s,a) ^T \theta }}) \\
= \phi(s,a) ^T \theta - log({\Sigma e^{\phi(s,a)^T \theta }})
$$

Taking derivative wrt $\theta$, this becomes
$$
\nabla_\theta \log \pi_\theta(s,a) = \phi(s,a) - \nabla_\theta log({\Sigma e^{\phi(s,a) ^T \theta }})
$$

We can rewrite $\nabla_\theta log({\Sigma e^{\phi(s,a)^T \theta }})$ as follows.
$$
\nabla_\theta log({\Sigma e^{\phi(s,a)^T \theta }})
= \frac{\nabla_\theta \Sigma e^{\phi(s,a)^T \theta}}{\Sigma e^{\phi(s,a) ^T \theta}}
= \frac{\Sigma \phi(s,a) e^{\phi(s,a) ^T \theta}}{\Sigma e^{\phi(s,a) ^T \theta}}
= \Sigma \phi(s,a) \pi_\theta(s,a)
$$

The final equation then becomes
$$
\nabla_\theta \log \pi_\theta(s,a) = \phi(s,a) - \Sigma \phi(s,a) \pi_\theta(s,a)
$$


Where would consciousness reside in an artificial intelligence? I think that my consciousness and intelligence remains located inside me, in my body or brain.
I know that I am a stalwart of a dualistic nature: mind and material.
But let's suppose that an artificial intelligence can be created purely from the material (e.g. by programming a neural network).
Where will the consciousness (and intelligence) of this AI be within it: its RAM, its processor, or will the programming itself create a bi-location with no precise boundary or absolute position within the machine?

There are still many different interpretations of the terms intelligence and consciousness. For the sake of this discussion I would define intelligence as the problem solving capabilities and consciousness as the state of being aware of and responsive to one's surroundings, including the experience of qualia.

If I understand your question correctly, you are focused on consciousness and not intelligence (based on the definitions above).

In the case of humans - the only species where we have first hand knowledge about what consciousness feels like - it feels like the consciousness resides inside the respective body. Most people experience it like "being" inside their head. Although debated, consciousness is most likely a simulation created by the human brain. Qualia is heavily - if not exclusively - influenced by the sensory input from the human senses. Therefore it only makes sense that it feels like the consciousness resides in the center of the sensors (eyes, ears, etc.).

Science hasn't figured out the nature of consciousness yet, so there is a lot of speculation involved. Applying above this logic to artificial consciousness and qualia, it would make sense that the machine will feel like being at the location if its sensors as well. If there is no dominant area of sensors, maybe a distributed system, it might be a split consciousness spread across several locations.

If the creation of consciousness happens accidentally - like it did during evolution - emergence at the sensors feels most likely. Should the creator of artificial consciousness do it on purpose, he should be able to shape the simulation in other ways as he sees fit.


What is the significance of this Stanford University "Financial Market Time Series Prediction with RNN's" paper? Researchers at Stanford University released, in 2012, the paper Financial Market Time Series Prediction with Recurrent Neural Networks.

It goes on to discuss how they used echo state networks to predict things such as Google's stock prices. However, to do this, once trained, the network's inputs are a day's stock price, and the output is the day's predicted stock price. The way the paper is worded is like this could be used to predict future stock prices, for example. However, to predict tomorrow's stock price, you need to give the neural network tomorrow's stock price.

All this paper seems to show is that the neural network is converging on a solution where it simply modifies its inputs a minimal amount, hence the output of the ESN is just a small alteration of its input.

Here are some Python implementations of the work shown in this paper:

https://github.com/kimanalytics/Recurrent-Neural-Network-to-Predict-Stock-Prices
https://github.com/europa502/RNN-based-Bitcoin-Value-Predictor

In particular, I was playing with the latter which produces the following graph:

If I take the same trained network, and alter the 7th's day's "real" stock price to say something extreme like $0, this is what comes out:

As you can see, it basically regurgitates its inputs.

So, what is the significance of this paper?

It has no use in any financial predictions, like the network shown in the paper Classification-based Financial Markets Prediction using Deep Neural Networks.

After a very brief look at the paper, I think that they are predicting the stock price for the next day, and not for the current day, which is quite common and reasonable: see equation (1), where they predict $x(t+1)$.
So I don't see any issue with this paper.

But I've only quickly looked at it, so I may have missed something, of course.


How can I read the .weights file that stores the weights of the pre-trained YOLO in Keras? I would like to use the pre-trained weights of YOLO (tiny version, v1) in Keras, which are given in a file with extension .weights (here is an example). How can I read such a file in Keras?
For the past two days, I've been relentlessly digging through Github, and the likes, in order to help me in this task, with more or less success.  I found some scripts on the internet in order to convert .txt or .h5 files (.h5 only for YOLO v2, and not v1), but none of these seemed to be working properly.
So, I guess my question is: how to read a .weights file? How are the data stored and structured in these files?

The .weights seems to be the extension for a framework called "darknet". You can read .h5 files with Keras.
However, if you really want to build an object detection framework, there is no necessity to stick to the darknet's .weights files. There are many pretrained models on the web. Or else you could fine-tune a pre-trained ImageNet model in Keras, which I think is the best option, although there is not much you can do, if you got the exact weights of the YOLO model. Instead, it is better to train one from scratch or at least by fine-tuning an ImageNet model. By doing so, you will learn a lot.


Why won't my model train with CTC loss? I am trying to train an LSTM using CTC loss, but the loss does not decrease when I train it. I have created a minimal example of my issue by creating training data where the network simply has to copy the current input element at each time step. Moreover, I have made the length of the label the same as the length of the input sequence and no adjacent elements in the label sequence the same so that both CTC loss and categorical cross-entropy loss can be used. I found that when using categorical cross-entropy loss the model very quickly converges, whereas when using CTC loss it gets nowhere.
I have uploaded by minimal example to colab. Does anyone know why CTC loss is not working in this case?

The problem was the dimensions of the logit_length argument to tf.nn.ctc_loss was incorrect.
It was this:
tf.repeat(tf.shape(y_pred)[-1], tf.shape(y_pred)[0])

But it should have been
tf.repeat(tf.shape(y_pred)[-2], tf.shape(y_pred)[0])


Can we code rules for an agent in python language other than predicate calculus? I have a medical dataset with 14000 rows dataset with 900 attributes. I have to predict disease severity using that. I would like to know whether we can write rules in python language for training an agent for medical diagnostic using machine learning.

Can an agent make the decisions by the rules coded in python and that agent get trained with some machine learning algorithms? If so is there any agent architecture and model for the agent which is good in this context?

Edit: By the rule, I meant something like this.."if x>y output z as action". By the word "Training" I meant "how to tell this agent to do this action"?

You could formulate the problem of predicting disease severity as a classification one, you give the algorithm those 900 attributes and their corresponding labels (severe/not severe) after training, you give it a new data point with just the 900 attributes, it returns the correct label severe or not.

There is an enormous number of algorithms in the ML literature for classification, some of them formulate the problem explicitly as learnable rules, i.e. let the machine figure out the rules given the attributes.

Classification and Regression Trees by Breiman et al (1984)
Random forest by Ho in 1995
XGBoost


How does the Ornstein-Uhlenbeck process work, and how it is used in DDPG? In section 3 of the paper Continuous control with deep reinforcement learning, the authors write

As detailed in the supplementary materials we used an Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein, 1930) to generate temporally correlated exploration for exploration efficiency in physical control problems with inertia (similar use of autocorrelated noise was introduced in (Wawrzynski, 2015)).

In section 7, they write

For the exploration noise process we used temporally correlated noise in order to explore well in physical environments that have momentum. We used an Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein, 1930) with θ = 0.15 and σ = 0.2. The Ornstein-Uhlenbeck process models the velocity of a Brownian particle with friction, which results in temporally correlated values centered around 0.

In a few words, what is the Ornstein-Uhlenbeck process? How does it work? How exactly is it used in DDPG?
I want to implement the Deep Deterministic Policy Gradient algorithm, and, in the initial actions, noise has to be added. However, I cannot understand how this Ornstein-Uhlenbeck process works. I have searched the internet, but I have not understood the information that I found.

The Ornstein Ulhenebck Process is defined as (in the continuous setting) :
$$dX_t = -\beta(X_t - \alpha)dt + \sigma dW_t$$
The analogue for this process in the discrete time case which I assume will be applicable in the RL case will be:
$$X_{t+1} = X_t -\beta(X_t - \alpha) + \sigma \{W_{t+1}-W_t\}=$$
$$X_{t+1} = (1 -\beta)X_t - \alpha + \sigma \{W_{t+1}-W_t\}$$
In the RL seting the terms in the equation probably means:

$X_t$ will stand for a state in RL i.e. the state is the number $\in \mathbb R$ where the particle moves to at time $t$.
$\beta$ and $\alpha$ are just constants which decide certain movement characteristics of the particle. Check here for graphs plotted for various $\beta$.
$W_t$ is a Weiner process which starts at $W_0 = 0$ and then adds independent increments of $\mathcal N(\mu,\sigma)$ as $W_{t+1} = W_t+\mathcal N(\mu,\sigma)$ which is basically a radom walk. More generally we use $\mathcal N(0,1)$. This is formulated as $W_t-W_s = \sqrt{t-s} \mathcal N(0,1)$. This is because of the fact, $W_t$ can be written recursively as $W_t = \mathcal N(0,1)+W_{t-1} = \mathcal N(0,1) + \mathcal N(0,1) + ...W_s$ and since the samplings are independent at each step the mean get added as $\mu_t+\mu_{t-1}...$ and the variances as $\sigma_t^2 + \sigma_{t-1}^2...$. SInce her the means and variances are $0$ and $1$ respectively, the final mean $\mu = 0$ and variance $\sigma^2 = (t-s)$. And hence, byproperties of Gaussian random variables you can write (it is easy to show this via variable tranformation) $W_t-W_s = \sqrt{t-s} \mathcal N(0,1)$.  Here, is the formulation of standard Weiner process.
$\sigma$ will be the weighting factor of the Weiner process, which just  means the amount of noise being added to the process.

Another useful resource on discrete Ornstein Ulhenbeck process, much less generalized.
I think now you can extend this to whatever scenario you are intereted in RL setting.


Is there any research on the development of attacks against artificial intelligence systems? Is there any research on the development of attacks against artificial intelligence systems?

For example, is there a way to generate a letter "A", which every human being in this world can recognize but, if it is shown to the state-of-the-art character recognition system, this system will fail to recognize it? Or spoken audio which can be easily recognized by everyone but will fail on the state-of-the-art speech recognition system.

If there exists such a thing, is this technology a theory-based science (mathematics proved) or an experimental science (randomly add different types of noise and feed into the AI system and see how it works)? Where can I find such material?

Yes, there is some research on this topic, which can be called adversarial machine learning, which is more an experimental field.

An adversarial example is an input similar to the ones used to train the model, but that leads the model to produce an unexpected outcome. For example, consider an artificial neural network (ANN) trained to distinguish between oranges and apples. You are then given an image of an apple similar to another image used to train the ANN, but that is slightly blurred. Then you pass it to the ANN, which unexpectedly predicts the object to be an orange.

Several machine learning and optimization methods have been used to detect the boundary behaviour of machine learning models, that is, the unexpected behaviour of the model that produces different outcomes given two slightly different inputs (but that correspond to the same object). For example, evolutionary algorithms have been used to develop tests for self-driving cars. See, for example, Automatically testing self-driving cars with search-based procedural content generation (2019) by Alessio Gambi et al.


Adding input features - is complete re-training required? I've never worked with very large models that require weeks or months of training, but in such a situation, what happens if you want to add extra features inputs, do you need to re-train the entire model again from scratch, or how is it handled?  E.g. if Tesla added an extra sensor to its cars, would it need to re-train its network again from scratch to include this extra sensor input?

I'll try to explain how I would do it and the intuition behind it, feel free to correct me if something doesn't make sense.
Lets consider you have an input of shape F where F is the number of features. If you were to construct a simple feed forward neural network you'll need to multiply the input with a weight matrix of shape (F, hidden_dim). Now if we want to add one more feature, the input will be of shape F+1 and the multiplication with the first layer will not work. What we could do to overcome this problem is to pad the weight matrix with an extra 0. In theory the new network should be able to reproduce the results of the first because it ignores the new feature. Now if we want to learn the "importance" of the new feature we could train the model to do so. I assume that the training time should be significantly lower since some of the weights might not need to be updated.


How are the inputs passed to the neural network during training for the XOR classification task? Let's suppose we have to train a neural network for the XOR classification task.

Are the inputs $(00, 01, 10, 11)$ inserted in a sequential way? For example, we first insert the 00 and change the weights, then the 01 and again slightly change them, etc. Or is there another way it can be implemented?

Hello and welcome to the community. There are multiple ways you can train a neural network: stochastic, mini-batch and batch. What you explained is the stochastic mode, where you input one training example 01 for example, calculate the gradients and update the networks weights before the next training example is fed. You could also select multiple such examples (a mini-batch) and update the weights only after you computed all the outputs (for this particular mini-batch). Finally you can use a batch size which is equal with the total number of examples in your dataset so you will update the weights only after you have all the outputs for all samples. Each of those methods have their own strengths and weaknesses, depending on your dataset you might prefer one over the others.


How do I create chatbots without machine learning? I am looking for an non-ML method for two chat bots to communicate to each other about a specific topic. I am looking for an "explainable AI" method, as opposed to a "black-box" one (like a neural network).

The easiest non-ML way would be to use a finite state machine. You could model various states of your conversation topics, and certain utterances of your bots could advance the bot's internal model along different paths. The complexity depends on the complexity of the topic.

You can then enhance the transitions with probabilities, and later move on towards ML by transforming it into an HMM.

However, even simple topics will probably lead to fairly complex state machines. But you should be able to keep track of what is going on in your conversation nevertheless.

Update: just to make it a bit clearer, I was thinking along the lines of having states for particular stages in the conversation. You could either have one model for the whole conversation, or one per participant.

Initially, there would be a state 'greeting'. Possible transitions would be to a further state 'greeting' (the response of the person who has been greeted), or that could be skipped to states such as 'statement', 'question', etc. 'Question' would have transitions to 'answer', 'ignore question', 'counter/clarification question' etc. The level of detail depends on your application.


What is the derivative of a specific output with respect to a specific weight? If I have a neural network, and say the 6th output node of the neural network is:
$$x_6 = w_{16}y_1 + w_{26}y_2 + w_{36}y_3$$
What does that make the derivative of:
$$\frac{\partial x_6}{\partial w_{26}}$$
I guess that it's how is $x_6$ changing with respect to $w_{26}$, so, therefore, is it equal to $y_2$ (since the output, $y_2$, will change depending on the weight added to the input)?

Formally speaking $x_6$ is a function of $w_{16},\ w_{26}$ and $w_{36}$, that is $$x_6 =f(w_{16}, w_{26}, w_{36})=w_{16}y_1 + w_{26}y_2 + w_{36}y_3.$$
The derivative w.r.t. $w_{26}$ is
$$\frac{\partial x_6}{\partial w_{26}}= \frac{\partial w_{16}y_1}{\partial w_{26}} +\frac{\partial w_{26}y_2}{\partial w_{26}}  +\frac{\partial w_{36}y_3}{\partial w_{26}} = 0 +y_2 \frac{\partial w_{26}}{\partial w_{26}} + 0= y_2.$$
The first equality is obtained using the fact that the partial derivative is linear (so the derivative of the sum is the sum of the derivatives); the second equality comes from again from the linearity and from the fact that $w_{16}y_1$ and $w_{36}y_3$ are constants with respect to $w_{26}$, so their partial derivative w.r.t. this variable is $0$.
Bonus
Not really asked in the original question, but since I'm here let me have a bit of fun ;).
Let's say $x_6$ is the output of the sixth node after you apply an activation function, that is $$x_6 =\sigma(f(w_{16}, w_{26}, w_{36}))=\sigma(w_{16}y_1 + w_{26}y_2 + w_{36}y_3).$$
You can compute the partial derivative applying the properties illustrated above, with the additional help of the chain rule
$$\frac{\partial x_6}{\partial w_{26}}=\frac{\partial \sigma(w_{16}y_1 + w_{26}y_2 + w_{36}y_3)}{\partial w_{26}}=\sigma'\frac{\partial w_{16}y_1}{\partial w_{26}} +\sigma'\frac{\partial w_{26}y_2}{\partial w_{26}}  +\sigma'\frac{\partial w_{36}y_3}{\partial w_{26}}=y_2\sigma'$$
$\sigma'$ denotes the derivative of sigma with respect to its argument.


What is teacher forcing? In the paper Neural Programmer-Interpreters, the authors use the teacher forcing technique, but what exactly is it?

Consider the task of sequence prediction, so you want to predict the next element of a sequence $e_t$ given the previous elements of this sequence $e_{t-1}, e_{t-2}, \dots, e_{1} = e_{t-1:1}$. Teacher forcing is about forcing the predictions to be based on correct histories (i.e. the correct sequence of past elements) rather than predicted history (which may not be correct). To be more concrete, let $\hat{e}_{i}$ denote the $i$th predicted element of the sequence and let $e_{i}$ be the corresponding ground-truth. Then, if you use teacher forcing, to predict $e_{t}$, rather than using $\hat{e}_{t-1:1}$, you would use $e_{t-1:1}$.

Recall that supervised learning can also be thought of as learning with a teacher. Hence the expression "teacher forcing", i.e. you force the predictions to be based on correct histories (the teacher's labels).

Of course, intuitively, teacher forcing should help to stabilize training, given that the predictions are not based on noisy or wrong histories.

See also the blog post What is Teacher Forcing for Recurrent Neural Networks? by Jason Brownlee.


Has Google's view of AI risen to the status of universal truth? In the early 20th century, anything Princeton staff represented related to computing and the simulation of human thought was assumed to be flawless.  In the late 20th century, it was MIT's AI lab that was trusted to define what was feasible and what were the next steps.  Is Google the new source of legitimacy in AI pronouncements?  Because of the connection between that corporation and the daily life of people on earth, has that publication honor that was given to the primary AI centers of research in the 20th century reached a level in the early 21st century that rivals deity?

It sometimes seems that, when people don't understand something in a paper with the word Google anywhere in the authorship block, it is automatically assumed that the reader is wrong because the paper is of universal authority.  They seem to jump to the assumption that their mind must adjust to whatever the paper states.  Is this equivalent to religious scripture?

Carl Jung wrote,

  Contemporary man is blind to the fact that, with all his rationality and efficiency, he is possessed by "powers" that are beyond his control. His gods and demons have not disappeared at all; they have merely got new names.

This is an AI StackExchange, so the proposal in this question is that perhaps the same brain machinery has been used for the following things

An ancient Egyptian gathering with others to worship the sun god Re
Casiodoris using the resources of the Roman Empire to send holy scripture out on horseback to the far reaches of the empire to save it from the invading barbarian hordes
AI researchers finding some unassailable source of truth in a university or a multinational corporation

Is Carl Jung correct, and, if so, is this human need to find legitimacy in an institution now manifesting in the AI community's absolute trust in a corporation that automated the 18th century library card catalog and built that automaton into a large multinational corporation?

Lastly, if both those questions can be answered in the affirmative, what risks can exist when the human mind treats a university or a corporation as a god from an economic or social risk management perspective?

No, "Google's view of AI" (which in and of itself already is kind of vague, there are plenty of people within Google who have very different ideas) has not risen to the status of universal truth.

  It sometimes seems that, when people don't understand something in a paper with the word Google anywhere in the authorship block, it is automatically assumed that the reader is wrong because the paper is of universal authority. They seem to jump to the assumption that their mind must adjust to whatever the paper states.

It does not seem like that to me at all.

Yes, there has been research (from Google actually) indicating that there is some bias in paper review for example. When reviewers can see the author names and their affiliations, there is some measurable bias towards accepting them more often when they are from famous / well-known than when they are not famous. That includes Google, but also other companies (like Facebook, OpenAI) and universities and simply individuals who happen to be well-known even if their affiliations aren't. The same thing probably also happens occasionally even when double-blind reviewing is used, because... well, it's not difficult to guess where the anonymous authors may be coming from when the experiments described in their paper use a huge collection of TPUs.

It's nothing more than that though, just a bias. They get their papers accepted (and probably also cited) a bit more often. They definitely do not always get their papers accepted. Plenty of papers from Google have been rejected too (which means the reviewers did not simply accept whatever was written without critical thinking), and that will continue to happen too. It also happens more often than not that I see/hear others being critical or confused by things they write. There is still plenty of critical thinking going around.

Of course, the fact of the matter still is that they have managed to attract many world-class researchers (who were also already known as such before joining Google). It is no surprise that such researchers do have many good ideas, do write solid papers, and with Google they also do have the resources (in terms of computation, finances, development support from highly skilled software engineers, etc.) to execute more ideas, execute them more quickly, and more thoroughly. This naturally gives them an important voice, but they are not immune to criticism.


What is the difference between search and planning? I'm studying Artificial Intelligence. A Modern Approach, Stuart Russell, Peter Norvig, specifically about search and planning arguments.

I don't understand the difference between the two terms. I was more confused when I saw that some search problems can be determined in planning way. My professor explained to me in a confusing way that the real difference on the search is that it uses an heuristic function, but my book says that planning use a heuristic too, for relaxing problem (in cap. 10.2.3).

I read this page that says in a certain way what I'm saying.

Is planning and search the problem? If not, what are the differences and how are these problems related?

What is planning?
Planning (aka automated planning or AI planning) is the process of searching for a plan, which is a sequence of actions that bring the agent/world from an initial state to one or more goal states, or a policy, which a function from states to actions (or probability distributions over actions). Planning is not just the use of a search algorithm (e.g. a state-space search algorithm, such as A*) to find the plan, but planning also involves the modelling of a problem by describing it in some way, e.g. with an action language, such as PDDL or ADL, or even with propositional logic.
There are different planning problems, such as

classical planning
planning in Markov Decision Processes (which are mathematical models that describe stochastic environments/problems, i.e. environments where an action $a$ may not always lead to the same outcome)
temporal planning (i.e. when multiple actions can be taken at the same time and/or their duration may be different/variable)
hierarchical vs non-hierarchical planning

There are different planning approaches, such as

planning by state-space search (with or without heuristics), i.e. formulate the planning problem so that you can apply a state-space search algorithm  (examples of state-space search algorithms are A*, IDA*, WA*, DFS or BFS); this is probably the approach to planning that makes people confuse planning with search (which typically implicitly refers to state-space search)

planning by SAT, i.e. formulate the planning problem so that you can you a SAT solver to solve it (e.g. SATPLAN)

planning by solving constraint satisfaction problems (similar to SAT)

planning by "symbolic" search methods (such as Binary Decision Diagrams)

There are many different planning algorithms or planners (of course, each of these corresponds to some approach or is used to solve a particular planning problem), such as

policy iteration in the context of Markov decision processes
GraphPlan
STRIPS (which stands for Stanford Research Institute Problem Solver, but note that STRIPS also and maybe more commonly refers to the corresponding description language used to describe the planning problem)

What is search?
In the context of AI, when people use the word search, they can either implicitly refer to state-space search (with algorithms like A*) or, more vaguely/generally, to any type of search, e.g. gradient descent can also be viewed as a search algorithm, typically, in a continuous state space (rather than discrete state space), e.g. the space of real-valued vectors of parameters for a certain model (e.g. a neural network).
What is the difference between planning and search?
The difference should more or less be clear from my first paragraph above that describes planning: search is used for planning, but the planning process/study also includes other things, such as describing the problem with an action language, such as PDDL. You can also view planning as an application of search algorithms of different types. In any case, planning is about searching for plans or policies. You should probably check this course for more info. This definition of (automated) planning is consistent with our usual notion of planning.


Why would the lookup table (of a table-driven artificial agent) need to store data at pixel precision? While reading the book AI A modern approach, 4th ed, I came across the section of "Agent program" with following text:

It is instructive to consider why the table-driven approach to agent
construction is doomed to failure. Let $P$ be the set of possible
percepts and let $T$ be the lifetime of the agent (the total number of
percepts it will receive).
The lookup table will contain $\sum_{i=1}^T |P|^T$ entries.
Consider the automated taxi: the visual input from a single camera
(eight cameras is typical) comes in at the rate of roughly 70 mb per
sec. (30 frames per sec, 1080 X 720 pixels, with 24 bits of color
information).
This gives a lookup table with over $10^{600,000,000,000}$ for an
hour's driving.

Could someone please explain how the lookup table number is derived? (or what the author's point is which I am missing). If I were to multiply all of the numbers $30 × 1080 × 720 × 24 × 8 × 3600$, then I get $1.6124314e+13$ which comes very close I think, but can't get what would be the reason to build a table (even though a theoretic one) in such a way - something which is obviously intractable
edit:
My core question is this:
Assuming $10^{600,000,000,000}$ is derived from $30 × 1080 × 720 × 24 × 8 × 3600$, what is the purpose of storing data in the look up table at pixel precision? Wouldn't storing higher level of details be enough to solve these kind of problems (ie, autonomous driving)? Coming more from standard software database systems, I am missing that point. Thanks

A tabular system for agent decisions is a direct and simple map of percept to control choice. For each percept received, the agent looks up the percept and cross-references it to the action it should take. In order to construct this, you need to list all percepts in full detail, with the associated control choice.
Clearly that is not going to be feasible for the automated taxi example. No-one would think to build such a table to handle natural image inputs. That is the author's point.
However, a tabular structure is a reasonable theoretical construct for mapping an arbitrary discrete function, and also is practical for simple environments.
To answer your extended question:

Assuming $10^{600,000,000,000}$ is derived from $30 × 1080 × 720 × 24 × 8 × 3600$, what is the purpose of storing data in the look up table at pixel precision?

It is the only way to get a map from percept to control using a tabular system.
By proposing any kind of summarisation or approximation of the input-to-output function to solve this, you have gone beyond the capability of a tabular system. That again is the author's point.

Wouldn't storing higher level of details be enough to solve these kind of problems (ie, autonomous driving)?

If it is really obvious to you that this is the solution, then that's a good thing as you are thinking ahead. However, you should also consider what that means in terms of what you might be giving up, from a theoretical perspective. For instance, a tabular system can make radically different decisions based on very minor differences between percepts, whilst any form of processing of the inputs to make them easier to handle is necessarily going to remove information that might be important.


Is there anything theoretically revolutionary about Deep Neural Networks? In recent years, we have seen quite a lot of impressive display of Deep Neural Network (DNN), as demonstrated most famously by AlphaGo and its cousin programs.
But if I understand correctly, deep neural network is just a normal neural network with a lot of layers. We know about the principles of the neural network since the 1970s (?), and a deep neural network is just the generalization of a one-layer neural network to many.
From here, it doesn't seem like the recent explosion of DNN has anything to do with a theoretical breakthrough, such as some new revolutionary learning algorithms or particular topologies that have been theoretically proven effective. It seems like DNN successes can be entirely (or mostly) attributed to better hardware and more data, and not to any new theoretical insights or better algorithms.
I would go even as far as saying that there are no new theoretical insights/algorithms that contribute significantly to the DNN's recent successes; that the most important (if not all) theoretical underpinnings of DNNs were done in the 1970s or prior.
Am I right on this? How much weight (if any) do theoretical advancements have in contributing to the recent successes of DNNs?

The first neural network machine was the stochastic neural analog reinforcement calculator (SNARC), built in the 1950s. As you can see, it's pretty old. After that, there were several advances regarding backpropagation and the vanishing gradient problem. However, the ideas itself are not novel. Simply put, we have the data and processing power today that we did not have back then.

You could look at the Wikipedia timeline.


Was Pierce way off the mark? “Funding artificial intelligence is real stupidity.”
  -- John R. Pierce

Was this computer pioneer way off the mark? – or was there important sub-text there?

Pierce was an expert for machine translation in the 1960s. He coauthored the following paper Pierce, John R., and John B. Carroll. "Language and machines: Computers in translation and linguistics." (1966). That means, he worked on the domain of AI but explained his subject as useless. Perhaps Marvin Minsky, Rodney Brooks and Sebastian Thrun would agree to him?

John R. Pierce led the Bell Labs research team that created the first transistor and gave it its name.  He was later the Chief Engineer at the Jet Propulsion Laboratory at CalTech.

His relationship to artificial intelligence research was mainly in regard to language translation.  He wrote the following.1

  The computer has opened up to linguists a host of challenges, partial insights, and potentialities. ... Certainly, language is second to no phenomenon in importance. ... The new linguistics presents an attractive as well as an extremely important challenge. ... The most reasonable government source of support for research in computational linguistics is the National Science Foundation. ... We estimate that work on a reasonably large scale ... would be justified at ... an annual expenditure of \$2.5 to \$3 million.

No official record of Pierce stating unequivocally that all artificial intelligence funding is stupid can be found.  Nils J. Nilsson's The Quest for Artificial Intelligence quotes Donald Knuth stating this.

  John R. Pierce, whom I have already mentioned in connection with the ALPAC report on machine translation (in section 7.2) and his negative comments about speech understanding (p 222) wrote me a very short letter
  in which he stated,

    Concerning artificial intelligence, I believe I invented the slogan, "Artificial intelligence is real stupidity."

    ...

    I resent artificial intelligence because I feel that it is unfair to computers. But then, artificial intelligence people did devise LISP, which is pretty good.

  The letter did not elaborate on the slogan or why AI is "unfair to computers."

The paragraphs that may have drawn the most negative attention (and possibly fabricated quips) are these.2

  It would be too simple to say that work in speech recognition is carried out simply because one can get money for it.  That is a necessary but not a sufficient condition.  We are safe in asserting that speech recognition is attractive to money.  The attraction is perhaps similar to the attraction of schemes for turning water into gasoline, extracting gold from the sea, curing cancer, or going to the moon.  One doesn't attract thoughtlessly given dollars by means of schemes for cutting the cost of soap by 10%.  To sell suckers, one uses deceit and offers glamor.

  It is clear that glamor and any deceit in the field of speech recognition blind the takers of funds as much as they blind the givers of funds.  Thus, we may pity workers whom we cannot respect.  People who work in the field are full of innocent (in their own view) enthusiasm.

In hindsight, we can see a few things Pierce called correctly.

Technology projections that are not based on realistic assessments of progress that has already been made and verified in the correspondence between theory, experimental results, and penetration into a market that exercises the technology in the real world can be unreliable in terms of both feasibility and cost in time and money.
Natural language translation and comprehension are important areas of research but research results from short term investments will not be substantial.  The problem complexity is high.

Whether Pierce meant that artificial intelligence is real stupidity literally or whether he was referring to the unrealistic expectations planted by researchers at MIT to obtain defense funding after the launch of Sputnik, we may never know for sure, but is easy to guess.

The comment about LISP is an obvious reference to Marvin Minsky's lab, which allegedly did not produce for the U.S. government what was promised.  It is merely an allegation because research failures can be fabrication to assist in classifying publicly funded research.

I say this as one who spent time with those in the closely linked AI lab at the United Technologies Research Center.  Everyone had national security clearances, and we all saw fabricated successes and fabricated failures.  There is good reason for mixing in misinformation when espionage is a known phenomenon.

Based on the way Pierce commonly wrote, the unfairness to computers was likely a satirical comment about research funding pitches and the unrealistic expectation they placed on the speed of computer technology development.  He is certainly not a person that would anthropomorphize a computer witlessly.

Pierce was given the leading role at the JPL because he gained credibility among fiscal stakeholders by telling the truth while at Bell Labs.  Some of the futurists in leading technology roles today might benefit from that history.

References

[1] Language and Machines, Pierce, 1966, ALPAC report to the U.S. National Science Foundation

[2] Whither Speech Recognition?, Pierce, 1969, Journal of the Acoustical Society of America, Vol. 46, No. 4


How do I calculate the probabilities of the BERT model prediction logits? I might be getting this completely wrong, but please let me first try to explain what I need, and then what's wrong.
I have a classification task. The training data has 50 different labels. The customer wants to differentiate the low probability predictions, meaning that, I have to classify some test data as "Unclassified / Other" depending on the probability (certainty?) of the model.
When I test my code, the prediction result is a numpy array. One example is:
[[-1.7862008  -0.7037363   0.09885322  1.5318055   2.1137428  -0.2216074
   0.18905772 -0.32575375  1.0748093  -0.06001111  0.01083148  0.47495762
   0.27160102  0.13852511 -0.68440574  0.6773654  -2.2712054  -0.2864312
  -0.8428862  -2.1132915  -1.0157436  -1.0340284  -0.35126117 -1.0333195
   9.149789   -0.21288703  0.11455813 -0.32903734  0.10503325 -0.3004114
  -1.3854568  -0.01692022 -0.4388664  -0.42163098 -0.09182278 -0.28269592
  -0.33082992 -1.147654   -0.6703184   0.33038092 -0.50087476  1.1643585
   0.96983343  1.3400391   1.0692116  -0.7623776  -0.6083422  -0.91371405
   0.10002492]]

I'm then using numpy.argmax() to identify the correct label.
My question is, is it possible to define a threshold (say, 0.6), and then compare the probability of the argmax() element so that I can classify the prediction as "other" if the probability is less than the threshold value?

Edit 1:
We are using 2 different models. One is Keras, and the other is BertTransformer. We have no problem in Keras since it gives the probabilities so I'm skipping Keras model.
The Bert model is pretrained. Here is how it is generated:
def model(self, data):
        number_of_categories = len(data['encoded_categories'].unique())
        model = BertForSequenceClassification.from_pretrained(
            "dbmdz/bert-base-turkish-128k-uncased",
            num_labels=number_of_categories,
            output_attentions=False,
            output_hidden_states=False,
        )

        # model.cuda()

        return model

The output given above is the result of model.predict() method. We compare both models, Bert is slightly ahead, therefore we know that the prediction works just fine. However, we are not sure what those numbers signify or represent.
Here is the Bert documentation.

Your call to model.predict() is returning the logits for softmax. This is useful for training purposes.
To get probabilties, you need to apply softmax on the logits.
import torch.nn.functional as F
logits = model.predict()
probabilities = F.softmax(logits, dim=-1)

Now you can apply your threshold same as for the Keras model.


Recognition of lines in a chalkboard I'm trying to develop a real-time application that, from the sequence of chalkboard images captured by a webcam, recognizes the lines being draw on it.

It must be able of recognize the lines from the chalkboard background, filter the presence in the image of the teacher, and translate these lines to some representation, something as a list of basic events like "start of line at xxx,xxx", "continue line at xxx,xxx", ...

After several days looking for references and bibliography, none is found. The most similar are the character recognition applications, in particular when they have a stroke recognition stage.

Any hint ?

Input will be a sequence as this one,  this one or this one (just without the presence of the students). I've expect the teacher not hidding his hand. We could imagine a start with an empty chalkboard.

Thanks.

Note: I am looking for more than an answer which says only something similar to "you can use a deep learning training it with two classes", without details or references.

I will assume that the camera is stable (no change in position, zoom or other settings during the video recording), otherwise the task becomes markedly more complicated.

Let's say that your dataset is an array of rasters (images in array format). You mention that you want to detect events "start line" and "end line".

One way of doing this would be to compute an approximate time derivative of your image series. For instance, take the image raster at index idx and the one right after, at index idx+1 (captured at instants $t$ and $t + \Delta t$, where $\Delta t$ is the sampling interval).

At coordinates $(i,j)$, this derivative could look something like: timeDerivative = (images[idx][j][i] - images[idx+1][j][i])/DeltaT. This is a crude estimate, and there are better ways of computing an approximate discrete derivative, but you get the idea.

Following this, you could declare a state of the recording: drawing line or not drawing line. The states, we assume are always alternating, as a teacher has to take his hand off the blackboard to draw a new line.

When a derivative with large values is detected (a region in the image goes from being black to white suddenly) and the state is "not drawing", the event "start line" is recorded and the state switches to "drawing". While a derivative with large values continues to be detected as time goes on in the vicinity of the previous spot with a large derivative, nothing changes. Once this is no longer true, the state changes to "not drawing" and the event "stop line" is recorded at the last location with a large derivative.

This is the main idea, which can be improved with:

Defining the area of the blackboard, either by hand or automatically
Thresholding the images to better isolate the chalk trace from the blackboard
Using a tracker such as a Kalman filter to know where to look next for the chalk


What is the difference between Bayes-adaptive MDP and a Belief-MDP in Reinforcement Learning? I have been reading a few papers in this area recently and I keep coming across these two terms. As far as I'm aware, Belief-MDPs are when you cast a POMDP as a regular MDP with a continuous state space where the state is a belief (distribution) with some unknown parameters.
Are they not the same thing?

Both Belief-MDPs and Bayes-Adaptive MDPs (BAMDPs) are special cases of POMDPs and their state space is augmented with a belief over their unobserved/hidden variables.
In a belief-MDP, the hidden variables can change over the course of an episode. (Eg. Both the position and the uncertainty in the position of the robot can vary during an episode).
In a  BAMDP, the hidden variables are usually the attributes of the transition/reward function and are held constant during an episode. (Eg. In a robot locomotion task, ground friction or load - dynamics attributes, current goal location - reward function attributes. Though variables defining these attributes are unknown to the agent and the agent has to infer them, the actual variables remain unchanged during an episode)
References:

M Ghavamzadeh, S Mannor, J Pineau, Aviv Tamar - Bayesian reinforcement learning: A survey. Foundations and Trends in Machine Learning, 8(5-6):359-483, 2015.

L Zintgraf, K Shiarlis, M Igl, S Schulze, Y Gal, K Hofmann, and S Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. arXiv preprint arXiv:1910.08348, 2019.


Why is the policy loss the mean of $-Q(s, \mu(s))$ in the DDPG algorithm? I am trying to implement the DDPG algorithm based on this paper.
The part that confuses me is the actor network's update.
I don't understand why the policy loss is simply the mean of $-Q(s, \mu(s))$, where $Q$ is the critic network and $\mu$ is the policy network.
How does one arrive at this?

This is not quite the loss that is stated in the paper.
For standard policy gradient methods the objective is to maximise $v_{\pi_\theta}(s_0)$ -- note that this is analogous to minimising $-v_{\pi_\theta}(s_0)$. This is for a stochastic policy. In DDPG the policy is now assumed to be deterministic.
In general, we can write
$$v_\pi(s) = \mathbb{E}_{a\sim\pi}[Q(s,a)]\;;$$
to see this note that
$$Q(s,a) = \mathbb{E}[G_t | S_t = s, A_t=a]\;;$$
so if we took expectation over this with respect to the distribution of $a$ we would get
$$\mathbb{E}_{a\sim\pi}[\mathbb{E}[G_t|S_t=s, A_t=a]] = \mathbb{E}[G_t|S_t=s] = v_\pi(s)\;.$$
However, if our policy is deterministic then $\pi(\cdot|s)$ is a point mass (a distribution which has probability 1 for a specific point and 0 everywhere else) for a certain action, so $\mathbb{E}_{a\sim\pi}[ Q(s,a)] = Q(s,a=\pi(s)) = v_\pi(s)$. Thus the objective is still to maximise $v_\pi(s)$ it is just that now we know the policy is deterministic we say we want to maximise $Q(s,a=\pi(s))$.
The policy gradient of this term was shown to be
\begin{align}
\nabla_\theta Q(s,a=\pi_\theta(s)) & \approx \mathbb{E}_{s \sim \mu}[\nabla_\theta Q(s,a=\pi_\theta(s))]\;; \\ & =  \mathbb{E}_{s\sim\mu}[\nabla_aQ(s,a=\pi(s)) \nabla_\theta \pi_\theta(s)]\;;
\end{align}
where if we put a minus at the front of this term then we would arrive at the loss from the paper. Intuitively this makes sense, you want to know how much the action-value function changes with respect to the parameter of the policy, but this would be difficult to directly calculate so you use the chain rule to see how much the action-value function changes with $a$ and in term how much $a$ (i.e. our policy) changes with the parameter of the policy.
I realise I have changed notation from the paper you are reading so here $\pi$ is our policy as opposed to $\mu$ and here where I have used $\mu$ I take this to be the state distribution function.