What is the difference between assisted and unassisted learning in relation to AI? Is this related to supervised and unsupervised machine learning?  Is it related to AI assisted human learning, and what is the distinction?

Also, why is assisted machine learning seen as an opportunity and unassisted machine learning seen as a threat?

Although there doesn't seem to be standardization of the terminology involving assisted learning, I've been noticing this concept, and the distinction, popping up in articles recently, surely driven by important milestones in unassisted machine learning.

My understanding is that Algorithmic Intelligence has traditionally been based on complex decision-making algorithms that are trained to some degree by the programmers.  (In essence, given a bit of a head start by utilizing human knowledge in regard to the problem.)
By contrast, unassisted machine learning is an Algorithmic Intelligence that learns only by its own analysis of models and problems.  No human knowledge goes into the automata, which develops intelligence entirely on its own.

A passage from Matthew Lai's Giraffe Chess paper may provide some insight:

  "This report presents Giraffe, a chess engine that uses self-play to discover all its
  domain-specific knowledge, with minimal hand-crafted knowledge given by the programmer.
  Unlike previous attempts using machine learning only to perform parametertuning
  on hand-crafted evaluation functions, Giraffe’s learning system also performs
  automatic feature extraction and pattern recognition. The trained evaluation function
  performs comparably to the evaluation functions of state-of-the-art chess engines - all of
  which containing thousands of lines of carefully hand-crafted pattern recognizers, tuned
  over many years by both computer chess experts and human chess masters."

DeepMind, for instance, is setting automata on video games and letting the automata learn with no human guidance whatsoever, and getting very good results.

As to why unassisted machine learning is worrisome relates to the issue of human/AI value alignment and the control problem: if we don't know what an an automata is thinking, or even how smart it has become, how can homo sapiens sapiens be assured of positive outcomes for our species?


Why is exact inference in a Bayesian network both NP-hard and P-hard? I should show that exact inference in a Bayesian network (BN) is NP-hard and P-hard by using a 3-SAT problem.

So, I did formulate a 3-SAT problem by defining 3-CNF:

$$(x_1 \lor x_2) \land (\neg x_3 \lor x_2) \land (x_3 \lor x_1)$$

I reduced it to inference in a Bayesian network, and produced all conditional probabilities, and I know which variable assignment would lead for the entire expression to be true.

I am aware of the difference between P and NP. (Please correct me if I am wrong):

Any P problem with an input of the size $n$ can be solved in $\mathcal{O}(n^c)$. For NP, the polynomial-time cannot be determined, hence, nondeterministic polynomial time. The question that scientists try to answer is whether a computer who is able to verify a solution would also be able to find a solution. P= NP?

However, I am still not sure how I can prove that exact inference in Bayesian network is NP-hard and P-hard.

It's not completely clear from your question, but it looks like you want to prove that exact inference in a Bayesian Network is both NP-Hard and P-Hard.

It appears that you have proven that it is NP-Hard, but are unsure how to show that it is also P-Hard.

This is more of a TCS question than an AI question, but shouldn't be too difficult. You just need to pick a P-Complete problem and reduce it to BN.


Continuous ground truth in supervised (metric) learning? I am writing my thesis in the field of (deep) metric learning (DML). I am training a network in the fashion of contrastive / triplet Siamese networks to learn similarity and dissimilarity of inputs. In this context, the ground truth is commonly expressed as a binary. Let's take an example based on the similarity of species:

Image A: german shepard (dog)
Image B: siberian husky (dog)
Image C: turkish angora (cat)
Image D: gray wolf (wolf)

  Image A and B are similar: same species, same sub-species (canis lupus) -> 1.0 == TRUE

  Image A and C are dissimilar: different species (canis lupus vs. felis silvestris) -> 0.0 == FALSE

  Image A and D ? same species, but different sub-species -> 0.8

Which metric learning approaches use a continuous ground truth for learning?

I could imagine that there is a lot of research out there using a continuous ground truth in classification settings. For instance to learn that the expression of a face is "almost (60%) happy", or more controversial, an image of a person depicts a "70% attractive person". Also in this fields I would be happy for hints / links.

Remarks:

I don't ask for opinions on whether this makes sense or not.

See Deep Metric Learning Beyond Binary Supervision in CVPR 2019


What approaches are there to apply AI to global economic processes? Today, AI is mainly driven by own-profit-oriented companies (e.g. Facebook, Amazon, Google). Admittedly, there's a lot of AI in the health sector (even in the public health sector) and there's a lot of AI in the sustainability sector – but also mostly driven by obviously own-profit-oriented companies (e.g. Tesla, Uber, Google).

On the other side, one often hears from hard-core economists that centrally planned (= public-profit-oriented) economies (or economic principles) are "the work of the devil" - and that they failed all over history (sometimes for understandable reasons).

But intelligently planning global economic processes and applying these plans with the help of state-of-the-art AI - given the huge amounts of really big data available, and given the argument that globalization is finally for the benefit of all - would seem to be a rewarding endeavour, at least for parts of the AI community.

Why isn't this endeavour undertaken more decidedly? (Or is it?)

Where do I find approaches to apply AI to global economic processes? (Not only describing and understanding but mainly planning and executing?)

It's currently just too complex
The different sources of information are too varied, in economics this is often referred to as a local knowledge problem, which hampers many large scale plans. Humans can react to slight differences like respecting local traditions, landscapes, history but an artificial intelligence would (currently at least) struggle not to generalise over such a large scale as a whole country's economy.
The real work in this case (and actually most 'AI' tasks) would be collecting all the necessary data. Here that part job is currently insurmountable.
Currently a human lead planned economy would do a better job.


What is the intuition behind the Xavier initialization for deep neural networks? The aim of weight initialization is to prevent layer activation outputs from exploding or vanishing during the course of a forward pass through a deep neural network

I am really having trouble understanding weights initialization technique and Xavier Initialization for deep neural networks (DNNs).

In simple words (and maybe with an example), what is the intuition behind the Xavier initialization for DNNs? When should we use Xavier's initialization?

Weight initialization is one of the most critical factors for successfully training a deep neural network. This explanation by deeplearning.ai is probably the best that one could give for the need for initializing a DNN with Xavier initialization. Here is what it talks about in a nutshell:

The problem of exploding and vanishing gradients has been long-standing in the DL community. Initialize all the weights as zeros and the model learns identical features across all hidden layers, initialize random but large weights and the backpropagated gradients explode, initialize random but small weights and gradients vanish. The intuition is aptly captured by this simple mathematical observation: $1.1^{50} = 117.390$, while at the same time, $0.9^{50} = 0.00515$. Note that the difference between the two numbers is just $0.1$ but it has a tremendous effect when multiplied repeatedly! A typical NN is a series of function compositions involving weight matrices and linear/non-linear activation functions. When stripped to a bare minimum, it essentially is a series of matrix multiplications. Therefore, the way in which the elements of these weight matrices are initialized plays a major role in how the network learns.

The standard weight initialization methods come into the picture here. They reinforce what are the de-facto rules of thumb when it comes to weight initializations: (1) the mean of the activations should be zero, and (2) the variance of these activations across all the layers should be the same.

Note: The link given above has complete mathematical justification for why Xavier initialization works, along with an interactive visualization for the same.


Is there an AI technology that can predict human behaviour? Is there an AI technology out there or being developed that can predict human behaviour, given that we as humans are irrational decision-makers?
I'm looking at this from an economic standpoint - the issue with current economic models is that they assume that humans are perfectly rational, but obviously this isn't the case. Could AI develop better models and therefore produce better models of recessions?

GPT2 predicts the next word that people will say. https://openai.com/blog/better-language-models/
Facebook predicts what will make you keep using their site.
Youtube predicts what videos you will click on.


Positive bias causes the calculation of incorrect gradients I have a data set with a positive bias (an image, where the values range from 0 to 1), that seems to be causing my network to calculate incorrect gradients.

If I just use the raw image as input, of shape (1,28,28), gradient checking fails  on the convolution layers, producing the following:

conv0: 1.1387914962261193e-06 1.4299464517486544     77.73286194827031
conv1: 7.635784666538102e-06  0.13917264577152977    2.5142484667483797e-06
conv2: 5.935327486981425e-07  2.706137838441397      1.6290818256837689e-06
fc0:   6.053556079013436e-07  1.1083181748299355e-05 2.0993768208698443e-06
fc1:   2.5819794868905293e-08 4.499689509732119e-06  1.108312270578618e-05

Where the first value is the absolute difference between the calculated weight updates, and the gradient tested weight updates, second being bias', and third being the calculated gradient to be passed to the previous layer. If this value is less than 0.1, it's a good indication the gradients are correct, as I use an epsilon of 1e-5.

Strangely if I use a randomly initialised input: inpt = np.random.randn(*im.shape)) using the same seed as above for network initialisation, then I get the following differences:

conv0: 0.002403805867625952   1.1140792221336904e-05 0.0004023637406479723
conv1: 0.0011835450640199086  6.134154765691235e-06  0.00014655404401104377
conv2: 0.00021913017743990792 0.0032477812666108496  7.169014825140573e-05
fc0:   0.04510183914723059    6.425385741260989e-06  1.709243452626911e-05
fc1:   0.0014606421065230097  1.9235668613026775e-06 5.995613024721222e-06

Would anyone have any idea what be causing this? I'm truly at a loss here, as I do the same procedure regardless of the numbers in the input, using the following to find the gradients to be passed back to the previous layer (for conv layers):

t = np.pad(grad, ((0,0), (self.xpad[1],self.xpad[0]), (self.ypad[1],self.ypad[0])))
self.gradient = np.zeros(self.incoming.output.shape)
for j in range(self.gradient.shape[0]):
    for i in range(t.shape[0]):
        self.gradient[j] += signal.convolve(t[i], self.kernels[i,j], mode='valid')

(This would be signal.correlate if I was using a convolution on the forward pass, but I'm not. Important to note that a convolutional neural network actually uses correlation on the forward pass, and only convolves on the backward)

EDIT:

I have done some more testing and found that if I change epsilon even slightly, I get MASSIVELY different results for the tested gradient. It seems decreasing epsilon increases the magnitude of the gradients exponentially. This is very strange behaviour, and I'm really not sure what is causing it. Could it be a discontinuity in ReLU where somewhere along the way I happen upon exactly 0?

Ok, so I was right in assuming it was the discontinuity at x=0 for relu(x). The positive bias actually had nothing to do with the incorrect gradients. As for why in my edit I was getting such massive differences between gradient values, it was because I initialised the input to be integers, and upon adding a decimal, no matter how small (> 0 though), numpy would jump entire integers, so -2 + 0.0001 = -1.

This is reproduced by:

>>> a = np.random.randint(-3,3,(1,3,3))
>>> a
array([[[-2,  1, -2],
        [-2, -1, -3],
        [-3,  0,  1]]])
>>> a[0][0][0] += 0.0001
>>> a
array([[[-1,  1, -2],
        [-2, -1, -3],
        [-3,  0,  1]]])

The problem with my input is large amounts of it were 0, so it produced a bunch of discontinuous gradients. As such, I simply check both sides of the graph (have epsilon positive then negative), and if they disagree, I set the value to 0, as in that case I'd just ignore any gradient passing through that node.


Standard deviation of the total input to a neuron Raul Rojas' Neural Networks A Systematic Introduction, section 8.2.1 calculates the standard deviation of the output of a hidden neuron.

From:
$$
\sigma^2 = \sum^n_{i=0}E[w_i^2]E[x_i^2]
$$

When I try what I get is (with $E[x_i^2] = \frac{1}{3}$ and $w_i \in [-\alpha, \alpha]$):

$$
\sigma^2 = \sum^n_{i=0}E[w_i^2]E[x_i^2]
= [n\frac{(\alpha-(-\alpha))^2}{12}][n\frac{1}{3}]=n^2\alpha^2\frac{1}{9}
$$

$$
\sigma = \sqrt{n^2\alpha^2\frac{1}{9}}=n\alpha\frac{1}{3}
$$

But Raul Rojas concludes:

$$
\sigma = \frac{1}{3}\sqrt{n}\alpha
$$

Am I missing some implicance of the law of large numbers use for the input to the node?

Thank you for your time :)

If n different edges with associated weights $w_1, w_2, . . . , w_n$ point to this node, then after selecting weights
  with uniform probability from the interval $[−α, α]$, the expected total input
  to the node is $$\sum_iw_ix_i$$.
  By the law of large numbers we can also assume that the total input to the node has a Gaussian distribution.

Note that the input to a node  is $$w_1x_1 + w_2x_2 + ... + w_nx_n $$

The variance of the total input to a node is:

$$\sigma^2 = \sum^n_{i=1}E[w_i^2]E[x_i^2] = n(E[w_1^2]E[x_1^2])
= n([\frac{(\alpha-(-\alpha))^2}{12}][\frac{1}{3}])=n\alpha^2\frac{1}{9}$$

since inputs and weights are uncorrelated.

I have taken out the summation and replaced it with $n$ times the variance of an input and a weight -- this can be done because both are independent and identically distributed in their own right and are independent of each other too. Variance of $w_1$ is the same as variance of $w_n$.

  Am I missing some implicance of the law of large numbers use for the input to the node?

Yes.

Implication of Law of Large Numbers:

Note that the random variable $E(w_ix_i)$ in  $$E(\sum^n_i(w_ix_i)^2)$$ is written as product of the random variables $E(w_i)$ and $E(x_i)$ $$\sum^n_{i=1}E[w_i^2]E[x_i^2]$$. This can only be written when the two random variables are independent. And the switching of expectation and summation is due to the linearity of expectation property.
How do we get independence here? LLN comes to the rescue.

That comes from the fact that -- according to the law of large numbers -- the random variable (total input) follows a normal distribution which implies the two random variables i.e. the weights and the inputs also follow normal distribution -- and it is also given that the two random variables are uncorrelated and thus they are independent.


How to put commands into local database in c# visual studio for voice recognition How can i add local database for voice recognition for visual studio c#. I want to get the commands from the database and also the function for send-keys I want to get it from the database how will I do that?

The easiest way is by using System. Speech. Recognition; here is described how to use it

Just change this code:

colors.Add(new string[] { "red", "green", "blue" });

And add results of a database query, for example, with Entity Framework and LINQ instead of new string [] {"red", "green", "blue"}. Something like this:

WordsToRecognize.Add(
context.MyTable
       .Where(x => x.Type == "type")
       .Select(x => new { x.Text }).ToList());

I'm not tested the code above, it just pseudocode


Convolutional layer to Fully Connected Layer implementation Im implementing a neural network framework from scratch in C++ as a learning exercise. There is one concept I don't see explained anywhere clearly: How do you go from your last convolutional/pooling/something layer which is "3 dimensional" to your first fully connected layer in the network?

Many sources say, that you should flatten the data. Does this mean that you should just simply create a $1D$ vector with a size of $N*M*D$ ($N*M$ is the last conv. layer's size, and $D$ is the number of activation maps in that layer) and put the numbers in it one by one in some arbitrary order?

If this is the case I understand how to propagate further down the line, but how does backprogation work here? Just put the values in reverse order into the activation maps?

I also read, that you can do this "flattening" as a tensor contraction. How does that work exactly?

Yes, you are correct (I think it is quite easily implementable in C++ with pointers). The arbitrary order is to be maintained though, since Fully Connected Neural Nets are "Translationally Invariant" i.e. you have to make sure if pixel $(1,5,6)$ is being supplied to node $38$ or being indexed as $37$ as a single datapoint to be input to a Fully Connected Neural Network, then from then on it must be fixed (cannot put say pixel $(1,6,5 )$ in node $38$.

Backpropagation works the same as it always works, it is tough to give a verbal explanation so I will give you this picture:

So, basically if you visualise like this you understand you have to differentiation will propagate, the "flattening" is only reshaping the value lookup table it is not changing the way the values affect final loss, so if you take gradient w.r.t each values and then convert it back to a $3D$ map same way as before and then propagate the gradients as you were doing in previous layers.


Is it possible to teach an AI to edit video content? Every week I will get a lot of videos from a game that I play, outside the game where you throw wooden skittle bats at skittles, and then I will cut videos, so that, at the end. there is video only about throws.

The job is simple and systematic. I have a lot of videos, so I was wondering:

Is it possible to teach AI to cut videos from the right place?

I was thinking to ask help or guidance where to start to solve this problem.

I can't use only sound, because sometimes you can hear skittles hit from outside of the video.  I also can't just use movement activity, because sometimes there are people moving around the field. Videos are always filmed from a fixed stand, so it should make it easier. So, is it possible and where to start?

Here is another example: https://www.youtube.com/watch?v=sHu6yMBV3xU

I believe the answer is yes, but the video edition is mostly programmatic. The AI part comes with detecting the right spots to cut.

You want to detect the right portion when someone picks a wooden skittle
you want to detect when the Skittles stop moving on the ground (to detect this, also when they start moving)

These will give you timestamps on the video. The next step is to add some padding and run the drop-down frame operations in the video, which can all be done with a simple script.

You might find this video very similar in concept, and maybe even as code source: https://youtu.be/DQ8orIurGxw

For the recognition of the objects in the video frames, there are lots of options. You might look into computer vision or object motion detection.


Will there be some promising techniques that can make AI greener and affordable in the future? The recent advances in machine learning were mostly achieved by the hardware, and the hardware is said to continue driving the development of AI, but I was still shocked by this thread which reads that the most expensive and largest model would cost 1B dollars in 2025. And I learned that universities are suffering from an academic AI brain drain partly due to the scarce hardware resources.

Some people proposed the so-called Green AI that encourages sustainable AI development but provides few constructive methods to prevent the trend.
I wonder if the redder and redder AI would be in fact truly inevitable. It seems to me that all companies should build an expensive compute infrastructure to be competitive, but I think the investment would be very risky since most companies cannot get a higher return.
But on the other hand, we human beings have evolved tens of millions of years or billions of years(life) on earth with hundreds of billions of brains that have ever lived on earth as a whole "human brain". The biological wetware seems much much redder than the nowadays hardware and has consumed much much more energy than all the supercomputers. To make machines as intelligent as we humans shouldn't we pay as high a price? It reminds me of the NFL theorem but it should be imprecise in this scenario.
So, will there be some promising techniques on the algorithm side that can make AI greener, affordable and sustainable in the future? If not, could anyone please explain why AI should be unavoidably red and inevitably redder?

As far as I know, green AI and red AI are very recent terms and/or research areas, but their importance will or should definitely be more highlighted in the coming years (for obvious reasons).
I don't know how many people are actively and directly researching this topic, but, in the past, I've already read a research paper on the topic, so a few people are already trying to at least raise awareness about these issues. Moreover, they also suggest that researchers report the number of floating-point operations (FLOPs) performed during the experiments, in addition to the usual performance metrics (such as the accuracy). I don't remember all the details of this paper, but I really recommend that you read it. This another possibly useful paper that people interested in this topic might want to read.
Although I'm currently not doing research on this topic, from an algorithmic perspective, improvements in the following areas will definitely contribute to greener AI

zero-, one- or few-shot learning
transfer learning
sample efficiency (in reinforcement learning)

Why? Because learning from fewer samples typically means fewer computations, so fewer emissions.
So, I think that making AI greener is definitely possible, as it's possible to use an electric car rather than a petrol/diesel car, although more research on the topics mentioned above (and other topics) needs to be done to make it practically useful.


How do I avoid an agent to tend to terminate in a negative state when time needs to be taken into account? In an unknown environment, how do I avoid an agent to tend to terminate its trajectory in a negative state when time needs to be taken into account?

Suppose the following example to make my question clear:

A mouse (M) starts in the bottom left of its world
Its goal is to reach the cheese (C) in the top right (+5 reward) while also avoiding the trap (T) (-5 reward)
It should do this as quickly as possible, so for every timestep it also receives a penalty (-1 reward)

If the grid world is sufficiently large, it may actually take the mouse many actions to reach the cheese.

Is there a scenario where the mouse may choose to prefer the trap (-1*small + -5 cumulative reward) versus the cheese (-1*large + 5 cumulative reward)?  Is this avoidable?  How does this translate to an unknown environment where the number of time steps required to reach the positive terminal state is unknown?

This is a common problem in reward shaping. You want a certain behavior from you agent but its challenging to describe it completely in terms of rewards. This situation you are describing is challenging specifically because as the grid world grows, the chance of randomly stumbling onto the goal state becomes less likely AKA the problem of exploration.
There are a few techniques that can be used to address this problem though, here are some.

0) This is an emergent property of the environment and gamma (0 based because its more immediate to your problem:p)

If gamma is small, your agent will value rewards that are in its immediate future more highly whereas as gamma approaches 1, the agent values rewards that are further in its future. In your grid world, the the size of the grid affects how this gamma affects your agent. Like in your example, if your grid was 100x100, if the trap was close to the agent, you would have to have a gamma closer to 0 in order for your agent to avoid the trap because its worse than moving to a cell that isn't a trap. This is interesting because the whole purpose of gamma is to increase the weighted value of temporally distant rewards but when you make the trap more favorable than the goal, going to the trap is the optimal strategy. :)

1) Include more observation data to your model

This isn't always a possibility but depending on your application and what you have available, you may be able to give your agent missing information that might be necessary to its ability to solve your task. For instance, in your infinite grid example, you may include the distance between the agent and the goal or the direction of the goal.

2) Include a reward that helps to shape the direction of progress.

One could easily create an infinite grid world where there isn't actually a goal but rather a continuing task where the agent has to cover distance in a desired direction while avoiding obstacles. How would you approach this problem? Perhaps a reward that specifically looks at the number of cells visited by the agent in the desired direction over time aka the discrete analog to velocity. Of course this is still dependent on a know direction but thinking about how to one might handle the "limit" of your environment as it grows (e.g., adding more and more cells to the grid world) helps to give an intuition of what your agent could be missing.

3) Use curiosity based approaches
Following from 2), if the direction isn't known, one thing to consider is rather than giving a penalty for each timestep, instead incentivize the agent to be faster, rewarding visiting "infrequently visited" states. As the task requires that the agent performs as quickly as possible, remaining or returning to a previously visited state clearly doesn't benefit the agent. Taking this notion of rewarding "visiting infrequently/unyet visited/ states" further results in the recent research topic of using curiosity to have RL agents that have novel exploration strategies.

Although there are many (often debated) ways to define curiosity, they all share an idea of giving the agent a bonus when it has entered a state that has not been visited before. A paper that gives a good recap of curiosity methods and also introduces a novel approach is Random Network Distillation from OpenAI.


Why do layered neural nets struggle with continous data? In this article here, the writer claims that a new type of neural net is required to deal with data that is both continuous, and also sparsely sampled.

It was my understanding that this was the entire purpose of techniques that use neural nets, to make assumptions about a system with a non-continuous data set.

So why do we need to switch to a non-layered design to deal with these data sets better?

They struggle because if your network have an inductive bias towards modeling datasets which are described with ODEs well, you will learn faster, and with smaller dataset. I think, this what the authors of the original article meant.

In a similar way, CNNs recognize images much better, because their features are translation invariant whereas fully connected net needs to learn to recognize a cat in each different position from scratch.


What are the dangers of AI applications for nuclear industry? I've found this old scientific paper from 1988 about introduction of AI into nuclear power fields.

Were or still are there any dangers by application of such algorithm? Are nuclear power plants or human life in risk if the algorithm will fail?

Especially applications to the core, like cooling systems and other components which can be affected in negative way.

Any technology in the nuclear industry represents variance--it may be an improvement in safety or efficiency, or it may contain some unseen defect that allows a catastrophe to happen.

But the simple possibility of harm isn't enough to swing the decision one way or the other. The application of AI methods--whether to the real-time control of plant variables, or the early detection of problems, or to the design of plants and their components--seems likely to be as beneficial as in other realms.

For example, check out the publication list of a lab active in this area. Their paper I'm most familiar with is one in which they build a fault detector paired with a fault library classifier, so that the operators can be alerted not just that something is abnormal but what fault has probably occurred. This is done in such a way that standardized plants (such as, say, the French nuclear system) can share records with each other, meaning that any plant has the experience of every plant at their fingertips.


Using convnet to classify language of text contained in images I hope this question is not too broad or general. I have a very large set of images all of which contain text (some have more, some less). All of them have been tagged as containing, say, English text or Korean. I wonder if convolutional neural networks would be a good approach to classify these images as containing English vs. Korean. Or is there any existing literature/method that does this already. Crucially though, I am not interested in "understanding" the text, so this is not an NLP task but, I suppose, a task of classifying orthographies in the images.

This sounds like a fairly straightforward task, with low risks. I think the proper term is that you're you are trying to detect the script, which would be either Latin ( "English") or Hangul ("Korean"). The chance is that you end up learning fonts, though.


Do we need non-linear activation function in neural networks whose task isn't classification? While researching why we need non linear activation functions, all the explanations revolve around neural network being able to separate values that aren't linearly separable.
So I wonder, if we have a neural network whose task is something else, say predicting an output value of a time series, is it still important to have an activation function that is non linear?

Yes, definitely.
In the simplest example, predicting an output value for a time series is classification. You take in the previous time steps and classify what is the most likely next value. You could do this with a RNN (Recurrent Neural Network) for example.
If the activation functions are all linear, the nerual network is just a glorified linear regression. Think of it like this: a neural network is trying to approximate a complicated function in $n$ dimensional space. It does this by combining operations on a series of known functions, to get a resultant function that hopefully mimics the desired function. The issue with combining linear functions is the only thing you'll ever get at the end is a linear function.
As a concrete example, try and approximate the function $y = x^3 + x^2 -x -1$ by adding a series of linear functions together. You'll find pretty quickly this is useless. However, if you use a non-linear function, such as a ReLU (Rectified Linear Unit) you can quite easily approximate this function. See this implementation on desmos.
If a problem has any sort of complexity to it, the function it follow is likely incredibly complicated, and futile to approximate using linear equations.


Audio enhancing using AI? (removing background noise during lecture recording) I wonder if it's possible to 'clean up' an audio recording of a lecture from a smartphone, using some type of AI system?

Yes, it is possible. Usually the noise reduction is done using regular signal processing methods, such as spectral subtraction due to demand for low latency. But of course, modern methods of deep learning is applicable to this problem. For example variational autoencoder is the first that come to my mind, you can check this  project. As another example you can look at this project , which is based on WaveNet.


Is one big network faster than several small ones? The basis of my question is that a CNN that does great on MNIST is far smaller than a CNN that does great on ImageNet. Clearly, as the number of potential target classes increases, along with image complexity (background, illumination, etc.), the network needs to become deeper and wider to be able to sufficiently capture all of the variation in the dataset. However, the downside of larger networks is that they become far slower for both inference and backprop.

Assume you wanted to build a network that runs on a security camera in front of your house. You are really interested in telling when it sees a person, or a car in your driveway, or a delivery truck, etc. Let's say you have a total of 20 classes that you care about (maybe you want to know minivan, pickup, and so on).

You gather a dataset that has plenty of nice, clean data. It has footage from lots of times of the day, with lots of intra-class variation and great balance between all of the classes. Finally, assume that you want this network to run at the maximum possible framerate (I know that security cameras don't need to do this, but maybe you're running on a small processor or some other reason that you want to be executing at really high speed).

Is there any advantage, computationally, to splitting your network into smaller networks that specialize? One possibility is having a morning, an afternoon/evening, and a night network and you run the one corresponding to the time of day. Each one can detect all 20 classes (although you could split even farther and make it so that there is a vehicle one, and a person one, and so on). Your other option is sharing base layers (similar to using VGGNet layers for transfer learning). Then, you have the output of those base layers fed into several small networks, each specialized like above. Finally, you could also have just one large network that runs in all conditions.

Question: Is there a way to know which of these would be faster other than building them?

In my head, it feels like sharing base layers and then diverging will run as slow as the "sub-network" with the most additional parameters. Similar logic for the separate networks, except you save a lot of computation by sharing base layers. Overall, though, it seems like one network is probably ideal. Is there any research/experimentation along these lines?

you need to understand about layerwise interpretability of neural networks before this . each layer is activated by some patterns "activations" that are much more complex than its previous layers (in terms of texture etc).
so deeper the network , more complex functions it can compute . therefore you can't replace a deeper network with a several shallow ones. i suggest you to skim through https://arxiv.org/abs/1310.6343 , In this paper it is proven that no t/2 layered network can capture the distribution that of a t-layered one.


How to use AI to depth map video? To be honest, I had no idea where to put this question, but it's sure that it's related to AI. I want to build an application which uses camera, and by the movement it can calculate the
-camera's position compared to the objects
-the objects creator and edge points by the movement.

What it means that if the camera is in a static position, it's just a picture. A set of coloured pixels. If we move the camera, we calculate the time, the gyroscope's values, but most importantly, we can have a comparison of two images taken by the same objects. This way:
-we can detect the edges
-from the edges, we can detect which is closer than the others

Today's phone camera's are accurate enough to create ~60 crystal clear images per second, and it should be enough resource to accurately create high res models from just moving the camera according to some instructions (that's why I'm surprised why it isn't existing in just a phone app). Here comes the problem. I think the idea is worth for the try, but I'm just a JavaScript developer. The browser can have access to the camera, with TensorFlow I can use machine learning to detect edges, but if I want to be honest, I have no idea where to start, and how to continue step by step. Can you please provide me some guidelines how it would be ideal to create the idea?

I think that the tasks you are referring to are the Simultaneous localization and mapping (SLAM) and, in particular, the Structure from Motion (SfM).

These methods are usually based on geometrical constraints and do not employ neural networks, but there exist some recent methods that make use of CNNs (such as this one).

Structure from Motion algorithms are a fundamental component of the Google ARCore (former Project Tango) and the Apple ARKit.
Unfortunately, these kits usually provide an interface to put elements in a specific position, but they don't provide the access to the 3d reconstruction of the scene.

If you'd like to play with SfM algorithms, I suggest you to start from this repository, but you can find a lot of other valuable tutorials online.


An approach on reading musical notes from photos I was looking around, a promising approach was this:

https://github.com/mpralat/notesRecognizer

the problem is:

it doesn't seem good enough. One should be able to read musical notes with lower quality images. You can see in her: "bad" images folder just tiny variations of lightning can already cause her problems with her high res images.

others are here, they all use high res sharp images:

https://github.com/suyalcinkaya/music-note-recognition

https://github.com/suyalcinkaya/music-note-recognition/blob/master/input_images/im2s.JPG?raw=true

https://github.com/nikolalsvk/note-play

https://github.com/nikolalsvk/note-play/blob/master/images/notes-1.png?raw=true

now this is unsatisfying if you want to snap a photo of some tunes, and want them to be recognized.

so what could one do to achieve a good solution?

I was thinking about treating the musical notes just like written letters. The computer can easily learn written characters with the arabic symbols. I wonder though, how easy would it be for a none arabic? e.g. in chinese or japanese, several characters combine into one. The same applies to music notes, they can be connected and form something slightly different through that e.g.

or:

in contrast to just simple notes like:

what would be a good approach to try ones luck, with interpreting those symbols successfully, even for slightly low res images or bit blurry deformed images. I'm not saying to read out a symfony out of a thumbnail. But less than optimal captures.

any subjective ideas or comments are more than welcome

refer to MNIST on kagle
to train model to recognize given set of notes in your case.

if you start training model will understand pattern depending on bits in your image.

issue comes with blurry images when last 2 sets of pics with minute change. will be difficult task for computer to distinguish.

there is one sure shot method which works almost every time but is resource intensive and needs plethora of data to train. is DNN.

this is my take on your question.


What is an "adversarial attack" I'm reading this really interesting article CycleGAN, a Master of Steganography and I understand everything up until this paragraph:

  we may view the CycleGAN
  training procedure as continually mounting an adversarial attack on
  G
  , by optimizing a generator
  F
  to generate adversarial maps that force
  G
  to produce a desired image. Since we have demonstrated
  that it is possible to generate these adversarial maps using gradient descent, it is nearly certain that the
  training procedure is also causing
  F
  to generate these adversarial maps. As
  G
  is also being optimized,
  however,
  G
  may actually be seen as cooperating in this attack by learning to become increasingly
  susceptible to attacks. We observe that the magnitude of the difference
  y
  ∗
  −
  y
  0
  necessary to generate
  a convincing adversarial example by Equation 3 decreases as the CycleGAN model trains, indicating
  cooperation of
  G
  to support adversarial maps

How is the CycleGAN training procedure an adversarial attack? I don't really understand the quoted explanation.

"Adversarial examples are inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake; they’re like optical illusions for machines.".

Source: "Attacking Machine Learning with Adversarial Examples"

You create an input and test against the output, tuning the input to maximize the error. There are different criteria for tuning the input, sometimes you might want the shortest possible input to create the largest error, often you would want very similar input to cause the greatest error.

Example: I take a \$1 bill and write a few more zeros after the one; you accept it as a \$1000 bill.


What is the name of an AI system that learns by trial and error? Imagine a system that controls dampers in a complex vent system that has an objective to perfectly equalize the output from each vent. The system has sensors for damper position, flow at various locations and at each vent. The system is initially implemented using a rather small data set or even a formulaic algorithm to control the dampers. What if that algorithm were programmed to "try" different configurations of dampers to optimize the air flows, guided broadly by either the initial (weak) training or the formula? The system would try different configurations and learn what improved results, and what worsened results, in an effort to reduce error (differential outflow).

What is that kind of AI system called? What is that system of learning called? Are there systems that do that currently?

Near solution to your problem definition is reinforcement learning. You can define some reward using the objective function and define some possible state space for the machine and finally solve the problem by reinforcement learning techniques (near to trial and error by learning the preferences).


Using sigmoid in LSTM network for multi-step forecasting I'm trying to develop a multistep forecasting model using LSTM  Network. The model takes three times steps as input and predicting two time_steps. both input and output columns are normalised using minmax_scalar within the range of 0 and 1.

Please see the below model architecture

Model Architecture

model = Sequential()
model.add(LSTM(80,input_shape=(3,1),activation='sigmoid',return_sequences=True))
model.add(LSTM(20,activation='sigmoid',return_sequences=False))
model.add(Dense(2))

In this case, using sigmoid as an activation function is it correct?

Yes, due the input, output being constrained between zero and one that would be the only viable activation function.


What to do if CNN cannot overfit a training set on adding dropout? I have been trying to use CNN for a regression problem. I followed the standard recommendation of disabling dropout and overfitting a small training set prior to trying for generalization. With a 10 layer deep architecture, I could overfit a training set of about 3000 examples. However, on adding 50% dropout after the fully-connected layer just before the output layer, I find that my model can no longer overfit the training set. Validation loss also stopped decreasing after a few epochs. This is a substantially small training set, so overfitting should not have been a problem, even with dropout. So, does this indicate that my network is not complex enough to generalize in the presence of dropout? Adding additional convolutional layers didn't help either. What are the things to try in this situation? I will be thankful if someone can give me a clue or suggestion.

PS: For reference, I am using the learned weights of the first 16 layers of Alexnet and have added 3 convolutional layers with ReLU non-linearity followed by a max pooling layer and 2 fully connected layers. I update weights of all layers during training using SGD with momentum.

Sorry if this is a bad use of answer to add comment but since my reputation is not high enough this is only way to leave a comment to OP's question.
I think some of the answers misunderstood the OP's intention.
Over fitting is used as a means to test the complexity of the model - if a model cannot overfit a small dataset then it's likely not able to generalize well.
It's not that OP misunderstood the meaning of over fitting.
For Instance, I think this discussion is relevant: https://stats.stackexchange.com/questions/492165/what-to-do-when-a-neural-network-cannot-overfit-one-training-sample


Are there independent evaluations of various major AI platforms? Has anyone had a chance to tinker with multiple major AI platforms such as TensorFlow, Cognitive Talk, Quill etc...

What are the strengths and weaknesses of different AI platforms?

Comprehensive articles that tackle this topic would be helpful.

In recent times different data science magazines and institutions have published their reviews of the top AI toolkits. In these reviews they tend to highlight the innovative features possessed by each platform as well as their reliability and ability to scale.

Below are a some evaluations of AI platforms that I recommend you have a look at:

KDnuggets review https://www.kdnuggets.com/2016/04/top-15-frameworks-machine-learning-experts.html

Paperspace review https://blog.paperspace.com/which-ml-framework-should-i-use/

Predictive Analytics review https://www.predictiveanalyticstoday.com/deep-learning-software-libraries/

Infoworld review https://www.infoworld.com/article/3026262/machine-learning/13-frameworks-for-mastering-machine-learning.html#slide4

Wikipedia comparison https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software

However if your main interest is to get an objective and unbiased assessment of the different ML toolkits. Then the best place to get such first hand reviews is at GitHub. Here you can look at the number of forks, stars and downloads, the update frequency, the number of issues, the number of contributors, the documentation, the license and even the code itself. This approach enables you to get an impartial and neutral appraisal directly from the developers themselves.

Below is my personal evaluation of the different ML toolkits:

Tensorflow - TensorFlow is a powerful open source library for numerical computation using data flow graphs. Some of the advantages of tensorflow are: It is portable and consequently can run on GPU's, CPU's, desktops, servers and mobile computing platforms. It comes with tensorboard which is useful in visualising and fine-tuning your network. Tensorflow has a large following with lots of researchers and developers using it. This means that most issues can be dealt with easily since they are usually the same issues that other developers run into.

Microsoft CNTK - Is a unified deep learning toolkit that describes neural networks as a series of computational steps via a directed graph. It delivers state of the art results when it comes to speech recognition, in recent times it has even outperformed humans in transcribing speech to text. The external CNTK toolkit available as open-source is also identical to the internal toolkit. This means you get to access all the features that internal Microsoft developers utilize in production.

Theano - Theano is a symbolic compiler library that builds an optimized Neural Network from mathematical expressions. A major advantage of theano is that it supports Python libraries meaning that a wide range of Python libraries i.e. Numpy, SciPy and Scikit-learn can easily be integrated with theano. Additionally theano has excellent high level API's such as keras and lasagne which greatly simplify the use of the Theano toolkit.

Torch - Is a scientific computing framework and script language that provides a wide range of algorithms for machine learning. The benefits of torch are: It has a very active open source community given that Facebook is standing behind it. Additionally it is based on Lua which is a very easy language to pick up. On top of that a significant number of AI researchers publish their research findings in torch.

Caffe - Is a deep learning framework originally developed at UC Berkeley. Some of the perks of using Caffe are: It has an excellent Matlab and Python interface, it supports functions such as ReLu straight out of the box furthermore Caffe excels at CNN's for images.


Fitness function in genetic algorithm based on an interval I am writing an app, where when a ball is shot from a canon it is supposed to land in a hole that is on a given distance. The ball is supposed to land between the distance of the beginning of the hole and the end of the hole. The size of the hole is 4m and the size of the ball is 0.4m. My problem is that I am not sure how to write the fitness function for this. The place where the ball falls should be close to this interval of [D, D+3.6], where D is the distance of the hole. If anyone could give me a hint on how to approach this problem, I would be grateful.

Genetic algorithms work best when given a scalar fitness value that increases smoothly, so that you can compare two population members regardless of whether they failed or succeeded at the task.

That usually requires you to analyse the problem, and come up with a measure that would improve as an individual gets closer to solving a task. It generally helps if you score better for an individual if they solve a problem "better". You want to avoid simple boolean success/fail metrics.

A simple measure of how well an individual has done is to use the absolute distance it was away from a "perfect" shot (where the middle of the ball hits the middle of the hole). The only issue with this is that a perfect shot scores 0, whilst a miss scores 2+, and you want the best result to have the highest fitness. This can be fixed simply, take the negative of the absolute distance:

$$F = -|D_{hole} - D_{ball}|$$

where $D_{hole}$ - and $D_{ball}$ are horizontal distances from origin to centre of each object.

There is no requirement for fitness score to be positive. This will score $F \lt -2$ for a miss, $-2 \lt F \lt 0$ for hit, and $0$ for a perfect shot.

For GAs you don't have to care as much about differentiability or gradient of the fitness function, provided it gives reasonably good ranks between quality of individuals. So there is no point using e.g. a squared error metric here, although you could if you wished.


How do I know if my backpropagation is implemented correctly? I'm working on implementation of the backpropagation algorithm for a simple neural network which predicts a probability of survival (1 or 0) and I can't get it above 80% no matter how much I try to set the right hyperparameters. I suspect that's because my backpropagation is implemented incorrectly since I tried 2 different types of code and both give me same results. Is my backpropagation implemented correctly? Also how can I improve my model to give a better prediction?

class NeuralNetwork(object):

def __init__(self, input_nodes, hidden_nodes, output_nodes, learning_rate):
    # Set number of nodes in input, hidden and output layers.
    self.input_nodes = input_nodes
    self.hidden_nodes = hidden_nodes
    self.output_nodes = output_nodes
    self.lr = learning_rate

    # Initialize weights
    self.input_hidden_weights = np.random.randn(hidden_nodes, input_nodes) # 10x7
    self.hidden_output_weights = np.random.randn(output_nodes, hidden_nodes) # 1x10

    # Sigmoid activation funciton
    self.sigmoid = lambda x: 1/(1+np.exp(-x))
    self.diff_sigm = lambda x: x*(1-x)

def train(self, input_list, label_list):

    # Create an array of inputs and labels
    inputs = np.array(input_list, ndmin=2).T # 7x1
    labels = np.array(label_list, ndmin=2) # 1x1

    # Forward propagation
    hidden_layer = self.sigmoid(np.dot(self.input_hidden_weights, inputs))
    output_layer = self.sigmoid(np.dot(self.hidden_output_weights, hidden_layer))

    final_output = output_layer

    # Error function
    output_errors = labels-final_output

    # Backpropagation
    output_delta = output_errors * self.diff_sigm(output_layer)
    hidden_delta = np.dot(self.hidden_output_weights.T, output_delta) * self.diff_sigm(hidden_layer)

    # Update the weights
    self.hidden_output_weights += np.dot(output_delta, hidden_layer.T) * self.lr
    self.input_hidden_weights += np.dot(hidden_delta, inputs.T) * self.lr

    """
    # Backpropagation
    hidden_errors = np.dot(self.hidden_output_weights.T, output_errors)
    hidden_grad = hidden_layer * (1.0 - hidden_layer)

    # Update the weights
    self.hidden_output_weights += self.lr * np.dot(output_errors.T, output_layer.T) # update hidden-to-output weights with gradient descent step
    self.input_hidden_weights += self.lr * np.dot(hidden_errors * hidden_grad, inputs.T)  # update input-to-hidden weights with gradient descent step
    """

For future reference, I will merely point you to a technique you can implement to test the correctness or lack thereof, of your backpropagation implementation.

Ps: don't feel too bad for having gotten it slightly wrong, "backpropagation is notoriously difficult to implement" - source :).

In fact, there is a technique called "Gradient checking" meant specifically for this purpose (for more information, see Andrew Ng's lecture video on gradient checking and, this notes). I would argue that even gradient checking is a little tricky to implement.

How does Gradient checking work:

Backpropagation computes the gradients (∂J/∂θ)(∂J/∂θ), where θ denotes the parameters of the model. J is computed using forward propagation and your loss function.
But because forward propagation is fairly straightforwards to implement, most people are usually confident that you got its implementation correct. So the trick is to use the value of J to verify your code for computing (∂J/∂θ)(∂J/∂θ).
We know that by definition, the gradient or derivative is given by:

(∂J/∂θ) = lim ε→0 J(θ + ε) − J(θ − ε) / 2ε

now, since we trust our calculation of (∂J/∂θ), we can easily compute the value of J(θ+ε)J(θ+ε) and J(θ−ε)J(θ−ε).

For more information, see here and, over here).


What is the best way to detect and recognize traffic signs in a picture? I'm working on a project for my college to recognize traffic signs in pictures. I searched a lot but can't find the best method to do it.
Can someone recommend me a paper, article, or even GitHub link that describes the best way to achieve this? It will be helpful.

Maybe this is helpful:
Recognising Traffic Signs With 98% Accuracy Using Deep Learning, by Eddie Forson.

Greetings
Mario


What type of neural network would be most feasible for playing a realtime game? For implementing a neural network algorithm that can play air hockey, I had two ideas for input, and I'm trying to figure out which design would be most viable.

The output must be two analog values that dictate the best position on half of the table for the robot to be at a specific point in time, evaluated 60 times per second.

Having consulted with a professor who has experience with implementing parallel algorithms, it was recommended to me to use a convolutional neural network with a single hidden layer, and directly process the image data as the input layer, after processing it to visualize a direct view of the table and somehow emphasize the puck and mallets with pre-processing. I have already started work on this and have successfully implemented object detection for the puck and mallets using OpenCV to get the center coordinates for all 3 entities.

However, having been able to successfully and accurately pre-process these data at 60 times per second, my thought was to feed those (after normalizing the values using the error function) directly on the input layer and possibly implement a deep learning algorithm that employs more than one hidden layer. The problem is that I don't have any experience implementing neural networks, so I'm not even sure what type of layer would be best for this, or how I should seed the weights.

Another reason I want to consider my idea is that given the relatively few inputs and outputs, probably won't need a GPU to execute forward-propagation 60 times per second, whereas with the convolutional network my professor recommended, I know I'll need to implement using CUDA somehow.

Which of the two input methods would be most recommended for this, and if I were to try and implement my idea, what types of layers should I consider? Also any recommendations for existing frameworks to use for either approach would be highly appreciated.

I recommend you read up on reinforcement learning. Seeing how AirHockey is similar to the old Atari game Pong, here is a write-up (with code) about how to implement a simple neural network, that plays the game; Deep Reinforcement Learning: Pong from Pixels.

As far as the choice of layers go, again, you should read up on deep reinforcement learning, perhaps study some existing models that work, for a similar task and take their architecture as your starting point. The above link is a nice introduction into such thinking.

As a continuation, I would recomend you employ one of the well known, and developed, libraries, such as TensorFlow or pyTorch. They will allow you to run your models on GPU, like you desire.

For a toolkit, that specializes to reinforcement learning, see openAI gym.


Have capsule neural networks been used to NLP problems? The capsule neural network seems to be a good solution for problems that involve hierarchies. For example, a face is composed of eyes, a nose and ears; a hand is made of fingers, nails, and a palm; and a human is composed of a face and hands.

Many problems in NLP can be seen as hierarchical problems: there are words, sentences, paragraphs, and chapters, whose meaning changes based on the style of lower levels.

Are there any research papers (which I should be aware of) on the application of capsule neural networks to NLP problems?

Are there related research papers, which have been investigating hierarchical complexity within the domain of NLP, which could be easily translated to Capsule Network?

For us to answer this question. First, we need to look at why capsule networks outperform convolution neural networks by as much as 45% in recognizing images that have been rotated, translated or are under a different pose. We can find Geof Hinton's paper on capsule networks here for reference https://arxiv.org/pdf/1710.09829v1.pdf

In a CNN architecture, a convolution layer is usually followed by a max-pooling layer. This is so that the lower levels can detect low-level features, like edges, while the high-level layers can detect abstraction like eyes. However, the application of max-pooling leads to the loss of important information regarding the location and spatial relationship between certain features.

On the other hand, this is where capsule networks excel, the way they represent certain features is locally invariant. This is why capsule networks can recognize images under different lighting conditions and deformations. They are likely to excel at applications such as video and object tracking but not necessarily NLP.

The current approach in NLP maps words and phrases to vectors. From there, we exploit the concept of vectors and distances between them (cosine, euclidean, etc.) to perform operations such as: finding the similarity between words and even documents, machine translation, and natural language understanding (NLU).

Capsule networks are unlikely to succeed in NLP. This is because algorithms that aim to find the hierarchical structure of natural languages or approaches that focus on grammar have met little success. Research by Stanford University aiming at finding the hierarchical structure of natural languages can be found here https://nlp.stanford.edu/projects/project-induction.shtml

Although conclusive research regarding other applications of capsule networks has not yet been conducted. They are likely to excel at applications such as video intelligence and object tracking but not necessarily NLP.


Is there a reason to choose regular momentum over Nesterov momentum for neural networks? I've been reading about Nesterov momentum from here and it seems like a nice improvement over regular momentum with no extra cost whatsoever.

However, is this really the case? Are there instances where regular momentum performs better than Nesterov momentum or is Nesterov momentum performs at least as good as the regular momentum all the time?

The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that

  Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the rate of convergence.

I'm not sure why this is, but the theoretical advantage depends on a convex problem, and from this, it sounds like the practical advantage does too - or at least, that it isn't applicable to typical neural network landscapes.

Perhaps it can be implemented more efficiently, but it seems to me that you need to do parameter updates twice (in order to calculate the gradient in a place you aren't moving two) and thus Nesterov requires more computation and memory than plain ole momentum.


In classification, how does the number of classes affect the model size and amount of data needed to train? When solving a classification problem with neural nets, be it text or images, how does the number of classes affect the model size and amount of data needed to train?
Are there any soft or hard limitations where the number of outputs starts to stall learning?
Do you know about any analysis of how the number of classes scales the model?
Does the optimal size increase proportionally with the number of outputs? Does it increase at all? If it does increase, is the relationship linear or exponential?

The most obvious way more classes increase the network size it the output layer, but I don't believe there is a rule of thumb for the size of the entire network.
As I understand it, there is no clear answer how big a network needs to be to achieve a certain performance with regard to the number of layers compared to the number of classes. This is a very active research field, just as an example compare the size of EfficientNet to other State of the art models when it was introduced and you can see the size difference.
Regarding the data needed, in The Deep Learning Book (https://www.deeplearningbook.org/) (which is a few years old now) they state as a general that with the models available at that time, you needed ~5000 examples per label for acceptable performance, while to exceed human performance (their words) you would need abut 10 million labeled examples.


Is a policy in reinforcement learning analogous to a field such as APF? If a policy maps states to actions in reinforcement learning, then for a path planning with obstacles, can't we simply use Artificial Potential Field fields for path planning and model policy mathematically as a field where the obstacles form repulsive field and goal form attractive field?
So, technically, is a policy simply a field?

Since most policies depend solely on actions and states/observations, then if you model the space of this field as the Cartesian Product of your state and action spaces, then the policy learns a surface over this combined space, similar to the way a field is parameterized.
The policy an agent learns could exhibit the same behavior as the field you describe above (obstacles form an repulsive field, and goal(s) form an attractive field).  However, unlike the field described above, it is not guaranteed that the learned policy will capture this behavior - the policy learned depends on:

The learning algorithm used
The approximators (e.g. neural networks) used for learning, and their respective hyperparameters
The formulation of the reward function
The number of episodes/total steps the policy/agent is trained over.

To sum this answer up, I believe you could train the policy in such a way (using the mechanisms above) such that it resembles the field you describe.


How can I design a hierarchy of agents each of which with different goals? I read some light material earlier about the possibility of building a hierarchy of agents, where the agents at the leaves solve primitive tasks while higher-level agents are optimized for orchestrating direct-descendant agents.

According to my understanding, each agent would have different objective functions and possible moves.

How can I design such a hierarchy of agents each of which with different goals? Are there any examples?

In the context of reinforcement learning, the idea of modeling your goal-oriented problem as a hierarchy of multiple sub-problems is called hierarchical reinforcement learning, which gives rise to concepts such as semi-Markov decision processes and options (aka macro actions). The article The Promise of Hierarchical Reinforcement Learning presents and describes the topic quite well, so I suggest you read it.
However, this idea of solving multiple sub-tasks in order to solve a bigger task isn't limited to RL. For example, in the paper Neural Programmer-Interpreters (NPIs), without referring to the traditional RL topics, a model is proposed to write programs by composing simpler ones.
Another example is genetic programming with automatically defined functions (ADFs), where sub-programs (the ADFs) can be re-used in different parts of the program if that's beneficial according to the fitness.
So, there are different ways you can design a system to solve a big task by solving multiple sub-tasks and then compose them. The approach that you choose depends on your use case. If you want to build programs, then NPIs can be a start. If you want to incorporate a time component in your system, then HRL is probably a viable approach. There are several HRL algorithms. Some of them (e.g. MAXQ-OP) have been successfully used to solve the RoboCup challenge.


Number of parameters in Keras/Tensorflow Dense layers I am a bit confused about how the number of parameters are calculated in Dense model for the Kera/Tensorflow.
For example, in the figure below I thought that both the statements were the same, but I found a different number of parameters for both.  In particular, I am talking about model.add(Dense(...)) command.

Check the documentation for Dense layer:

Note: If the input to the layer has a rank greater than 2, then Dense computes the dot product between the inputs and the kernel along the last axis of the inputs and axis 1 of the kernel (using tf.tensordot). For example, if input has dimensions (batch_size, d0, d1), then we create a kernel with shape (d1, units), and the kernel operates along axis 2 of the input, on every sub-tensor of shape (1, 1, d1) (there are batch_size * d0 such sub-tensors). The output in this case will have shape (batch_size, d0, units).

That is what happening in your first case - for input dimensions (4,1) you've got d0=4 and d1=1. So it creates a kernel of shape (1,32) that gets applied along the axis of dimension 4. That's why your output shape is (4,32) and you've got 32 weights + 32 biases = 64 parameters.
In second case you've got a "standard" 32 * 4 fully-connected weight matrix + 32 biases = 160.


Are humans intelligent according to the definition of an intelligent agent? Given the following definition of an intelligent agent (taken from a Wikipedia article)

  If an agent acts so as to maximize the expected value of a performance measure based on past experience and knowledge then it is intelligent

and given that we, humans, all make mistakes, which means that we are not maximizing the expected value of a performance measure, then does this imply that humans are not intelligent?

"the human mind is a battleground of higher level goals and lower level goals "— Marvin Minsky paraphrasing Sigmund Freud

I argue that in general human agents try to maximise a hierarchy of performance measures.

performance measures of humans

Survival of genetic data

Energy supply and Water
Sex

myriad subgoals....

Mysterious mental mechanisms which neuroscientists do not understand yet force the average human agent to maximise various evaluation metrics.
With the overarching goal of survival of genetic information. Successful genes are immortal. We are still under the yoke of an ancient genetic algorithm.

These measures are optimised throughout a humans life time. A 30 year old agent is better at survival than a 10 year old agent. A 30 year old agent makes fewer mistakes.

We remember our mistakes. Mistakes are burned into our memory by high levels of neurotransmitters (and reinforcing of synapses) so we don't make them again.

We attempt to optimise a swarm of subgoals that are all connected in one way or another to the main goal gene survival.

status
money
education
happiness


Why isn't ethics more integrated into current AI systems? I am a PhD student in computer science, and currently creating a state of the art overview in applications done in Machine Ethics (a multidisciplinary field combining philosophy and AI, that looks at creating explicit ethical programs or agents). It seems that the field mostly contains theoretical arguments and there are relatively little implementations, even though there are many people with a technical background in the field.

I understand that because ethics are involved, there is no ground truth and since it's part of philosophy one can get lost in arguing over which type of ethics should be implemented and how this can be done best. However, in computer science, it is usual to even try a simple implementation to show the possibilities or limitations of your approach.

What are the possible reasons there is so little done in explicitly implementing ethics in AI and experimenting with it?

This is necessarily a high-level answer, and highly speculative, but I've been thinking on this question, and here are my thoughts:

Implementing ethical algorithms requires a mathematical basis for philosophy because computers are difference engines

After Russell & Whitehead's famous failure, and Gödel's incompleteness theorem, this would seem to be problematic.

AI is a highly applied field, especially today per continuing validation of deep learning, and no company wants to go near the issue of ethics unless they are forced to

Thus, you see it in self-driving cars because the engineers have no choice but to grapple with the problem.  By contrast, I don't think you'll see many algorithmic stock trading firms, where the business is Pareto efficiency, worrying about the ethics or social impacts of financial speculation. (The solution to "flash crashes" seems to have been rules for temporary suspension of trading, instead of addressing the social value of high-frequency algorithmic trading.) A more obvious example is social media companies ignoring the extreme amounts of information abuse (disinformation and misinformation) being posted on their sites, pleading ignorance, which is highly suspect in that the activity generated by information abuse positively affects their bottom-lines.

Applied fields tend to be predominantly driven by profit

The primary directive of corporations is to return a profit to investors.  It's not uncommon for corporations to break the law when the fines and penalties are expected to be less than the profit made by illegal activity.  (There is the concept of ethics in business, but the culture in general seems to judge people and companies based on how much money they make, regardless of the means.)

Implementation of machine ethics is being explored in areas where they are necessary to sell the product, but elsewhere, it's still largely hypothetical

If superintelligences evolve and wipe out humanity (as some very smart people with superior mathematics skills are warning us about,) my feeling is that it will be a function of nature, where unrestricted evolution of these algorithms is due to economic drivers which focus on hyper-partisan automata in industries like financial speculation and autonomous warfare.  Essentially, chasing profits at all costs, regardless of the impacts.


Is there any research work that attempts to combine neuroevolution with deep reinforcement learning? Neuroevolution can be used to evolve a network's architecture (and weights, of course). Deep reinforcement learning, on the other hand, has been proven to be extremely powerful at optimising the network weights in order to train really well-performing agents. Can we use the following pipeline?

search for the best network topology/weights through neuroevolution
train the best candidate selected above through DQN or something similar

This seems reasonable to me, but I haven't found anything on the matter.

Is there any research work that attempts to combine neuroevolution with deep reinforcement learning? Is it feasible? What are the main challenges?

There are 3,810 articles easily found in an academic article search. These are three examples.

Neuroevolution for reinforcement learning using evolution strategies — C Igel — The 2003 Congress on Evolutionary Computation, 2003, ieeexplore.ieee.org — "We apply the CMA-ES, an evolution strategy which efficiently adapts the covariance matrix of the mutation distribution, to the optimization of the weights of neural networks for solving reinforcement learning problems. It turns out that the topology of the networks considerably ..."
Neuroevolution strategies for episodic reinforcement learning — V Heidrich-Meisner, C Igel — Journal of Algorithms, 2009, Elsevier — "Because of their convincing performance, there is a growing interest in using evolutionary algorithms for reinforcement learning. We propose learning of neural network policies by the covariance matrix adaptation evolution strategy (CMA-ES), a randomized variable-metric ..."
Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning — FP Such, V Madhavan, E Conti, J Lehman — 2017, arxiv.org — "Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement ..."

To clarify the strategy proposed, we can rewrite the approach as a set of design features. Let's not assume that topology formation is based on weights, which diminishes the concept of morphology and topology. If we were to reduce the model to a traditional artificial network of orthogonal topology, it would then not be neuroevolution; it would then be basic machine learning.

Search for the best network topology through neuroevolution
Train the best candidate selected above through Q-learning

The second item doesn't seem address the relationship between the inputs, outputs, and objectives of neuroevolution designs and those of Q-learning and other reinforcement learning strategies. Q-learning algorithms are not designed to run on feed forward networks in general and certainly not easily mapped to the topologies that may form during neuroevolution. There are probably billions, if not an infinite number, of ways to combine the two strategies, but simplistic concatenation of the two processes is not possible without further research and consideration of how they will interrelate collaboratively toward program goals.

It may be useful to search for articles, study, and then formulate your research trajectory. It is recommendable to start with learning about neuroevolution and reinforcement independently, and then start reading articles like the above three. Pour the foundation, let it dry, and then frame the house.


How to predict time series with accuracy? I am trying to predict Forex time series. The nature of the market is that 80% of the time the price can not be predicted, but in 20% of the time it can be. For example, if the price drops down very deep, there is 99% probability that there will be a recovery process, and this is what I want to predict.

So , how do I train a feed-forward network the way it would only predict those cases that have 99% of certainty to take place and, for the rest of the cases it would output "unpredictable" status?

Imagine that my data set has 24 hours of continuous price data as input (as 1 minute samples), and then as output I want the network to predict 1 hour of future price data. The only restriction I need to implement is that if the network is not "sure" that the price is predictable, it would outupt 0s. So, how do I implement safety in predictions the network is outputting?

It seems that my problem is similar to Google Compose, where it predicts the next word as you are typing , for example, if you type "thank you", it would add " very much" and this would be like 95% correct. I want the same, but it is just that my problem has too much complexity. Google uses RNNs, so maybe I should try a deep network of many layers of RNNs?

Things like this a really hot topic in research right now, and it's very difficult to get high accuracy on a chaotic system like the stock market. That being said, I would probably recommend preprocessing your data rather than having your primary neural network decide what to accept and what not to.

For example, in your specific case, you could model a bubble bursting as perhaps a negative exponential drop-off or something of the sort. This could include machine learning too. You could gather historical drops in stock market data, and use some sort of regression (Bayesian would probably work well) to estimate the best function to use as an indicator to whether a steep drop has occurred. If so, then use your neural network specifically to classify the fate of the stock. I would think you would have more success following a specialised route such as this rather than trying to train a network on general trends in the market.

In terms of the structure of your neural network, you may want to consider a convolutional neural network (CNN) instead of a recurrent neural network (RNN). RNNs assume the current point in your time-series depends on all previous points, from the beginning of your data. I wouldn't think this would hold true in general for the stock market. The filters a CNN learns are suited to learning to extract certain features and the CNN will apply the filters to specific portions of the data, in the way it considers optimal. They are both nonlinear models, but the CNN will be less computationally costly to train. You could also try a gradient-boosting regression approach instead of a neural network. That being said, something like an LSTM RNN (long short-term memory) won't necessarily be bad - just my two cents.


Why is tic-tac-toe considered a non-deterministic environment? I have been reading about deterministic and stochastic environments, when I came up with an article that states that tic-tac-toe is a non-deterministic environment.
But why is that?
An action will lead to a known state of the game and an agent has full knowledge of the board and of its enemies' past moves.

The game of TIC-TAC-TOE can be modelled as a non-deterministic Markov decision process (MDP) if, and only if:

The opponent is considered part of the environment. This is a reasonable approach when the goal is to solve playing against a specific opponent.

The opponent is using a stochastic policy. Stochastic policies are a  generalisation that include deterministic policies as a special case, so this is a reasonable default assumption.

An action will lead to a known state of the game and an agent has full knowledge of the board and of Its enemies past moves.

Whilst this is true, the next state and reward as observed by an agent may not be due to the postion it plays in (with the exception being if it wins or draws on that move), but the position after the opponent plays.
It is also possible to frame TIC-TAC-TOE as a partially observed MDP (POMDP) if you consider the opponent to not have a fixed policy, but to be reacting to play so far, perhaps even learning from past games. In which case, the internal state of the opponent is the unknown part of the state. In standard game playing engines and in games of perfect information, this is resolved by assuming the opponent will make the best possible (or rational) move, which can be determined using a search process such as minimax. When there is imperfect information, such as in poker, it becomes much harder to allow for an opponent's action.


Why do we also need to normalize the action's values on continuous action spaces? I was reading here tips & tricks for training in DRL and I noticed the following:

  always normalize your observation space when you can, i.e., when you know the boundaries
  normalize your action space and make it symmetric when continuous (cf potential issue below) A good practice is to rescale your actions to lie in [-1, 1]. This does not limit you as you can easily rescale the action inside the environment

I am working on a discrete action space but it is quite difficult to normalize my states when I don't actually know the full range for each feature (only an estimation).

How does this affect training? And more specifically, why on continuous action spaces we need to normalize also the action's values?

Notably, these two tips/tricks are useful because we are assuming the context of deep reinforcement learning here, as you pointed out. In DRL, the RL algorithm is guided in some fashion by a deep neural network, and the reasons for normalizing stem from the gradient descent algorithm and the architecture of the network.

  How does this affect training?

An observation from the observation space is often used as an input to a neural network in DRL algorithms, and normalizing the input to neural networks is beneficial for many reasons (e.g. increases convergence speed, aids computer precision, prevents divergence of parameters, allows for easier hyperparameter tuning, etc.). These are standard results in DL theory and practice, so I won't provide details here.

  And more specifically, why on continuous action spaces we need to normalize also the action's values?

Most popular discrete action space DRL algorithms (e.g. DQN) have one output node for each possible action in the neural net. The value of the output node may be a q-value (value-based algorithm) or a probability of taking that action (policy-based algorithm).

In contrast, a continuous action space DRL algorithm simply cannot have an output node for each possible action, as the action space is continuous. The output is usually the actual action to be taken by the agent or some parameters that could be used to construct the action (e.g. PPO outputs a mean and standard deviation and then an action is sampled from the corresponding Gaussian distribution - this phenomenon is mentioned in your linked reference). Therefore, normalizing the action space of a DRL algorithm is analogous to normalizing the outputs of the corresponding neural network, which is known to increase training speed and prevent divergence. Again, a quick search will yield some good resources if you are interested in these results.


In YOLO, when is $\mathbb{1}_{i j}^{\mathrm{obj}} = 1$, and what are the ground-truth labels for $x_i$ and $y_i$? I'm trying to implement a custom version of the YOLO neural network. Originally, it was described in the paper You Only Look Once: Unified, Real-Time Object Detection (2016). I have some problems understanding the loss function they used.
Basic information:

An input image is divided into $S \times S$ grid (that gives the total of $S^2$ cells) and each cell predicts $B$ bounding boxes and $c$ conditional class probabilities. Each bounding box predicts $5$ values: $x,y,w,h,C$ (center of bounding box, width and height and confidence score). This makes the output of YOLO an $S \times S \times (5B*c)$ tensor.

The $(x,y)$ coordinates are calculated relative to the bounds of the cell and $(w,h)$ is relative to the whole image.

I understand that the first term penalizes the wrong prediction of the center of a bounding box; the 2nd term penalizes wrong width and height prediction; the 3rd term the wrong confidence prediction; the 4th is responsible for pushing confidence to zero when there is no object in a cell; the last term penalizes wrong class prediction.

My problem:
I don't understand when $\mathbb{1}^\text{obj}_{ij}$ should be $1$ or $0$. In the paper, they write (section 2.2. Training):

$\mathbb{1}_{i j}^{\mathrm{obj}}$ denotes that the $j$th bounding box predictor in cell $i$ is "responsible" for that prediction.

and they also write

Note that the loss function only penalizes classification error if an object is present in that grid cell (hence the conditional class probability discussed earlier). It also only penalizes bounding box coordinate error if that predictor is "responsible" for the ground truth box

So, is it right that, for every object in the image, there should be exactly one pair of $ij$ such that $\mathbb{1}_{i j}^{\mathrm{obj}} = 1$?

If this is correct, this means that the center of the ground truth bounding box should fall into $i$th cell, right?

If this is not the case, what are other possibilities when $\mathbb{1}_{i j}^{\mathrm{obj}} = 1$, and what ground truth labels $x_i$ and $y_i$ should be in these cases?

Also, I assume that ground truth $p_i(c)$ should be $1$ if there is an object of class $c$ in the cell $i$, but what ground truth $p_i(c)$ should be equal to in case there are several objects of different classes in the cell?

The correct interpretation (based on the comment by the author of the question below):
Yep, you are right. It is actually only a single cell per object that contributes to the loss with $\mathbb{1}^\text{obj}_{ij}$ factor. That cell is identified as the one that contains the centre(oid) of the ground truth box of the corresponding object.
My original (incorrect) interpretation of the paper:
From what I read in the paper it is not a single bounding box for all cells in the grid for a particular object (your original suggestion).
Rather for every object and every cell that is assigned to this object only one bounding box contributes to the loss. The network generates $B$ bounding boxes for every cell in the grid, but we pick only one and only when that cell actually belongs to an object.
Every cell $i$, which is not a background cell, will have exactly one box $j$, such that $\mathbb{1}^\text{obj}_{ij} = 1$
This is based on the following paragraph from the paper:

Note that the loss function only penalizes classification error if an object is present in that grid cell (hence the conditional class probability discussed earlier). It also only penalizes bounding box coordinate error if that predictor is "responsible" for the ground truth box (i.e. has the highest IOU of any predictor in that grid cell).


Is the agent aware of a possible different set of actions for each state? I have a use case where the set of actions is different for different states. Is the agent aware of what actions are valid for each state, or is the agent only aware of the entire action space (in which case I guess the environment needs to discard invalid actions)?
I presume the answer is yes, but I would like to confirm.

This is actually an implementation choice, and will depend on how you chose to represent the agent's model of the function that maps from states to actions.
If you explicitly represent the entire state space, as you might chose to do with simple benchmark problems that you solve by directly solving an MDP with something like value iteration, then you can also easily explicitly represent exactly the set of actions that the agent can perform in each state, and the agent can learn the expected value of just taking those actions.
If your state space is very large, you may not be able to represent it explicitly, and your agent is more likely to use some approximation of the value function or its policy, as is commonly done in Q-Learning. Here, it is often preferable to define your model of the environment so that taking an invalid action in a state causes some well-defined outcome, or causes the agent to randomly re-select its actions until it ends up picking a valid one. The agent will eventually learn that selecting an invalid action leads to bad outcomes, without "realizing" that the action is invalid.


Taxi-v3 help. What is meant exactly by convergence of the algo, the highest reward and optimal action for every state? I started learning about Q table from this blog post Introduction to reinforcement learning and OpenAI Gym, by Justin Francis, which has a line as below -

  After so many episodes, the algorithm will converge and determine the optimal action for every state using the Q table, ensuring the highest possible reward. We now consider the environment problem solved.

The Q table was updated by Q-learning formula
Q[state,action] += alpha * (reward + np.max(Q[state2]) - Q[state,action])

I ran 100000 episodes of which I got the following -

Episode 99250 Total Reward: 9
Episode 99300 Total Reward: 7
Episode 99350 Total Reward: 6
Episode 99400 Total Reward: 14
Episode 99450 Total Reward: 10
Episode 99500 Total Reward: 10
Episode 99550 Total Reward: 9
Episode 99600 Total Reward: 14
Episode 99650 Total Reward: 5
Episode 99700 Total Reward: 7
Episode 99750 Total Reward: 3
Episode 99800 Total Reward: 5

I don't know what the highest reward is. It does not look like it has converged. Yet, the graph shows a trend in convergence but it was plotted for a larger scale.

What should be the sequence of actions to be taken when the game is reset() but the "learned" Q table is available? How do we know that and the reward in that case?

Your Q learning update expression looks correct. The Total Reward will not be the same at the end of each episode because the starting position of taxi is different in each episode so the number of steps necessary to reach the final destination will be different in each episode. The graph that you posted shows that the algorithm converged after short amount of episodes so 100000 episodes might be too much. Since the environment is simple try manually calculating optimal policy for some starting specific position and then see if the algorithm does the same sequence of actions.


What is the current state-of-the-art in unsupervised cross-lingual representation learning? What is the current state-of-the-art in unsupervised cross-lingual representation learning?

The blog post Unsupervised Cross-lingual Representation Learning (2019), the related paper and slides by Sebastian Ruder (a researcher currently at DeepMind) summarize what you are looking for. In fact, the authors write

  We will introduce researchers to state-of-the-art methods for constructing resource-light cross-lingual word representations and discuss their applicability in a broad range of downstream NLP applications, covering bilingual lexicon induction, machine translation (both neural and phrase-based), dialogue, and information retrieval tasks. We will deliver a detailed survey of the current cutting-edge methods, discuss best training and evaluation practices and usecases, and provide links to publicly available implementations, datasets, and pretrained models and word embedding collections.


Why do small datasets require more samples, while big datasets require fewer samples in negative sampling? In the deep learning specialization course by Andrew Ng, in the video Sequence Models (minute 4:13), he says that in negative sampling we have to choose a sample of words from the corpus to train rather than choosing the whole corpus. But he said that, for smaller datasets, we need a bigger number of samples, for example, 5-20, and, for larger datasets, we need a smaller sample, for example, 2-5. By sample, I am referring to the number of words along with the target word we have taken to train the model.

Why do small datasets require more samples, while big datasets require fewer samples?

He likely found this to a be a best practice to avoid over fitting, with a small data set if you only use small and easy to learn (less words -> less degrees of freedom) sequences then you open your model to the risk of over fitting that data set where as on a large data set that has alot more total information you can train on small sequences without being at risk of over fitting because although the smaller sequences will be easier to learn the variance of sequences will be much higher.


How can reinforcement learning be unsupervised learning if it uses deep learning? I was watching a video in my online course where I'm learning about A.I. I am a very beginner in it.

At one point in the course, the instructor says that reinforcement learning (RL) needs a deep learning model (NN) to perform an action. But for that, we need expected results in our model for the NN to learn how to predict the Q-values.

Nevertheless, at the beginning of the course, they said to me that RL is an unsupervised learning approach because the agent performs the action, receives the response from the environment, and finally takes the more likely action, that is, with the highest Q value.

But if I'm using deep learning in RL, for me, RL looks like a supervised learning approach. I'm a little confused about these things, could someone give me clarifications about them?

Supervised learning

The supervised learning (SL) problem is formulated as follows.

You are given a dataset $\mathcal{D} = \{(x_i, y_i)_{i=1}^N$, which is assumed to be drawn i.i.d. from an unknown joint probability distribution $p(x, y)$, where $x_i$ represents the $i$th input and $y_i$ is the corresponding label. You choose a loss function $\mathcal{L}: V \times U \rightarrow \mathbb{R}$. Then your goal is to minimize the so-called empirical risk

$$R_{\mathcal{D}}[f]=\frac{1}{N} \sum_{i=1}^N \mathcal{L}(x_i, f(x_i)) \tag{0}\label{0}$$

with respect to $f$. In other words, you want to find the $f$ that minimizes the average above, which can also be formally written as
$$
f^* = \operatorname{argmin}_f R[f] \tag{1}\label{1}
$$
The problem \ref{1} is called the empirical risk minimization because it is a proxy problem for the expected risk minimization (but you can ignore this for now).

Reinforcement learning

In reinforcement learning, you typically imagine that there's an agent that interacts, in time steps, with an environment by taking actions. At each time step $t$, the agent takes $a_t$ in the state $s_t$, receives a reward $r_t$ from the environment and the agent and the environment move to another state $s_{t+1}$.

The goal of the agent is to maximize the expected return

$$\mathbb{E}\left[ G_t \right] = \mathbb{E}\left[ \sum_{i=t+1}^\infty R_i \right]$$

where $t$ is the current time step (so we don't care about the past), $R_i$ is a random variable that represents the probable reward at time step $i$, and $G_t = \sum_{i=t+1}^\infty R_i $ is the so-called return (i.e. a sum of future rewards, in this case, starting from time step $t$), which is also a random variable.

In this context, the most important job of the programmer is to define a function $\mathcal{R}(s, a)$, the reward function, which provides the reinforcement (or reward) signal to the RL agent. $\mathcal{R}(s, a)$ will deterministically or stochastically determine the reward that the agent receives every time it takes action $a$ in the state $s$. (Note that $\mathcal{R}$ is different from $R_i$, which is a random variable that represents the reward at time step $i$).

What is the difference between SL and RL?

In RL, you (the programmer) need to define the reward function $\mathcal{R}$ and you want to maximize the expected return. On the other hand, in SL you are given (or you collect) a dataset $\mathcal{D}$, you choose $\mathcal{L}$ in \ref{0}, and the goal is to find the function $f^*$ that minimizes the empirical risk.  So, these have different settings and goals, so they are different!

However, every SL problem can be cast as an RL problem. See this answer. Similarly, in certain cases, you can formulate an RL as an SL problem. So, although the approaches are different, they are related.

Is RL an unsupervised learning approach?

In RL, you do not tell the agent what action it needs to take. You only say that the action that was taken was "bad", "good" or "so so". The agent needs to figure out which actions to take based on your feedback. In SL, you explicitly say that, for this input $x_i$, the output should be $y_i$.

Some people may consider RL is an unsupervised learning approach, but I think this is wrong, because, in RL, the programmer still needs to define the reward function, so RL isn't totally unsupervised and it's also not totally supervised. For this reason, many people consider RL an approach that sits between UL and SL.

What is deep learning?

The term/expression deep learning (DL) refers to the use of deep neural networks (i.e. neural networks with many layers, where "many" can refer to more than 1 or 1000, i.e. it depends on the context) in machine learning, either supervised, unsupervised, or reinforcement learning. So, you can apply deep learning to SL, RL and UL. So, DL is not only restricted to SL.


Can AI stop attacks like WannaCry? I have done some research regarding the application of machine learning to cyber-security. After these recent attacks (like WannaCry), I think that AI-based cyber defense can prevent them. I have also read about research regarding the same in MIT, and that AI can detect more than 80% of malware. Is AI actually so promising in this department?

There are projects out there attempting to apply Machine Learning / AI to cyber-security in different ways.  One that I'm familiar with is Apache Metron.  Another related project is Apache Spot.  I think if you read over the docs for these two projects respectively, they will probably give you some good insights on this subject.