Who manufactures Google's Tensor Processing Units? Does google manufacture TPUs? I know that google engineers are the ones responsible for the design, and that google is the one using them, but which company is responsible for the actual manufacturing of the chip?

Google has not released the manufacturing details for their TPUs. However, it's suspected that they're produced by either Taiwan Semiconductor Manufacturing or GlobalFoundries, as these are some of the largest companies in the industry.

Source


How fast is a GPU compared to a CPU for training deep learning models, and which GPU should I buy? First of all, I don't play games at all, and I am still quite new to deep learning. I was using AlexNet (transfer learning, actually) in MATLAB to classify images in my current laptop (i5-3230, without any GPU). And it was taking roughly 25-30 hours to finish training, and I don't dare using GooLeNet.
I have to buy a laptop. How fast will be if I use GPU?.
And I am not that rich to buy 2080 or 1080ti that easily. Actually, I am poor and I have struggled financially, if the price exceeds $1200. I have the intention to do research on signal and medical image processing using deep learning in the future.
Will it be enough for me to buy a laptop with GTX 1060 6GB, or is it essential/overkill to buy one with RTX2060?
Feel free to give any other option that could fill my requirements.

I think you should redirect your focus.
Learn and play with ML, and only when compute becomes the main bottleneck for your learning, invest in hardware.

I've recently engaged into a ML project for which I 've assembled a machine with gtx1080, installed gui-less ubuntu, configured ssh, drivers etc, and than spend 3 months on data collection. And the dataset was so small (30k images), that I could probably learn mobilenet v3 on my CPU instead.

I think that in most cases focusing on hardware is a form of procrastination when it comes to machine learning.
Also, a laptop with really strong GPU makes little sense (unless it's really well cooled like Alienware r3 that I use at work, but it's to heavy to be really considered a laptop...). It will throttle, It will be less durable due to constant overheating when you learn models.

So I'd say, go for gtx1060 laptop and only when you really need compute (you think you have a chance to actually win a competition due to a great idea you have, or you have a client that has seen a demo and really want's to pay for your model if you improve it), invest into standalone PC or cloud to tune your prototype model.


Training Multi class classification (One-vs-all) on Neural network I am very new to Machine learning and following the course offered by Andrew Ng. I am very confused How we train our neural network on Multi-class Classification(suppose take K classes). For K classes we will be training K different neural networks.

But Do we train one neural network at a time for all features or we train all K NN at a time for one feature? please explain the complete procedures.

Please help me with this. Consider that I have a very very basic understanding of neural networks in multi-class classification.

Let us suppose that you are training a neural network for classfing images of vehicles , then the input vector , image of the "vehicle" will be a 2D array of pixels. This undergoes several transformations at each layer of the neural network , the last layer of the neural network produces another vector whose dimensions are lesser than the original image vector.

So the network is mapping images to some vectors in a high dimensional space. In order to classify the images it is now sufficient to classify the vectors obtained from the network of their corresponding images, you  can do this with a simple "linear" classifier using a softmax layer.

So all the layers of the network except the last layer are transforming the image representation to a "vector". This vector is classified by a linear softmax classifier by the last layer of neural network.


What is the best activation function for the embedding layer in a deep auto-encoder? I am designing a deep autoencoder for graph embedding (exactly node embedding) following this paper SDNE. In the original paper, they used the sigmoid activation for all hidden layers in the autoencoder model, even for the embedding layer.
However, I think the embedding layer should use the tanh activation and the reconstruction layer should be used ReLU activation. Because, embedding is in the range $[-1, 1]$ and reconstruction layer is in the range $[0, x]$, which generates better results due to a larger range for representation and directed graph. Instead of in the range $[0,1]$ from sigmoid will lead to a lack of embedding information.
So, what is the best activation function for deep autoencoders to capture good information about the structure of graph?

It is hard to tell what exactly is better because these are hyperparameters. However, the sigmoid activation function is closer to biological neurons.
In the paper below, Bengio demonstrates why ReLU activation functions are better for hidden layers. In summary, they increase the sparsity of calculations (matrix in each layer shod multiply to its relative weights), and because of that, it made it possible for data to classify faster and easier.
http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf


What is the difference between a stacked restricted Boltzmann machine and deep belief network? What is the difference between stacked restricted Boltzmann machine (RBM) and deep belief network (DBN)?

Are they the same entity? If so, why? Is the latter some specific type of the former? If so, how to tell if stacked RBM is a DBN?

For what it's worth, Wikipedia says that

  deep belief networks can be formed by "stacking" RBMs.

Hinton writes in Scholarpedia:

  A deep belief net can be viewed as a composition of simple learning modules each of which is a restricted type of Boltzmann machine.

So, a deep belief network is definitely a stacked RBM.

I have never heard of different stacked RBMs, but it is easy to imagine something like convolutional stacked RBMs, where some RBMs are used as filters that slide over the input data or something. Whether that would still be called a deep belief network, is probably up to the guy who publishes it first.


Is it necessary to know the details behind the AI algorithms and models? I am interested in the field of artificial intelligence. I began by learning the various machine learning algorithms. The maths behind some were quite hard. For example, back-propagation in convolutional neural networks.

Then when getting to the implementation part, I learnt about TensorFlow, Keras, PyTorch, etc. If these provide much faster and more robust results,  will there be a necessity to code a neural network (say) from scratch using the knowledge of the maths behind back-prop, activation functions, dimensions of layers, etc., or is the role of a data scientist only to tune the hyper-parameters?

Further, as of now the field of AI does not seem to have any way to solve for these hyperparameters, and they are arrived at through trial and error. Which begs the question, can a person with just basic intuition about what the algorithms do be able to make a model just as good as a person who knows the detailed mathematics of these algorithms?

This is a good question. I tend to think the answer is yes it is necessary to know the details, because a person without mathematical understanding of these algorithms cannot consistently make a model as good as someone who does have that understanding.

The reason is right at the core of computer science: abstractions are useful, but usually obscure details. When those details matter, someone who only knows the abstraction and not the details that lie beneath can't understand what's going on.

As an example, if you don't understand the math behind optimizing the weights of a neural network, it might not be apparent how parameters like the learning rate are impacted by properties like network depth when some of the inputs have not been properly normalized. If you understand the optimization process mathematically, you can reason through the effects even if you are trying to work on an unfamiliar problem. This ability to reason through the probable effects of parameter decisions in new domains is the main thing that you miss by working from intuition.


Can the normalization factor for the belief state update be zero? In order to update the belief state in a POMDP, the following formula is used:
$$b'(s')=\frac{O(a, s', z) \sum_{s\in S} b(s)T(s, a, s')}{\mathbb{P}(z \mid b, a)}$$
where

$s$ is a specific state in the set of states $S$
$b'(s')$ is the updated belief state of being in the next state $s'$
$T(s, a, s') = \mathbb{P}(s' \mid s, a)$ is the propability (function) of having been in $s$ and ending up in $s'$ by taking action $a$;
$O(a, s', z) = \mathbb{P}(z \mid s', a)$ the probability (function) of observing $z$, performing action $a$ and ending up in $s'$
$\mathbb{P}(z \mid b, a)$ is defined as follows $\sum_{s \in S}b(s)\sum_{s' \in S} T(s, a, s')O(a, s', z)$

Looking at $\mathbb{P}(z \mid b, a)$ it is possible that the result is $0$. This would be the case if the agent is in a state where no further actions are possible. But, in that case, there is a problem with updating $b'(s')$, since this causes a zero division. Is this a common problem and is the only possibility to avoid that a programming solution like an if-statement? Or is $\mathbb{P}(z \mid b, a)$ always non-zero?

I think that the normalisation factor is assumed to be non-zero. So, in practice, I guess, you must eventually check that $P(z \mid b, a)$ is non-zero (even though, I guess, it will likely never be zero because of round-off errors in computers).

The formula to calculate $b'(s')$ comes from its definition, which is based on Bayes' theorem, where the denominator is assumed to be non-zero (in general).

The definition of $b'(s')$ is $P(s' \mid z, a, b)$, that is, the new belief $b'$ of being in state $s'$ is defined as the probability of landing in the next state $s'$, given that we have observed $z$, have taken action $a$ from the previous state $s$ and we had the previous belief $b$. We will expand this definition, but first let us recall a few probability definitions.

Recall that $P(A, B) = P(A \mid B) P(B) = P(B \mid A) P(A)$, where $A$ and $B$ can actually be multiple events (that is, $A$ could actually be the intersection of multiple events). In other words, suppose we want to calculate $P(A, B, C)$, we can actually consider e.g. $B$ and $C$ as one event. Let $(B \cap C) = (A, B) = D$ (note that the notation $(A, B)$ means the "intersection" of events $A$ and $B$, in the case $A$ and $B$ are events). Then

\begin{align}
P(A, B, C)
& = P(A, (B, C)) \\
&= P(A, D) \\
&= P(A|D)P(D) \\
&= P(A|B, C)P(B, C) \\
&= P(A|B, C)P(B|C)P(C) \\
\end{align}

In general, this idea generalises to more variables/events.

Note also that $\frac{P(A, B)}{P(B)} = P(A \mid B)$.

At this point, we are prepared to expand $P(s' \mid z, a, b)$ and understand its expansion.

We can expand $P(s' \mid z, a, b)$ as follows

\begin{align}
P(s' \mid z, a, b)
&= \frac{P(s', z, a, b)}{P(z, a, b)}\\[0.7em]
&= \frac{P(s', z, a, b)}{P(z \mid b, a)P(a|b)P(b)}\\[0.7em]
&= \frac{P(z \mid s', a, b) P(s' \mid a, b) P(a | b) P(b)}{P(z \mid b, a)P(a|b)P(b)} \\[0.7em]
&= \frac{P(z \mid s', a, b) P(s' \mid a, b)}{P(z \mid b, a)}
\end{align}

It then turns out that (I will maybe explain this more in detail later)

\begin{align}
P(s' \mid z, a, b)
&= \frac{P(z \mid s', a, b) P(s' \mid a, b)}{P(z \mid b, a)} \\[0.7em]
&=\frac{O(a, s', z) \sum_{s\in S} b(s)T(s, a, s')}{\sum_{s \in S}b(s)\sum_{s' \in S} T(s, a, s')O(a, s', z)} \\[0.7em]
&=b'(s')
\end{align}

So, by assumption, $P(z \mid b, a) = \sum_{s \in S}b(s)\sum_{s' \in S} T(s, a, s')O(a, s', z)$ must be different from zero for the equality above to hold.

You can see this from a very simply example of the Bayes' theorem. Let $P(A \mid B) = \frac{P(B \mid A) P(A)}{P(B)}$. Now, intuitively, $P(A \mid B)$ (which is what we want to calculate using Bayes' theorem) means the probability of $A$ occurring given that $B$ has occurred, which means that $B$ couldn't have had a probability of $0$ of happening if we wanted to calculate $P(A \mid B)$, so $P(B)$ couldn't have been zero if we wanted to calculate $P(A \mid B)$ using Bayes' theorem. We can also apply this reasoning to the definition of $b'(s')$ above.

For completeness, note also that, in the normalisation factor $$\sum_{s \in S}b(s)\sum_{s' \in S} T(s, a, s')O(a, s', z),$$ $b(s)$, $T(s, a, s')$ and $O(a, s', z)$ are probability distributions, which means that not all terms of $b(s)$, $T(s, a, s')$ and $O(a, s', z)$ can be zero, for all $s$, $s'$ and $a$ (given they must sum up to $1$).

Note also that $\sum_{s' \in S} T(s, a, s')O(a, s', z)$ is a convex combination of all $O(a, s', z)$ (for all $s'$), where the coefficients are $T(s, a, s')$. The normalisation factor is also a convex combination where the coefficients are $b(s)$ (for all $s$).


What is the difference between policy and action in reinforcement learning? I'm confused with the two terminology - action and policy - in Reinforcement Learning. As far as I know, the action is:

  It is what the agent makes in a given state.

However, the book I'm reading now (Hands-On Reinforcement Learning with Python) writes the following to explain policy:

  we defined the entity that tells us what to do in every state as policy.

Now, I feel that the policy is the same as the action. So what is the difference between the two, and how can I use them apart correctly?

A policy is a function that maps states to a probability distribution over all possible actions.

So, in a typical Atari game, there might just be a handful of actions, represented by the keys that are used to play the game. In this context, the policy of a reinforcement learner might be represented by a pretty complex neural network that gets pixels as input and gives action probabilities as output.


How could a NN be trained to output a cyclic (e.g. hue) number? I was thinking about training a neural network to colourize images. The input would be the luminosity/value for each pixel, and the output would be a hue and/or saturation. Training data would be easily obtained just by selecting the luminosity/value channel from a full colour image.

Suppose all channels are scaled to 0.0-1.0, there is a problem with pixels whose hue is nearly 0.0 or nearly 1.0.

The input data may have sharp discontinuities in hue which are not visible to the human eye. This is an unstable, illusory boundary which seems like it would destablilize the training.
Also if the network outputs a value of 1.001 instead of 0.001 then this should NOT be penalised since.

Possible workarounds might be to preprocess the image to remap e.g. 0.99 to -0.01 if that pixel is near a region dominated by near-0 hues, or similarly to to remap e.g. 0.01 to 1.01 if that pixel is near a region dominated by near-1 hues. This has its own problems. Similarly, outputs could be wrapped to the range 0-1 before being scored.

But is there a better way to encode cyclic values such as hue so that they will naturally be continuous?

One solution I thought of would be to treat (hue,saturation) as a (theta,r) polar coordinate and translate this to Cartesian (x,y) and have that be the training target, but I don't know how this change of colour space will affect things (it might be find, I haven't tried it yet).

Are there alternative colour representations which are better suited to machine learning?

Your solution is pretty much on spot. It corresponds to the YUV scheme used in television and designed to match human perception characteristics. As you already noticed, such an encoding wouldn't suffer from discontinuities.


Perfect play in information incomplete games As titled, is there such thing as perfect play (or at least "perfectly optimal") in a game with incomplete information? Or at least a proof as to show why there cannot?

Naively (and seemingly obviously), the answer would be a resounding no, since the agent would be likely be forced to pick between "lottery events".

But in practice (using competitive video games as an analogy), we'd see that players would stick to a meta-game that is well equipped to defend against a majority of events that might happen, given incomplete information. Of course the response to that would be that there probably exists a "hard-counter" for any given meta-game, but if it is indeed the case that the meta-game is the "most-optimal" it probably is the case also that such a hard counter puts the player in an unfavourable position most of the time, thus the "hard-counter" itself is not optimal. Thus we'd likely see that any given first encounter players would still stick to their "optimal meta-game" rather than a hard counter of their optimal play.

A more rigour analogy would be to ask: "Under Hofstadter's notion of superrationality, how would agents play information incomplete games", but I couldn't find any readings on trying to import the notion of super-rationality into information incomplete games.

Alternatively: is there such thing as a "perfectly optimal meta-game"?

This may be an evolving answer, because the question is, in some sense, a (useful) rabbit hole. I apologize if I don't go deeply into meta-games per se, as it's a little outside of my scope, which is non-chance games of perfect information, but I think it's worthwhile to think about the underlying problem of indeterminacy in relation to games in general.

Bounded Rationality* is a useful concept because it pre-supposes a condition of computational intractability. Computational intractability can be introduced into games in several forms:

Complexity
Hidden Information
Randomness ("quantum" indeterminacy)

[For more details on my use of "quantum" in regards to randomness, see Deterministic Games.]

The underlying purpose of game theory is to determine "optimal" strategies for any given problem.  I put optimal in quotes because optimality is a spectrum, and subjective in a condition of computational intractability.

Thus, we cannot know if AlphaGo plays optimally, only that it played more optimally than Lee Sedol in 4 out of 5 games.

This is distinct from strongly solved games such as tic-tac-toe, where we can know with total certainty that a choice is optimal, because the problem of tic-tac-toe is computationally tractable.

Part of the confusion may be semantic, because the concepts are subtle and profound, and require language, what TS Eliot might have called "the intolerable wrestle with words and meanings." (For instance, I used hidden information above to avoid having to distinguish between incomplete and imperfect information.)

Perfect Play is generally defined as a strategy that leads to the best possible outcome for a participant, regardless of the choices of the opponent.

Thus minimax is of central importance, and provided the foundation for game theory.

Even in games with incomplete information, whether "deterministic" (Battleship) or involving "quantum indeterminacy" (Prisoner's Dilemma), there are optimal strategies. For simultaneous games such as Dilemma and all of the numerous extensions minimax is used.  In Battleship, there are at least three strategies of increasing optimality, and although there doesn't appear to be a strategy that can yield P > .5, if one player employs a more optimal strategy, they will win in aggregate. Even Rock, Paper, Scissors seems to have an optimal strategy, which blows my mind, and carries the caveat that I need to look into it more.

Thus, perfect play, as defined, is certainly achievable, but does not necessarily connote (objectively) optimal choices, which is a little confusing, because "perfect" implies objectivity, a condition which is only possible in regard to tractable problems.

It is also important to note that there may not be a "winning" strategy in the sense of being better off than the opponent, and in this condition, perfect or optimal play is mitigation of loss.

*In terms of incomplete information games specifically, I think there's a case for extending the concept of Bounded Rationality is extended to include information that cannot be observed or "known".

Colloquially, this would include the "unknowns" (both known and unknown) and the "unknowable" (quantum indeterminacy and superpositions).


Is there any research on the application of AI for drug design? Is there any research on the application of AI for drug design?
For example, you could train a deep learning model about current compounds, substances, structures, and their products and chemical reactions from the existing dataset (basically what produces what). Then you give the task to find how to create a gold or silver from the group of available substances. Then the algorithm will find the chemical reactions (successfully predicting a new one that wasn't in the dataset) and gives the results. Maybe the gold is not a good example, but the practical scenario would be the creation/design of drugs that are cheaper to create by using much simpler processes or synthesizing some substances for the first time for drug industries.
Was there any successful research attempting to achieve that using deep learning algorithms?

Yes, many people have worked on this sort of thing, due to its obvious industrial applications (most of the ones I'm familiar with are in the pharmaceutical industry). Here's a paper from 2013 that claims good results; following the trail of papers that cited it will likely give you more recent work.


Why is the access to the dynamics model unrealistic in Q-Learning? Pieter Abbeel says that having access to the dynamics model, $P(s' \mid s,a)$, is unrealistic because it assumes we know the probability that we will reach all future states.

I don't understand how this is unreasonable? Could someone explain this to me like in a simple way?

It is an unreasonable assumption for many scenarios to know the full transition model of an environment.

In the deterministic toy scenarios used for explaining RL, such as a grid world, it seems like a trivial requirement. However, it starts to get harder very quickly as problems and state features get more complex.

Even in relatively simple environments that are possible to model in this way, the calculations start to become difficult enough that there is an argument to not bother. For instance, a common toy example is the game of Blackjack - in order to calculate the probability of different rewards after sticking you would need to resolve the remaining game tree of typically 0,1,2, but up to more, for the dealer, accounting for their starting card, and perhaps accounting for which cards have been drawn so far (if you are simulating this more realistically, without replacement).

However, that could be resolved if we wanted. To make things harder, you need to consider agents that are set to solve problems where we don't have any meaningful probabilistic model of how the state will progress.

A good example would be any real world agent that used a vision system as part of its state. Consider a flying drone set with the task of locating and tracking migrating whales over some area of the ocean. It is rewarded for capturing footage of whales. Its state is the current image plus GPS and telemetrics data.

Knowing $P(s' \mid s,a)$ in the whale-tracking example implies that you would know the probability distribution of the next video frame, somehow accounting for the chaotic movement of waves, the likely behaviour of any creatures in shot, and effects of wind turbulence on drone position etc. Even the most sophisticated physics simulations with ray-tracing rendering engines cannot rise to that challenge with much accuracy, and the computation required even for a rough render of expected image(s) based on a physics engine would be far beyond anything you could run in real time on the drone. However, it is still possible to build an agent which learns how to solve this kind of environment through gathering statistics on its task using images as state data.

Scaling that challenge back a bit, you can view many of the Atari game-playing challenges like this. The input state is a collection of still frames from the game. Knowing $P(s' \mid s,a)$ implies knowing the probability distribution of next set of frame images, given the current inputs. This is a near impossible challenge, even if you know the code for the game, providing an analytical result for $P(s' \mid s,a)$ is unrealistic - the best way to do it would be to have a memory clone of the game that you could use to look ahead then reset. Whilst this is theoretically possible in a computer game, it is not something you can consider when interacting with the real world.


What is the difference between Q-learning, Deep Q-learning and Deep Q-network? Q-learning uses a table to store all state-action pairs. Q-learning is a model-free RL algorithm, so how could there be the one called Deep Q-learning, as deep means using DNN; or maybe the state-action table (Q-table) is still there but the DNN is only for input reception (e.g. turning images into vectors)?
Deep Q-network seems to be only the DNN part of the Deep Q-learning program, and Q-network seems the short for Deep Q-network.
Q-learning, Deep Q-learning, and Deep Q-network, what are the differences? May be there a comparison table between these 3 terms?

Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN).

Tabular Q-learning (TQL)
Deep Q-learning (DQL)
Deep Q-network (DQN)

Is it an RL algorithm?
Yes
Yes
No (unless you use DQN to refer to DQL, which is done often!)

Does it use neural networks?
No. It uses a table.
Yes
No. DQN is the neural network.

Is it a model?
No
No
Yes (but usually not in the RL sense)

Can it deal with continuous state spaces?
No (unless you discretize them)
Yes
Yes (in the sense that it can get real-valued inputs for the states)

Can it deal with continuous action spaces?
Yes (but maybe not a good idea)
Yes (but maybe not a good idea)
Yes (but only the sense that it can produce real-valued outputs for actions).

Does it converge?
Yes
Not necessarily
Not necessarily

Is it an online learning algorithm?
Yes
No, if you use experience replay
No, but it can be used in an online learning setting


Why don't people always use TensorFlow Lite, if it doesn't decrease the accuracy of the models? I have been exploring edge computation for AI, and I came across multiple libraries or frameworks, which can help to convert the model into a lite format, which is suitable for edge devices.

TensorFlow Lite will help us to convert the TensorFlow model into TensorFlow lite.
OpenVino will optimise the model for edge devices.

Questions

If we have a library to optimise the model for edge devices (e.g. TensorFlow Lite), after conversion, could it make the accuracy decrease?

If not, then why do people prefer don't always use e.g. TensorFlow Lite?

This partly answer to question 1.
There is no general rule concerning accuracy or size of the model. It depends on the training data and the processed data.
The lightest is your model compared to the full accuracy model the less accurate it will be. I would run the lite model on test data and compare to the accuracy of the full model to get an exact measure of the difference.

Tensor flow has different options to save the "lite" model (optimized in size, latency, none and default).

The following mostly answer question 2.

Tensor flow lite is intented to provide the ability to use the model to on line predict only and load the model not to train the model.

On the other hand Tensor flow is used to build (train) the model off line.

If your edge platform support any of the binding language provided for TensorFlow (javascript, java/kotlin, C++, python) you can use Tensorflow for prediction. The accuracy or speed options you might have selected to create the model will not be affected whether you use Tensor Flow or Tensor Flow Lite. Typically Tensor flow lite can be used on mobile devices (Ios, Android). There are other supported target, see this link


If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy? I wrote a simple implementation of Flappy Bird in Python, and now I'm trying to train an agent to play it at a reasonable skill level using TFLearn.

I feed the network an input vector of size 4:

the horizontal distance to the next obstacle
the agent's vertical distance from the ground-
the agent's vertical distances from the top, and
the agent's vertical distances from the bottom parts of the opening in the obstacle.

The output layer of the network contains one unit, telling me the Q value of the provided state with the assumption that the action taken in that state will be determined by the policy.

However, I don't know what policy would make the agent learn to play the best.  I can't just make it choose random actions because that would make the policy non-stationary.  What can I do?

The output layer of the network contains one unit, telling me the Q value of the provided state with the assumption that the action taken in that state will be determined by the policy.

Typically in Reinforcement Learning, the symbol $Q$ is used when you calculate an action value, and if you are evaluating for a specific policy, it is noted $q_{\pi}(s,a)$ where $\pi$ is the policy, $s$ is the current state, and $a$ the action to be taken.

What you appear to be calculating with your network is not the action value, but the state value, $v_{\pi}(s)$. Note that $v_{\pi}(s) = q_{\pi}(s,\pi(s))$ when you have a deterministic policy, and there is a similar relationship for stochastic policies.

With your setup, it would be possible to learn the value of any fixed policy that you provided as input. It is also possible to drive policy improvements, with some caveats, but that would not be Q learning. The rest of this answer assumes you want to implement Q learning, and probably something like Deep Q Networks (DQN).

  However, I don't know what policy would make the agent learn to play the best. I can't just make it choose random actions because that would make the policy non-stationary.

Actually that would make the policy stochastic. Non-stationary means that the policy would change over time. Choosing random actions is a perfectly valid stationary policy (provided the probabilities in each state remain the same).

In addition, in an optimal control scenario you actually will have and want a non-stationary policy. The goal is to start with some poor guess at a policy, like completely random behaviour, and improve it based on experience. The policy is going to change over time. That makes learning Q values in RL a non-stationary problem.

Typically you really do start control problems with a random policy. At least in safe environments such as games and simulations.

  I'm trying to train an agent to play it at a reasonable skill level using TFLearn.  What can I do?

First, modify your network so that it estimates action values $Q(s,a)$. There are two ways to do that:

Take the action $a$ as an input, e.g. one hot encoded, concatenated with the state $s$ to make the complete inputs to the neural network. This is simplest approach conceptually.
Assign a number in range $[0,N_{actions})$ to each action. Change the network to output a value for each possible action, so that your estimate $\hat{q}(s,a)$ is obtained by looking at output indexed by $a$. This is a common choice because it is more efficient for selecting best actions to drive the policy later.

Using action values for $Q(s,a)$, not state values $V(s)$, is an important part of Q learning. Technically it is possible to use state values if you have a model of the environment, and you could add one here. But it would not really be Q learning in that case, but some other related version of Temporal Difference (TD) learning.

Once you start working with action values, then you have a way to determine a policy. Your best estimate of the optimal policy is to take the action in state $a$ with the highest action value. This is:

$$\pi(s) = \text{argmax}_a \hat{q}(s,a)$$

So run your neural network for the current state and all actions. Using the first form of NN above this means run a minibatch, using the second form you run it once which is more efficient (but constructing the training data later is more complex).

If you always take the best action, this is called acting greedily. Usually in Q learning, you want the agent to explore other possibilities instead of always acting the same way - because you need to know whether changing a policy would be better. A very common approach in DQN is to act $\epsilon$-greedily, which means take this greedy action by default, but with some probability $\epsilon$ take a completely random action. Usually $\epsilon$ starts high, often at $1$, and then is decayed relatively quickly down to some lower value e.g. $0.1$ or $0.01$, where it stays during learning. You set $\epsilon = 0$ for fully greedy behaviour when evaluating or using the policy in production.

So that has set your policy, as per your question. There is still more to implement in DQN, summarised below:

Unfortunately you cannot train the agent directly online on each step and expect good results in practice. DQN requires experience replay as the source of training data. For each step of experience, should store data for start state, action, immediate reward, end state and whether this was the last step in an episode ($s,a,r,s',done$)
To train the network, improve the Q estimates and thus improve the policy, then after each step you use the Bellman relation for optimal policy as a target: $q^{*}(s,a) = r + \gamma \text{max}_{a'} q^{*}(s',a')$:

Construct a minibatch of training data from the experience replay table, using your NN to calculate the TD target for each one $\hat{g} = r + \gamma \text{max}_{a'} \hat{q}(s',a')$
Train your NN towards $\hat{q}(s,a) \rightarrow \hat{g}$ on this minibatch.

It is usual to use a separate NN to calculate the TD target, and every so many steps (e.g. 1000 steps) make this "target network" a clone of the current learning network. It helps with stability, but you may not need that for a simple environment like Flappy Bird.


In what ways can connectionist A.I. be integrated with GOFAI? In what ways can connectionist artificial intelligence (neural networks) be integrated with Good Old-Fashioned A.I. (GOFAI)? For instance, how could deep neural networks be integrated with knowledge bases or logical inference? One such example seems to be the OpenCog + Destin integration.

A neural net with even a single hidden layer is capable of Universal function approximation - it can approximate any continuous function 'as closely as you like'.

Hence, one option would be to look for GOFAI applications that would benefit from this property - for example, in state-space search approaches where the utility of a state is not readily defined in advance, and could instead be learned.


What is meant by the rank of the scoring function here? I've been reading this paper on Knowledge Graph Reasoning for Explainable Recommendation lately, and I don't understand a particular section:

Specifically, the scoring function $f((r,e)|u)$ maps any edge $(r,e)$ to a real-valued score conditioned on user $u$. Then, the user-conditional pruned action space of state $s_t$ denoted by $A_t(u)$ is defined as:
$A_t(u) = \{(r,e)| rank(f((r,e)|u))) \leq \alpha ,(r,e) \in A_t\} $
where $\alpha$ is a predefined integer that upper bounds the size of the action space.

Details about the scoring function can be found in the attached paper - what I don't understand is - What does rank mean, here? Is the thing inside of it a matrix?
It would be great if someone could explain the expression for the user conditional pruned action space in greater detail.
References:

Reinforcement Knowledge Graph Reasoning for Explainable
Recommendation [Yikun Xian, Zuohui Fu, S. Muthukrishnan, Gerard
de Melo, Yongfeng Zhang]

$\text{rank}(f((r,e)|u))$ in $A_t(u)$ means to compute the value of scoring function $f$ for all pairs $(r,e)\in A_t$ which are conditioned by $u$, then sort them in a descending order. The rank of the $f((r,e)|u)$ in this order is equal to $\text{rank}(f((r,e)|u))$. Hence $\text{rank}(f((r,e)|u)) \leqslant \alpha$ means to select the $\alpha$ top most scored pairs.


Why is the update in-place faster than the out-of-place one in dynamic programming? In Barto and Sutton's book, it's written that we have two types of updates in dynamic programming

Update out-of-place
Update in-place

The update in-place is the faster one. Why is that the case?
This is the pseudocode that I used to test it.
if in_place:
    state_values = new_state_values
else:
    state_values = new_state_values.copy()
old_state_values = state_values.copy()

for i in range(WORLD_SIZE):
    for j in range(WORLD_SIZE):
        value = 0
        for action in ACTIONS:
            (next_i, next_j), reward = step([i, j], action)
            value += ACTION_PROB * (reward + discount * state_values[next_i, next_j])
        new_state_values[i, j] = value

max_delta_value = abs(old_state_values - new_state_values).max()
if max_delta_value < 1e-4:
    break

Why is the in-place version faster, and what is the difference? What I think is that it is only better for storage usage, I don't understand the speed increase part.

When you make updates in-place, then some of the entries in state_values[next_i, next_j] that you are referencing here
value += ACTION_PROB * (reward + discount * state_values[next_i, next_j])

will already be updated earlier in the same loop. Which means you get to use the latest and likely more accurate values earlier.
The strength of this effect varies depending on the order that states and actions are visited. If you can manage to visit them in reverse order that they would appear in natural trajectories, then the speed improvement will be very noticeable.


Why does the implementation of REINFORCE algorithm minimize the gradient term but not the loss? I read the book "Foundation of Deep Reinforcement Learning,  Laura Graesser and Wah Loon Keng", and when I go through the REINFORCE algorithm, they show the objective function:
$$
J\left(\pi_{\theta}\right)=\mathbb{E}_{\tau \sim \pi_{\theta}}[R(\tau)]=\mathbb{E}_{\tau \sim \pi_{\theta}}\left[\sum_{t=0}^{T} \gamma^{t} r_{t}\right]
$$
and the gradient of the objective:
$$
\nabla_{\theta} J\left(\pi_{\theta}\right)=\mathbb{E}_{\tau \sim \pi_{\theta}}\left[\sum_{t=0}^{T} R_{t}(\tau) \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right)\right]
$$
But when they implement it,
class Pi(nn.Module):
    def __init__(self, in_dim, out_dim):
        super(Pi, self).__init__()
        layers = [
                nn.Linear(in_dim, 64),
                nn.ReLU(),
                nn.Linear(64, out_dim)
        ]
        self.model = nn.Sequential(*layers)
        self.onpolicy_reset()
        self.train()

    def onpolicy_reset(self):
        self.log_probs = []
        self.rewards = []

    def forward(self, x):
        pdparam = self.model(x)
        return pdparam

    def act(self, state):
        x = torch.from_numpy(state.astype(np.float32))
        pdparam = self.forward(x) # (1, num_action), each number represent the raw logits for that specific action
        # model contain the paremeters theta of the policy, pd is the probability
        # distribution parameterized by model's theta
        pd = Categorical(logits = pdparam)
        action = pd.sample()
        log_prob = pd.log_prob(action)
        self.log_probs.append(log_prob)
        return action.item()

def train(pi, optimizer):
    T = len(pi.rewards)
    rets = np.empty(T, dtype = np.float32)
    future_ret = 0.0
    for t in reversed(range(T)):
        future_ret = pi.rewards[t] + gamma*future_ret
        rets[t] = future_ret

    rets = torch.tensor(rets)
    log_probs = torch.stack(pi.log_probs)
    loss = -log_probs*rets
    loss = torch.sum(loss)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    return loss

def main():
    env = gym.make('CartPole-v0')
    # in_dim is the state dimension
    in_dim = env.observation_space.shape[0]
    # out_dim is the action dimension
    out_dim = env.action_space.n
    pi = Pi(in_dim, out_dim)
    optimizer = optim.Adam(pi.parameters(), lr = 0.005)
    for epi in range(300):
        state = env.reset()
        for t in range(200): # max timstep of cartpole is 200
            action = pi.act(state)
            state, reward, done, _ = env.step(action)
            pi.rewards.append(reward)
        # env.render(mode='rgb_array')
            if done:
                break
        loss = train(pi, optimizer)
        total_reward = sum(pi.rewards)
        solved = total_reward > 195.0
        pi.onpolicy_reset()
        print(f'Episode {epi}, loss: {loss}, total reward: {total_reward}, solve: {solved}')
    return pi

In train(), they minimize the gradient term, and I can not understand why is that.
Can someone shed light on that?
I am new to this so please forget me if this question is stupid.

They are not maximizing the gradient, the gradient is of the form
\begin{equation}
\nabla_{\theta} J \approx \sum_{t=0}^T G_t \nabla_{\theta} \log(\pi_{\theta}(a_t|s_t))
\end{equation}
that means that when implementing it in software you can form your objective as
\begin{equation}
J = \sum_{t=0}^T G_t \log(\pi_{\theta}(a_t|s_t))
\end{equation}
and then taking the gradient of that objective is equal to the policy gradient.


Selecting a library for recognition You need a library to recognize the machine numbers.

There is a choice between two libraries Keras and OpenCV

Which is better to choose? Or is there an even better solution?

Programming language Python3

This two-part article by Trung Tran about Real Time Object Recognition demonstrates differences between Keras and OpenCV from a coding point of view is a reasonable treatment.  It also covers implementing pre-trained VGG16 models.

The application in this Campus Hippo lesson uses both Keras and OpenCV to simulate a self-driving car.

This tutorial by Adrian Rosebrockis about installing Keras for deep learning contains some good installation information to get both Keras and OpenCV running in the same virtual environment, which may be useful, especially for initial experimentation.

Being a user of Python3, I never limit my options to Python3 libraries.  Python is used as a scripting language in place of Bash scripts in operating systems today because it can call Javac, C or C++ executables, or any other library or executable and pass values, arrays, or matrices back and forth with ease and simplicity.

Regarding choice, you may wish to define what better means in your context.  If you wished to quantify or at least qualitatively describe acceptance or comparison criteria, there are a few typical dimensions to consider in your selection process.

Time to the provision of an answer
Accuracy (error margin, which may not apply in this case)
Reliability (pass/fail)
Minimum required preparation of images to achieve accuracy or reliability
Maintainability of the software that calls the library or framework
Current active participation in maintaining the library or framework
Projected future active maintenance based on trend of use and development

Tangential Comment:  For clarity, you may wish to specify more information about the machine numbers in the question.  We can only guess whether you mean some numbers rasterized using some font and kerning rules.


Does optimal policy implies deterministic? Let M be an MDP with two states, A, B, A is the starting state and
you always transit to the final state B using two possible actions.
$A1$ gives you rewards which are normally distributed N(0, 1) and $A2$
gives you rewards that are normally distributed N(0, 3). How many
optimal policies do you have? What is the optimal value of the
states? Is any policy preferred over the other? Why? If you prefer
one over the other is there some way to detect it using what we have
studied?
According to me, there are infinite policies that give the same expected reward.
$\pi_\alpha$ be a policy that is stochastic which maps as follows - $\pi_\alpha(s, A1) = \alpha $ and $ \pi_\alpha (s, A2) = 1 - \alpha$ for $ \alpha \in [0,1]$
It is clear that for each $\alpha$, we get infinite policies but have the same expected return.
But, according to some google searches (link below), I found Optimal policies are always deterministic. Hence this implies, there are only 2 policies i.e., either take action $A1$ or $A2$ but not probabilistic.
https://ai.stackexchange.com/questions/10591/is-the-optimal-policy-always-stochastic-if-the-environment-is-also-stochastic#:~:text=No.,enough%20to%20disambiguate%20between%20locations. (Here it says optimal policies are generally deterministic)
So, my doubt is what are the optimal policies here? Is it deterministic (only 2 policies) or stochastic (infinite)? or is it an assumption that optimal policies are deterministic?

I think the result you are referring to is the one that says that there always exists a deterministic optimal policy for an MDP. This is true. But note that this does not imply that a stochastic optimal policy can not exist at the same time.
Suppose you have an MDP with one state and two actions $a_1$ and $a_2$, both yielding the reward 0 in expectation (as in your example). Then consider a policy that takes action $a_1$ with probability $\alpha \in [0,1]$ and $a_2$ with probability $1-\alpha$. Either of the two deterministic policies with $\alpha=0$ or $\alpha=1$ are optimal, but so is any stochastic policy with $\alpha \in (0,1)$. All of these policies yield the expected return of 0.
This is all assuming your optimality criterion is the expected cumulative (discounted) reward. If you have a different optimality criterion, such as something that accounts for risk, you might distinguish between rewards that have the same expected value but a different variance - but I think that is beyond the scope of the question.


Create an AI to solve a puzzle (by deduction) Some puzzle games have a unique solution that can be solved by deduction rather than guesswork (e.g. Slitherlink, Masyu). Using a computer to solve this puzzle it's pretty easy, we can use a backtracking method to find the best solution in second (in general, the puzzle size is not too big).

  Is it possible to train a bot to solve this kind of puzzle by deduction?

I think by train it to watch a previous step-by-step solution several times the bot can find some implicit rules/patterns to solve a specific puzzle. Is this possible? are there any references for this method?

What you are trying to achieve sounds a lot like inductive logic programming:

  Given an encoding of the known background knowledge and a set of examples represented as a logical database of facts, an ILP system will derive a hypothesised logic program which entails all the positive and none of the negative examples.


In a single neuron output layer should the output be a scalar? Given a neural network with 3 inputs, 4 hidden layers, and 1 output, should the output neuron be a vector or a scalar? I thought that at the end of the summation only one number between 0 and 1 would be left over for each neuron in the last output layer but my program returns a 1x4 vector.

class Network:
    def __init__(self, inputs, layers):
        # each neuron has n weights attached
        # each neuron has a bias
        self.weights = []
        self.biases = [np.random.randn(y,1) for y in layers[1:]]
        # did this to connect neurons excluding input layer and last neuron
        # because last neuron isn't connecting anything in front
        for x,y in zip(layers[:-1], layers[1:]):
            self.weights.append(np.random.randn(y,x))

    # add bias vector to previous vector matrix product
    def forward(self, inputs):
        for b,w in zip(self.biases, self.weights):
            inputs = sigmoid(np.dot(w, inputs) + b)
            print("layer",inputs)
        self.output = inputs
        print("output",self.output)

input = [random.randrange(256),random.randrange(256),
                random.randrange(256)]

net = Network(input, [3,4,4,1])
net.forward(input)

Output

layer [[7.80671510e-176 2.86955013e-159 1.00000000e+000 9.10681919e-004]
 [8.91574010e-176 3.27719954e-159 1.00000000e+000 1.03991921e-003]
 [3.44313504e-176 1.26560896e-159 1.00000000e+000 4.01858875e-004]
 [1.43702955e-175 5.28215553e-159 1.00000000e+000 1.67506508e-003]]
layer [[0.79051612 0.79051612 0.91469971 0.79084515]
 [0.80073915 0.80073915 0.70932703 0.80058277]
 [0.35187686 0.35187686 0.00982275 0.35113385]
 [0.51881022 0.51881022 0.55334083 0.5188783 ]]
layer [[0.26291072 0.26291072 0.16592316 0.26264314]]
output [[0.26291072 0.26291072 0.16592316 0.26264314]]

If my output is supposed to decide whether the neuron fires, how could I use the vector to determine this?

Look, your code says your network has many outputs.
Look at the two lines below. This two lines says the output depends on the dimension of np.dot(w, inputs). In your case it's 4 diminutional vector. And in the last line you are assigning them as output. You can write self.output = sigmoid(np.dot(new_weihts, inputs)) instead of self.output = inputs. Must ensure that new_weights is a vector and has same shape as previous layer's output.

inputs = sigmoid(np.dot(w, inputs) + b)
self.output = inputs

Note: Your network don't look like a 4 layers network. It looks like a one layer network with 4 unit of neutrons.


Why one unit in the layers of neural network is not enough? In a deep connected network, when every unit gets all the input features(X) so it has one parameter for every feature and every unit tweaks its parameters for loss optimization. What if we use only one unit and that one unit will have all the parameters which it can tweak for loss optimization. Is there a reason or benefit of using multiple units in every layer except the output layer?

So if I understand correctly, you're proposing to use a neutral net with $N$ input units (let's say data is in $\mathbf{R}^N$), 1 hidden unit, and whatever the necessary output needs to be.
Let's say we try to do this. Then each unit of the output layer is responsible for computing its output based on a single scalar input. So it's like as if you're saying the problem can be compressed to another problem that is 1-dimensional. Now let's say you're doing regression. By modeling your NN this way, you're saying there's a function that (approximately) maps a scalar (one dimensional object) to the target output. Suppose your output layer is a linear layer (which is usually the case in regression), this is saying that the dimension of the output space is at most 1. This is almost never the case, especially with complex data.
Now suppose you had several hidden units (call this number $M$) and the weights are initialized properly. Then, it's likely that your hidden unit will have dimension larger than one, perhaps even $M$, which allows for a richer output space as $M$ increases.
You can make similar arguments for the classification setting, but the output nonlinearity just makes it a little less clear.


Can strong AI now be achieved with Prolog? This survey of Artificial Intelligence with Prolog provides a short perspective on the Prolog programming language and its position in the history of AI.

It proposes that a return to Prolog Research and Development as begun in the 1980s when Japan began work on a Fifth Generation Computer System.  Has the reduction in cost and the increase in speed of computing platforms re-opened a door that was closed primarily because of technology immaturity?

The application of Prolog, the declarative programming language may or may not be closely related to AI strength.  Prolog is a language that was designed to produce a Turing Complete backward chaining general programming language.  The strength or weakness of Prolog programs, just as for programs in Scheme, Common LISP, Java or Scala with DRools rules engine, or neuromorphic libraries in Python or C++ depends on more than the language and libraries chosen.

The strength of an artificially intelligent system, from a mathematical perspective, is as ambiguous in meaning as IQ test results.  Did human kind within one decade cross some critical level where they were once stupid and became intelligent or their weak intelligence became strong?There is no evidence for such discrete transition.  Evidence indicates that DNA advancements, many of them, and cultural changes to increase intelligence capacities occurred over thousands or tens of thousands of years.

The same is true of general and narrow AI.  All intelligent systems have strengths, weakness, generalizations, and narrows. This is true whether the intelligence arises out of billions of organic neurons and the brain's complex systems of chemistry, organelle structures, and plasticity capabilities or whether the intelligence arises from designs involving ones and zeros on a collection of interconnected silicon wafers.

Classifying systems into a four quadrant system is counter productive.

Weak narrow
Strong narrow — Extremely effective systems applicable to a narrow set of usage contexts
Weak general — Systems that show adaptive and proactive capabilities over a wide range of usage contexts but demonstrating only limited effectiveness in any given context compared to a typical human
Strong general

The binary system of weak = narrow and strong = general is even more limiting because of the existence of the middle two bullet points above in human populations.

Judgments about intelligence are ALWAYS

Relative to some standard,
Time dependent, and
Context dependent.

More importantly, there is more than one or two dimensions to intelligence — possibly hundreds.

Ken Thompson and Linux Torvalds, clearly the most successful operating system writers in terms of influence on currently deployed von Neuman computers may have great difficulty with writing a song that sustains sales over decades.  Bono and Paul McCartney may have great difficulty with writing an operating system that is downloaded every minute for decades.  Both features and breadths of intelligence have value in current culture and economics.

The appreciation of the Prolog programming language by those like the author of the page referenced in the question is not without reason.  The design of the language provides an advantage for R&D over other languages in rules based contexts.

There may be value in reconsidering the value of the language in light of the significant improvements in computing platforms since earlier efforts were made to use inference to produce learning systems.  The rule type used to learn other rules were called meta-rules, and meta-rules have produced some valuable behaviors in adaptive systems based on Java components such as DRools.


What types of applications qualify as "compound intelligences"? A question about swarm intelligence as a potential method of strong general AI came up recently, and yielded some useful answers and clarifications regarding the nature of swarm intelligence. But it got me thinking about "group intelligence" in general.

Here organism is synonymous with algorithm, so a complex organism is an algorithm made up of component algorithms, based on a set of instructions in the form of a string.

Now consider the Portuguese man o' war, not a single animal, but a colonial organism. In this case, that means a set of animals connected for mutual benefit.

And physalia physalis are pretty smart as a species in that they've been around for a while, I'm not finding them on any endangered lists, and based on their habitat it looks like global warming will be a jackpot for them. And they don't even have brains.

Each component of the physalia has a narrow function, colony organism itself has a more generalized function, which is the set of functions necessary for maintenance and reproduction.

{Man o' War} ⊇ { {pneumatophore}, {gonophores, siphosomal nectophores,vestigial siphosomal nectophores}, {free gastrozooids, tentacled gastrozooids, gonozooids, gonopalpons}, {dactylozooids}, {gonozooids}, {gastrozooids} }

What types of applications qualify as "compound intelligences"? What is the thinking on groups of neural networks comprising generally stronger or simply more generalized intelligence?

I recognize the underlying problem is ultimately complexity and that "strong narrow AI" is, by definition, limited, so I use "generalized" and omit "strong" because human-like and superintelligence are not conditions.  Compound intelligence is defined as a colony of dependent intelligences.*

Utility software is often a form of an expert system that manages a set of functions of varying degrees of complexity. There's currently a great deal of focus on autonomous vehicles, which would seem to require sets of functions.

Links to research papers on this or related subjects would be ideal.

Portuguese Man o' War (oceana.org)

The Bugs Of The World Could Squish Us All

Swarm intelligence, compound intelligence, or group intelligence may emerge as an important concept as AI develops toward higher complexity.  Whether these terms should be considered synonymous is doubtful.

Compound features in biology are the result of control in differentiation during the development of an organism from a single cell.  Compounding in biology is a single function performed across like elements.

Swarms are a result of distinct organisms operating in proximity.  Swarming in biology is similarity in complex independent behaviors that appear coordinated but are generally engaged in as a defense from predators.

Group intelligence may be distinct from compound intelligence in that adjacent units may be in agreement or opposition, as can be case with intelligent beings in groups.  The agreement in the group that a variety of models is permissible.  We call them opinions, but they are distinct matches of models to problems that produce different projections and suggest different selections from among the group's options.

The terms general and strong appear in this question and many others and it appears that, mainly for historical reasons, may continue to frustrate clarity.

All intelligence is general in that it learns a generality from specific experiences and then applies that generality to future scenarios to achieve objectives within those scenarios.  What makes the application of the generality intelligent is that it is expected to work because it has been working for similar scenarios.

All intelligence is specific in that it is limited to scope of what generalities have been discovered.  Certain techniques are significantly more general that others because they are allegedly domain independent.  We call this mathematics.

Consequently, the development of artificial intelligence is not a path from specific to general but one of discovering generalities and applying them to specifics.  One could say that the most general thinking of mathematics applied to the computer is the primary activity of applied artificial intelligence, and the capabilities that emerge are a more sophisticated as more sophisticated mathematics is represented in working software and hardware.

The generalities take on greater complexity so that they can apply to a greater number of specific scenarios.  That is not a gain in strength but a gain in the breadth of potential application.

Physalia physalis is a symbiotic colony of organisms of four types, the pneumatophore (or float), dactylozooids (long tentacles), gastrozooids (feeding tentacles), and gonozooids which produce reproductive gametes.  To avoid breaking the historic conception of animals, these organisms are called polyps.  The venom of the colony does not come from any of those organisms but rather from another symbiotic organism, cnidocytes, which attach to the tentacles and are released under strictly controlled conditions.

In addition to all this symbiosis, there are several species of fish that use the colony for shade and protection that the colony lets swim among the tentacles and that have formed partial immunity to the cnidocytes.  These complex symbiotic networks are not fully understood, but they seem to operate well and create sustainable inter-specie systems.

It's not likely that the organisms, within their lifetimes adapt or remember, but the DNA of each organism has involved in a way that resembles intelligence in that the colony and its symbionts have adapted to the ocean's surface and its biology.

The idea that the system of the biosphere is the first example of broad intelligence is likely, in that design excellence has emerged from an evolutionary processes.  It is not altogether ridiculous to propose that human intelligence is a higher speed approach in neurons to the slower speed of DNA replication in larger organisms.  Because of the metabolic requirements of growth slows evolution for larger organisms with a greater cell count, these larger organisms may have needed to develop a way to achieve the nimble adaptivity of their lower ancestors.

Neurology facilitates the approximation of some aspects of evolution and may have been the most attainable natural solution to reacquire nimble adaptivity.

Attempting to apply these various ideas to the current and ongoing development of autonomous vehicles reveals a gap in understanding.  We don't yet have the mathematics developed to understand how compound, swarm, or group intelligence can be used in the laboratory to accomplish well defined problems in controlled execution scenarios.  That is probably a prerequisite to using these ideas in vehicle control.

The system designs of future cars may be like a colony of independent or semi-independent components that each have a role and purpose.  Compound, group, and symbiotic designs are likely to develop.  The current activities of vehicles on the road or in the air near airports is like a swarm, so the application of that idea is obvious: Avoid collisions.


What's the role of bounding boxes in object detection? I'm quite new to the field of computer vision and was wondering what are the purposes of having the boundary boxes in object detection.
Obviously, it shows where the detected object is, and using a classifier can only classify one object per image, but my question is that

If I don't need to know 'where' an object is (or objects are) and just interested in the existence of them and how many there are, is it possible to just get rid of the boundary boxes?

If not, how does bounding boxes help detect objects? From what I have figured is that a network (if using neural network architectures) predicts the coordinates of the bounding boxes if there is something in the feature map. Doesn't this mean that the detector already knows where the object is (at least briefly)? So, continuing from question 1, if I'm not interested in the exact location, would training for bounding boxes be irrelevant?

Finally, in architectures like YOLO, it seems that they predict the probability of each class on each grid (e.g. 7 x 7 for YOLO v1). What would be the purpose of bounding boxes in this architecture other than that it shows exactly where the object is? Obviously, the class has already been predicted, so I'm guessing that it doesn't help classify better.

A bounding box is a rectangle superimposed over an image within which all important features of a particular object is expected to reside. It's purpose is to reduce the range of search for those object features and thereby conserve computing resources: Allocation of memory, processors, cores, processing time, some other resource, or a combination of them. For instance, when a convolution kernel is used, the bounding box can significantly limit the range of the travel for the kernel in relation to the input frame.

When an object is in the forefront of a scene and a surface of that object is faced front with respect to the camera, edge detection leads directly to that surface's outlines, which lead to object extent in the optical focal plane. When edges of object surfaces are partly obscured, the potential visual recognition value of modelling the object, depth of field, stereoscopy, or extrapolation of spin and trajectory increases to make up for the obscurity.

  A classifier can only classify one object per image

A collection of objects is an object, and the objects in the collection or statistics about them can be characterized mathematically as attributes of the collection object. A classifier dealing with such a case can produce a multidimensional classification of that collection object, the dimensions of which can correspond to the objects in the collection. Because of that case, the statement is false.

  1) If I don't need to know 'where' an object is (or objects are) and just interested in the existence of them and how many there are, is it possible to just get rid of the boundary boxes?

If you have sufficient resources or patience to process portions of the frame that don't contain the objects, yes.

Questions (2) and (3) are already addressed above, but let's look at them in that context.

  2.a) If not, how does bounding boxes help detect objects?

It helps by fulfilling its purpose, to reduce the range of the search. If by thrifty method a bounding shape of any type can be created, then the narrowing of focus can be used to reduce the computing burden on the less thrifty method by eliminating pixels that are not necessary to peruse with more resource-consuming-per-pixel methods. These less thrifty methods may be necessary to recognize surfaces, motion, and obscured edges and reflections so that the detection of object trajectory can be obtained with reliability.

That these thrifty mechanisms to find the region of focus and these less thrifty mechanisms to use that information and then determine activity at higher levels of abstraction are artificial networks of this type or that or use algorithms of this type or that is not relevant yet. First understand the need to reduce computing cost in AI, which is a pervasive concept for anything more complex than tic-tac-toe, and then consider how bounding boxes help the AI engineer and the stakeholders of the engineering project to procure technology that is viable in the market.

  2.b) From what I have figured is that a network (if using neural network architectures) predicts the coordinates of the bounding boxes if there is something in the feature map. Doesn't this mean that the detector already knows where the object is (at least briefly)?

  2.c) So continuing from question 1, if I'm not interested in the exact location, would training for bounding boxes be irrelevant?

Cognition is something AI seeks to simulate, and many hope to have robots like in the movies that can help out and be invaluable friends, like TARS in the Nolan brothers 2014 film Interstellar. We're not there. The network knows nothing. It can train a complex connection between an input signal through a series of attenuation matrices and activation functions to produce an output signal statistically consistent with its loss function, value function, or some other criteria.

The inner layers of an artificial net may, if not directed to do so, produce something equivalent to a bounding region only if velocity of convergence is present as a factor in its loss or value function. Otherwise there is nothing in the Jacobian leading convergence to reduce its own time to completion. Therefore, the process may complete, but not as well as if cognition steps in and decides that the bounding region will be found first and then used to reduce the total burden of mechanical (arithmetic) operations to find the desired output signal as a function of input signal.

  3) Finally, in architectures like YOLO, it seems that they predict the probability of each class on each grid (e.g. 7 x 7 for YOLO v1). What would be the purpose of bounding boxes in this architecture other than that it shows exactly where the object is? Obviously, the class has already been predicted so I'm guessing that it doesn't help classify better.

Reading the section in A Real-Time Chinese Traffic Sign Detection Algorithm Based on Modified YOLOv2, J Zhang, M Huang, X Jin, X Li, 2017, may help further comprehension of these principles and their almost universal role in AI, especially the text around their statement, "The Network Architecture of YOLO v2 YOLO employs a single neural network to predict bounding boxes and class probabilities directly from full images in one inference. It divides the input image into S × S grids." This way you can see the use of these principles in the achievement of specific research goals.

Other such applications can be found by simply reading the article full text available on the right side of an academic search for yolo algorithm and using ctrl-f to find the word bound.


What features should a dataset to predict monthly retail sales for a motorcycle spare parts shop have? I am making an AI model to predict monthly retail sales of a motor cycle spare parts shop, for that to be possible I have to first create a dataset. The problem I am facing is what features should the dataset have?

I already did some research on some other datasets but still I want to know specifically what features should it have other than Date, Product Name, Quantity, Net amount, Gross amount..?

This is the prototypical problem in AI/ML known as feature-selection. If you do not have too many features, typically one would just use them all(with the exception of features that are known to to be correlated), in this case you would want to perform feature engineering wherever possible as well. One the other hand, if you have lots of features, and some are likely to be useless, you would use a feature-selection technique.

There are many many methods for doing this. One can simply use their intuitive grasp as their problem domain at the simplest level. However, there are also many algorithmic ways of performing it. Some examples of this are a low variance filter, or using a ensemble(classifier or regression depending on the problem) which can then be used to order features according to their derived importance(I personally like this).

A search of "feature selection methods in ML" will yield many potential ways to accomplish your goal.

As far as what to grab if you don't have many, as many as possible wrt domain knowledge.


Is the singularity something to be taken seriously? The term Singularity is often used in mainstream media for describing visionary technology. It was introduced by Ray Kurzweil in a popular book The Singularity Is Near: When Humans Transcend Biology (2005).

In his book, Kurzweil gives an outlook to a potential future of mankind which includes nanotechnology, computers, genetic modification and artificial intelligence. He argues that Moore's law will allow computers an exponential growth which results in a superintelligence.

Is the technological singularity something that is taken seriously by A.I. developers or is this theory just a load of popular hype?

I can say that among AI researchers I interact with, it far more common to view it as wild speculation than as settled fact.

This is borne out by surveys of AI researchers, with 80% thinking strong forms of AI will emerge in "more than 50 years" or "never", and just a few percent thinking that such forms of AI are "near".

Software Developers are not the same as AI researchers, and I have found the Singularity myth to be much more widespread among developers. It has a nice ring to it: Computers keep getting faster, at some point they'll be faster than brains, at that point we just simulate brains. Soon after, we simulate something better than brains.

I suspect that the reasons AI researchers are less optimistic are rooted in the fact that we still don't have a good understanding of human intelligence, or even enough of an understanding of the brain to simulate it. For example, in the last two weeks we have discovered previously unknown types of brain cells. This gives the (correct) impression that even if we had a fast enough computer, we are not at all close to being able to accurately simulate a human brain. We don't really know what a human brain is.

Even if we did know that, simulations are necessarily lossy. We may not have good simulation techniques. Even if we did have good techniques, we may simulate the brain and discover our simulation does not behave as expected for reasons that we don't understand. This is very probable when simulating new systems. In some sense, proponents of the Singularity resemble people predicting that weather control was near in the 1940s. After all, we could simulate simple weather patterns already then, and generate forecasts that sort of worked. How much more complex could it really be to generate perfect forecasts?


How do we classify an unrecognised face in face recognition? If we have classified 1000 people's faces; how do we ensure the network tells us when it encounters a new person?

Specifically for face recognition (and other identification algorithms) there are better approaches than using classifiers directly.

Most identity recognition algorithms generate some kind of metric - typically an "embedding" of the original image into an abstract space i.e. a vector of real numbers. The space might be based on real-world biometrics e.g. normalised measurements of eye distance, eyebrow arch etc, which would be trained as a regression algorithm. The problem with this is that it requires a lot of labelled data, and the biometrics are not necessarily good at differentiating between identities. An alternative is to get the neural network to find the best abstract space for identities. You can do this if you have at least two images of each identity, and using triplet loss to train the network - the loss function directly rewards embedding the same identity close and different identity far apart.

Once you have an embedding, you no longer directly classify identities using the neural network. Instead, you base identity on distance between measured embedding and stored embeddings. This requires implementing a search function that looks at known embeddings and sorts by distance.

  how do we ensure the network tells us when it encounters a new person?

Embeddings don't solve this problem directly, but give a useful heuristic - distance in embedding space. Typically a maximum allowed distance is set as a cutoff to consider an image as showing a new identity. This is a hyperparameter of the model. This is an area that triplet loss helps with, since it is trained to make the distance as large as possible between images that show different identities. If it has generalised well during training, then it should ignore differences due to lighting, pose, makeup etc, but still be able to differentiate similar looking people.

As the embedding is approximate, any such system may make mistakes, and needs to be carefully tested. The quality and quantity of training data are important, and it should match images used in production. But that is no different to the pure classifier, which must in addition be re-built and re-trained for every new class added.

Whether to use a more basic classifier or something like triplet loss is a question of scale - if the number of identities that need to be tracked is high, or the rate of change in identities is high, then embeddings trained on triplet loss (or similar) are more practical.


How do I take the correct classification predictions of an ml algo (i.e. random forest/neural net) and sort the instances in each category? I am trying to sort the instances within each of 5 classification categories in a dataset that has been put through both a random forest classifier and a neural network with 99% accuracy on each.
Essentially what I am trying to do is stack a sorting algorithm on top of a random forest or a neural net (depending on which will boast a much more efficient and swift process in being integrated with a sorting algorithm post-output) so that the correctly classified instances within each category can be sorted into separate organized and easily comprehensible lists.
I have tried researching this but all I have found are examples of ensemble learning and traditional stacking in order to achieve higher overall accuracy on an algorithm.
I am trying to see how I can take the correct predictions from either of these algorithms and sort them by some arbitrary features that I will engineer, but all I am wondering at the moment is how to integrate a sorting algorithm in the first place into the output of a classification algorithm.

For the ANN, it should be the average of the error per instance from testing (prediction) when each instance is left out of training.  ANNs can unfortunately learn based on the order of instances used for training, so it helps to train/test and then shuffle (permute, or randomly re-order) and then assign to k-folds, then train/test again in order to prevent the ANN from learning based on instance order.  The RF won't do this, since the in-bag objects in all the bootstraps (trees) that were sampled with replacement will vary appreciably.
(FYI - there's commonly an option to train using "batch" or "online" error updates during back-propagation.  Batch sums the error over the training objects, then updates coeffs.   Online updates coeffs as each training instance is processed.  I tend to like online, since online seems to reduce error faster.  Batch can get off the gradient path easier, since the summed error can come in large "chunks" and throw the ANN off course)
As far as sorting instances within each class label after training/testing, you can do anything you wish.  Certainly tracking each instance as it is tested/predicted and building an array/vector of errors will give you a distribution of error per instance.
Quite often, test objects can widely very in their feature values, so e.g. k-means cluster analysis can be used during ANN training.  See my blog on this.
In short, it's not just sorting when done.  Ensembles are on the back end of an attack plan, but the checklist for best practices would look like:

Randomly shuffle the order of all instances.
Assign instances to $k$-folds (partitions), e.g. 10-fold.
Train with instances in folds 1-9, test (predict class) of instances
in fold 10.  Increment confusion matrix.  Then train with instances
in folds 1-8,10 and test instances in fold 9, update confusion
matrix, ...., cycle through all test folds.
Repeat steps 1-3 nine more times (this is called doing another "re-partition").
Calculate average error for each predicted instance, then sort.

The above steps are only for the ANN.  The RF performs its own bookkeeping.


Why do DQNs tend to forget? Why do DQNs tend to forget? Is it because when you feed highly correlated samples, your model (function approximation) doesn't give a general solution?
For example:

I use level 1 experiences, my model $p$ is fitted to learn how to play that level.

I go to level 2, my weights are updated and fitted to play level 2 meaning I don't know how to play level 1 again.

You are referring to catastrophic forgetting which could be an issue in any neural net. More specifically for DQN refer to this article.


Given two optimal policies, is an affine combination of them also optimal? If there are two different optimal policies $\pi_1, \pi_2$ in a reinforcement learning task, will the linear combination (or affine combination) of the two policies $\alpha \pi_1 + \beta \pi_2, \alpha + \beta = 1$ also be an optimal policy?
Here I give a simple demo:
In a task, there are three states $s_0, s_1, s_2$, where $s_1, s_2$ are both terminal states. The action space contains two actions $a_1, a_2$.
An agent will start from $s_0$, it can choose $a_1$, then it will arrive $s_1$，and receive a reward of $+1$. In $s_0$, it can also choose
$a_2$, then it will arrive $s_2$, and receive a reward of $+1$.
In this simple demo task, we can first derive two different optimal policy $\pi_1$, $\pi_2$, where $\pi_1(a_1|s_0) = 1$, $\pi_2(a_2 | s_0) = 1$. The combination of $\pi_1$ and$\pi_2$ is
$\pi: \pi(a_1|s_0) = \alpha, \pi(a_2|s_0) = \beta$. $\pi$ is an optimal policy, too. Because any policy in this task is an optimal policy.

Yes, in general any linear combination of probability distributions between optimal policies is also an optimal policy. In fact any combination with each state treated separately will also be an optimal policy.
This can be seen using the equation for optimal deterministic policy in terms of optimal value function:
$$\pi^*(s) = \text{argmax}_a [\sum_{r,s'}p(r,s'|s,a)(r + \gamma v^*(s'))] = \text{argmax}_a [q^*(s,a)]$$
The only way to have multiple equivalent optimal policies is when there are two or more actions tied for $\text{max}_a [q^*(s,a)]$. If that is the case, then it does not matter - in terms of expected future reward - which of those actions is taken in the affected state. It is possible to take one or the other action.
You can use any rule you wish to decide which of the tied-for-max actions to take, including a random choice. Creating a new policy that is a linear combination of optimal policies is one way to make that random choice. You could equally say that you would make one action choice on a Monday and another on Tuesday - in terms of being optimal choices it does not matter at all how you break ties.


Shouldn't Gödel's incompleteness theorems disprove the physical symbol system hypothesis? According to the Wikipedia page of the physical symbol system hypothesis (PSSH), this hypothesis seems to be a vividly debated topic in philosophy of AI. But, since it's about formal systems, shouldn't it be already disproven by Gödel's theorem?

My question arises specifically because the PSSH was elaborated in the 1950s, while Gödel came much earlier, so at the time the Incompleteness theorems were already known; in which way does the PSSH deal with this fact? How does it "escape" the theorem? Or, in other words, how can it try to explain intelligence given the deep limitations of such formal systems?

The PSSH is often attacked via either Godel's theorems or Turing's incomputability theorem.
However, both attacks have an implicit assumption: that to be intelligent is to be able to decide undecidable questions. It's really not clear that this is so.
Consider what Godel's theorems say, in essence:

"powerful" formal systems cannot prove, using only techniques from within the system, that they are self-consistent.
There are statements that are true that cannot be proven within a given "powerful" formal system.

Suppose that we allow both of those facts. The missing step in the argument is the following statements:

You need to be able to prove the consistency of your own reasoning system to be considered intelligent.
You need to be able to correct reason out a proof of all true statements to be considered intelligent.

The main problem is, under this definition, humans are probably not considered intelligent! I certainly have no way to prove that my reasoning is sound and self-consistent. Moreover, it is objectively not so! I frequently believe contradictory things at the same time.
I also am not able to reason out proofs of all the statements that appear to be true, and it seems entirely plausible that I cannot do so because of the inherent limitations of the logical systems I'm reasoning with.
This is a contradiction. The overall argument was that one of these 4 statements is false:

Godel's theorems say symbol systems lack some important properties.
Intelligent things have the properties that Godel says symbol systems lack.
Humans are intelligent.
Humans can't do the things Godel says symbol systems can't do.

Some authors (like John Searle) might argue the false premise is 4. Most modern AI researchers would argue that the false premise is 2. Since intelligence is a bit nebulous, which view is correct may rely on metaphysical assumptions, but most people agree on premises 1 & 3.


Is there any programming practice website for beginners in Reinforcement Learning I am doing an online course on Reinforcement Learning from university of Alberta.
It focus too much on theory. I am engineering and I am interested towards applying RL to my applications directly.

My question is, is there any website which has sample programmers for beginners. Small sample programs.
I have seen several websites for other machine learning topics such as CNN/RNN etc. But the resources for RL are either limited, or I couldn't find them

Firstly, since you are a beginner, I strongly recommend you start reading Sutton's book. It is a really great book.

Then, some tutorials:

udemy rl

udemy deep rl

rl-with-tensorflow

learndatasci

stackabuse


Detecting license plate using tensorflow I'm currently working on license plate recognition. My system consist of 2 stage: (1) License Plate region extraction & (2) License Plate region recognition.

I'm doing (1) with Raspberry pi 3 model b. I find license plate candidate first by merging bounding boxes based on their similarity. In this way, i have only 1~7 license plate region proposals. And it took less than .3 seconds.

Now i have to reduce number of region proposal to be around only 1~2 so that i can send these images to server to do job (2). For license plate extraction, I made my own classifier function in tensorflow and the code is below. It gets proposed license plate as input.

First, I resize all license plate to be [120, 60] and converted to gray image. And there are 2 classes: 'plate', 'non_plate'. For non_plate image, i collected various image that might appear in image as background. I have 181 images for 'plate' class and 56 images for 'non_plate' for now, i trained for about 3000 steps so far and current loss is .53 .

When i did prediction on test set,  i encountered problem that for some of plate image, it doesn't recognize license plate which is very obviously license plate image from my eyes. It is okay for me to wrongly recognize non plate image as plate but it is problem if it wrongly recognize plate as non_plate because it will not be sent to server to be fully recognized.

It happens like 10 out of 100 test images and this rate is far worse than i expected. I need help for adressing this problem. Would there be any improvement that i can make?

(1) Is my training set too small to classify between license plate
    and non license plate? Or is number of steps is too small?

(2) Is my graph structure bad? I needed to have small graph
    structure for my raspberry pi to recognize less than 1 second. Could
    you suggest better structure if it is bad?

(3) Is it bad to resize any proposed image to [120, 60] to be used
    as input for graph? I think it loses some information. But isn't
    this close to roi pooling like used in fast rcnn?

 inputs=tf.reshape(features[FEATURE_LABEL],[-1,120 , 60 ,1],name="input_node") #120 x 60 x 1, which is gray

conv1=tf.layers.conv2d(inputs=inputs,
                       filters=3,
                       kernel_size=[3,3],
                       padding='same',
                       activation=tf.nn.leaky_relu
                       )
#conv1 output shape: (batch_size,120,60,3)

pool1=tf.layers.max_pooling2d(inputs=conv1,pool_size=[2,2],strides=2,padding='valid')

#pool1 output shape: (batch_size,60,30,3)

conv2=tf.layers.conv2d(inputs=pool1,filters=6,kernel_size=[1,1],padding='same',activation=tf.nn.leaky_relu)

#conv2 output shape: (batch_size, 60,30,6)

pool2=tf.layers.max_pooling2d(inputs=conv2,pool_size=[2,2],strides=2,padding='valid')

#pool2 output shape: (batch_size, 30,15,6)

conv3=tf.layers.conv2d(inputs=pool2,filters=9,kernel_size=[3,3],padding='same',activation=tf.nn.leaky_relu)

#conv3 output shape: (batch_size, 30,15,9)

pool3=tf.layers.max_pooling2d(inputs=conv3,pool_size=[2,2],strides=2,padding='valid')

#pool3 output shape: (batch_size, 15,7,9)

#dense fully connected layer
pool2_flat=tf.reshape(pool3,[-1,15*7*9]) #flatten pool3 output to feed in dense layer

dense1=tf.layers.dense(inputs=pool2_flat,units=120,activation=tf.nn.relu)

logits=tf.layers.dense(dense1,2) #input for softmax layer

 [training non plate image example]
[]4
[training plate image example. It is region proposed image]

Such a task is getting easier to complete, but there are still difficulties with rotations, skewing, scaling---to name a few issues. Your network has the benefit of simplicity and lightness for the target hardware, but it may suffer under the above conditions.

So 237 images (181+56) may be small for a "generic" approach, depending on how representative and diverse the dataset is. Also, the dataset is unbalanced (the first class has twice as many examples), which causes bias in learning.

There are several ways to expand a base dataset:

Transform images and add them (with the same label, if supervised learning) to the dataset. Many libraries allow to rotate, skew, scale, or even blur images. Be careful as transforming needs be "reasonable", and changing image format over and over creates artifacts that can confuse the machine (e.g. too much JPEG savings on transformed images).
Generate synthetic data. Assuming license plates have a known format, it may be easy to generate images with good fidelity to real plates. This is not always possible, but license plates are formalized, so there should be only a handful patterns (typically plain, light, diplomatic, military vehicles).

Aside the dataset (potential) issue, the graph is fine. However, it may be worth trying different settings. It really depends on how you have ended up with the current graph. Removing pooling layers helps keeping more information, as well as a larger input image. 120x60 looks pretty small, and the comparison to fast-RCNN's RoI pooling layer looks odd, given that RoI comes after the feature maps. So a larger input image could give better results.


What is the appropriate way to deal with multiple paths to same state in MCTS? Many games have multiple paths to the same states. What is the appropriate way to deal with this in MCTS?

If the state appears once in the tree, but with multiple parents, then it seems to be difficult to define back propagation: do we only propagate back along the path that got us there "this" time? Or do we incorporate the information everywhere? Or maybe along the "first" path?

If the state appears once in the tree, but with only one parent, then we ignored one of the paths, but it doesn't matter because by definition this is the same state?

If the state appears twice in the tree, aren't we wasting a lot of resources thinking about it multiple times?

If the state appears twice in the tree, aren't we wasting a lot of resources thinking about it multiple times?

You're right. Precisely the same problem was also noticed decades before MCTS existed, in the classic minimax-style tree search algorithms (alpha-beta search, etc.) that were used in games before MCTS. The solution is also mostly the same; transposition tables.

In the case of MCTS, the statistics used by the algorithm that are normally associated with nodes (or their incoming edges) may instead be stored in entries of a transposition table. I mean stuff like visit counts and sums (or averages) of backpropagated scores.

A brief description of how it would work, and references to more extensive relevant literature, can be found in subsubsection 5.2.4 of the well-known 2012 Survey paper on MCTS.

This does require that you can efficiently (incrementally) compute hash values for the states you encounter, which may not always be easy (should usually be possible, depends on the details of your problem domain). Use of transpositions in MCTS is also not always guaranteed to actually improve performance. It does come with computational overhead, and in games where transpositions are very rare it may be more efficient to simply ignore them and use the regular tree structure.


Which functions can be activation functions? What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions?

For example, which of the functions below can be used as an activation function?

$$f(x) = \frac{2}{\pi} \tan^{-1}(x)$$

which looks like

or

$$f(x) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{x} e^{-\frac{t^2}{2}} dt$$

which looks

The main characteristic of an activation function is to bring a non-linearity property into the neural network. For the hidden layer, there is no need for the function to be bounded. The last layer should use a function whose range corresponds to what you want.
For regression, you usually re-scale your output data to $[-1,1]$ or $[0,1]$ and you use a tanh (hyperbolic tangent) or sigmoid function in the last layer
For classification, you want to obtain probabilities: use a softmax function in the last layer.
For the hidden layers some functions are better than others :

The gradient should be fast to compute (from the perspective of your computer).

If you use too many hidden layers, you will have the vanishing gradient problem if the derivative of your activation is too close to zero. You need a large zone of the domain with a derivative not close to zero.

In practice, the ReLU function defined as $f(x)=\max(0, x)$ works very well and is very simple.


How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$? This is the exercise 3.18 in Sutton and Barto's book.
The task is to express $v_\pi(s)$ using $q_\pi(s,a)$.
Looking at the diagram above, the value of $q_\pi(s,a)$ at $s$ for each $a \in A$ we take gives us the value function at $s$ after taking the action $a$ and then following the policy $\pi$.
This is probably wrong, but if
$$v_\pi(s) = E_\pi[G_t | S_t = s]$$
and
$$q_\pi(s) = E_\pi[G_t | S_t = s, A_t = a]$$
isn't then $v_\pi(s)$ just the expected action value function at $s$ over all actions $a$ that are given by the policy $\pi$, namely
$$v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{a \in A}\pi(a|s) q_\pi(s,a)$$?

isn't then $v_\pi(s)$ just the expected action value function at $s$ over all actions $a$ that are given by the policy $\pi$, namely

$v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{a \in A}\pi(a|s) q_\pi(s,a)$?

Yes this is 100% correct.
There is no "trick" to this or deeper thought needed. You have correctly isolated the key part of the MDP description that controls relationship between $v_{\pi}$ and $q_{\pi}$ in that direction.
Note that for a deterministic policy, with $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ then the relationship is
$$v_\pi(s) = q_\pi(s, \pi(s))$$
The related exercise in the book - expressing $q_{\pi}$ in terms of $v_{\pi}$ and the MDP characteristics - is more complex because it involves a time step.


What are the minimum requirements to call something AI? I believe artificial intelligence (AI) term is overused nowadays. For example, people see that something is self-moving and they call it AI, even if it's on autopilot (like cars or planes) or there is some simple algorithm behind it.

What are the minimum general requirements so that we can say something is AI?

It's true that the term has become a buzzword, and is now widely used to a point of confusion - however if you look at the definition provided by Stuart Russell and Peter Norvig, they write it as follows:

  We define AI as the study of agents that receive percepts from the environment and perform actions. Each such agent implements a function that maps percept sequences to actions, and we cover different ways to represent these functions, such as reactive agents, real-time planners, and decision-theoretic systems. We explain the role of learning as extending the reach of the designer into unknown environments, and we show how that role constrains agent design, favoring explicit knowledge representation and reasoning.

Artificial Intelligence: A Modern Approach - Stuart Russell and Peter Norvig

So the example you cite, "autopilot for cars/planes", is actually a (famous) form of AI as it has to use a form of knowledge representation to deal with unknown environments and circumstances. Ultimately, these systems also collect data so that the knowledge representation can be updated to deal with the new inputs that they have found. They do this with autopilot for cars all the time

So, directly to your question, for something to be considered as "having AI", it needs to be able to deal with unknown environments/circumstances in order to achieve its objective/goal, and render knowledge in a manner that provides for new learning/information to be added easily. There are many different types of well defined knowledge representation methods, ranging from the popular neural net, through to probabilistic models like bayesian networks (belief networks) - but fundamentally actions by the system must be derived from whichever representation of knowledge you choose for it to be considered as AI.