Is the following the correct implementation of the Q learning algorithm for a neural network? I just wanted to confirm that my understanding of Q learning was correct (with respect to a neural network).

The network, Q, is initialised randomly.
for n "episodes":
    The state, s1, is initialised randomly
    while s1 != terminal state:
        s1 is fed into Q to get action vector q1
        action a1 is chosen based off the max of q1 (or randomly)
        state s2 is found by progressing s1 based on a1

        s2 is fed into Q to get action vector q2

        The expected output for Q at q1, y, is found by:
        {If s2 is terminal, it is the reward at s2
        {Otherwise: reward at s2 + gamm*max(q2)
        (The "otherwise" doesn't match bellmans equation as α=1)

        Do gradient step where error = (y - max(q1))^2, only the max of q1 gets any gradient
        s1 = s2

This does not directly follow equations found by searching Q-learning as I find them rather ambiguous.

I am also not taking into account storing network states (in this case, the network is called Q) for proper learning to avoid catastrophic forgetting, as I'm more concerned on getting the specifics right before good practice.

Your pseudocode is fairly close to something that could be implemented. There are a couple of omissions, where it is not clear whether your implementation would do the right thing. There are a few non-general assumptions that are worth noting - they don't prevent you implementing for a specific purpose, but do prevent you writing the most generic DQN that you could. There is one mistake that prevents it working at all.

Your mistake is here:

  Do gradient step where error = (y - max(q1))^2, only the max of q1 gets any gradient

The loss function and gradient must be applied to the state/action pair s1, a1, i.e. the action that was taken. This is an important distinction for when the agent takes exploratory (non-greedy) actions, as it must if it is going to learn about their values.

A corrected statement might read: Do gradient step where error = (y - Q(s1,a1))^2, only the output for a1 gets any gradient.

Regarding the rest of the pseudocode I could make the following observations:

  The state, s1, is initialised randomly

This is fine if it is possible. However, it is more usual in Q learning to initialise the state to whatever the normal start of the episode would be. That might be from some randomised starting rules, but that is not quite the same as
picking a random state from all possible states.

  action a1 is chosen based off the max of q1 (or randomly)

It is not quite clear what you mean by "or randomly". For Q learning - and most RL - exploration is important. So this is not a design time choice for your algorithm, but a runtime choice that must itself be made so that all actions can be tried. Typically you would use an $\epsilon$-greedy policy here that took the argmax action by default, but with probability $\epsilon$ took a random sample from actions with same chance of each possible action.

  state s2 is found by progressing s1 based on a1

This is one of a few places where you have not quite captured the nature of a time step in a general MDP. You appear to have made the assumption that rewards are directly associated with states (with language like "reward at s2"). Whilst some MDPs are like that, it is not true in general.

A better, more general statement might be:

s2, reward, done are found by taking a step from s1, using a1 as the action

Typically the environment (or a wrapper for it that establishes state representations and rewards) will provide a step method that returns the tuple $s_{t+1}, r_{t+1}, (s_{t+1} = s_T)$, although you could also hold these values in the environment model for the current time step and return them as required.

This statement:

  (The "otherwise" doesn't match bellmans equation as α=1)

I don't know which Bellman equation you are referring to, but they don't include a learning rate, $\alpha$ in any version I have seen. Perhaps you are referring to a tabular Q learning update? In any case, it doesn't seem relevant, as whatever learning rate you are using is now inside the gradient step, and does not need to be part of the pseuodcode.

Finally, regarding this:

  I am also not taking into account storing network states (in this case, the network is called Q) for proper learning to avoid catastrophic forgetting, as I'm more concerned on getting the specifics right before good practice.

DQN will very likely not work at all, for most environments, unless you implement experience replay i.e. training the neural network on a random sample of stored s1,a1,r,s2,done taken from a table that you build up on each time step. Usually you cannot simply use the latest data from the current step. This is not just "good practice", but an almost-always necessary stability improvement.


Appropriate convolutional neural network architecture when the input consists of two distinct signals I have a dataset consisting of a set of samples. Each sample consists of two distinct desctized signals S1(t), S2(t). Both signals are synchronous; however, they show different aspects of a phenomena.
I want to train a Convolutional Neural Network, but I don't know which architecture is appropriate for this kind of data.
I can consider two channels for input, each corresponding to one of the signals. But, I don't think convolving two signals can produce appropriate features.
I believe the best way is to process each signal separately in the first layers, then join them in the classification layers in the final step. How can I achieve this? What architecture should I use?

I don't know what you mean by desctized signals but if I understand your question correctly, separating two signal and passing them through same architecture of CNN (even with different parameters) is not a good idea. Because when they are together (as different channels) they will be treated differently by the CNN (each channel has its own parameters) and even this way the network is able to combine these two signals and get better results by information extracted from this combination.


Does it make sense to include constant states into reinforcement learning formulation? Does it make sense to incorporate constant states in the Markov Decision Process and employ a reinforcement learning algorithm to solve it?
For instance, for applications of personalization, I would like to include users' attributes, like age and gender, as states, along with other changing states. Does it make sense to do so? I have a sequential problem, so I assume the contextual bandit does not fit here.

It is, I suppose, a philosophical question whether data that describes a whole episode and does not respond to events within it is part of the state, or is part of some other structure.
However, the practical response is to view such descriptive data as defining an instance of a class of related environments, and to include it in the state features. This may be done for two main reasons:

The static data is a relevant parameter of the environment, affecting state transitions and rewards.

It is possible to generalise over the population of all values that the parameters can take.

In simple environments, generalisation might only be that the same agent can learn about all variations in a single combined training session. You could use a tabular RL method, starting randomly with one of the possible variations until all were sufficiently covered.
In more complex environments, generalisation may also occur through functon approximation, in a similar manner to contextual bandits. In your personalisation example, you are not expecting to train the agent for all possible user descriptions, but hope that people with similar age, gender etc descriptions will respond similarly to an agent that personalises content.
Philosophically, the contextual data is either part of a larger state space (with a restriction that transitions between different contexts do not happen within an episode), or it is metadata that impacts the "real" state transitions and rewards. Pragmatically, to allow the data to influence value functions and policies, it is necessary to use it in the arguments of those functions. Whether you then view it as part of the state feature vector or as something that is concatenated to state features is a personal choice. Most of the literature I have seen assumes without comment that it is part of the state.


What does a consistent heuristic become if an edge is removed in A*? For the A* algorithm, a consistent heuristic is defined as follows:
if, for every node $n$ and every successor $m$ of $n$ generated by any action $a$, $h(n) \leq c(n,m) + h(m)$.

Suppose the edge $c(n,m)$ is removed, how do we define consistency for nodes $n, n'$ ? Since the $n'$ is not generated from $n$.

Consistency is a property of heuristics. You can think of consistency as the common sense idea that our guess at the time to go from $A \rightarrow B \rightarrow C$ cannot be more than the time to go from $A \rightarrow B$, plus our guess of the time to go from $B \rightarrow C$.

Supposing we remove a given edge $c(n,m)$ from our graph, but that our heuristic function gives the same value for $h(n)$ as before. Can the function be inconsistent? Let's try a proof sketch:

Since the function was consistent before we removed this edge, it must already be the case that $h(n) < c(n, m') + h(m')$ for every other node $m'$ adjacent to $n$.
Removing the edge $c(n,m)$ doesn't change these relationships, because the other edges still have the same costs as before, and the heuristic function still gives the same value for every node.
The same argument may be applied to node $m$.
Since none of these relationships have changed, $h$ is still consistent.


Are Q values estimated from a DQN different from a duelling DQN with the same number of layers and filters? I am confused about the Q values of a duelling deep Q network (DQN). As far as I know, duelling DQNs have 2 outputs

Advantage: how good it is to be in a particular state $s$
Value: the advantage of choosing a particular action $a$

We can make these two outputs into Q values (reward for choosing particular action $a$ when in state $s$) by adding them together.

However, in a DQN, we get Q values from the single output layer of the network.

Now, suppose that I use the same DQN model with the very same weights in my input and hidden layers and changing the output layer which gives us Q values to advantage and value outputs. Then, during training, if I add them together, will it give me the same Q value for a particular state, supposing all the parameters of both my algorithms are the same except for the output layers?

Dueling-DQN has different network architecture comparing to vanilla DQN, so I don't think your version will work as well as the Dueling architecture.

  From Wang et al., 2016, Dueling Network Architectures for Deep Reinforcement Learning

On the other hand, since we only have the target Q-value, separating the Q-value into state value and advantage result in the identifiability issue. That is the network might simply learn $V(s)=0$, $A(s,a)=Q(s,a)$ for every state.

To tackle with this issue, we should force an additional constraint on the advantage estimate. We can simply use the equation below as mentioned in the paper, that is, normalize the advantages across actions before combining with the state value.

$$Q(s,a;\theta,\alpha,\beta)=V(s;\theta,\beta)+(A(s,a;\theta,\alpha)-\frac{1}{|A|}\sum\limits_{a'}A(s,a';\theta,\alpha))$$


Sutton & Barto's notation $V_{t+n}$ in Chapter 7: $n$-step Bootstrapping Until Chapter 6 of Sutton & Barto's book on Reinforcement Learning, the authors use $V$ for the current estimate of a state value. Equation (6.1), for example, shows:

$$ V(S_t) \leftarrow V(S_t) + \alpha[G_t - V(S_t)]\ \ \ \ \ \ (6.1)$$

However, on Chapter 7 they add a subscript to $V$. The first time this appears is on page 143 when they define the return from $t$ to $t+1$:

$$ G_{t:t+1} \dot{=} R_{t+1} + \gamma V_t(S_{t+1})$$

and say that $V_t : \mathcal{S} \rightarrow  \mathbb{R}$ is "the estimate at time $t$ of $v_\pi$."

At first I thought I understood this as a natural consequence of considering $n$ steps ahead in the future and needing an extra index to go over the $n$ steps. But then this stopped making sense when I realized that an estimate for a state must be consolidated, no matter at which of $n$ steps that is coming from. After all, a state $s$ has a single value to estimate, $v_\pi(s)$, and that does not depend on $t$.

Then I thought that they are just taking into account that there are many successive estimates of $V$ as the algorithm progresses, so $V_t$ is just the estimate after processing the $n$ steps starting at time $t$. In other words, the subscript would be a rigorous mathematical way of denoting the sequence of algorithmic updates. But this does not make sense either since even in Chapter 6 and before, the estimate is also successively updated. See Equation (6.1), for example. The $V$ on the left-hand side is a different variable from the one on the right-hand side (this is why they must use $\leftarrow$ indicating an assignment as opposed to a mathematical equality with $=$). It could have easily been written with an index as well.

So, what is the purpose of the new index for $V$ in Chapter 7, and why is it more important at this particular chapter?

Edit and elaboration: Going back to the text, it seems to me that the new subscript is indeed added as an attempt for greater clarity, even though the subscript-less notation $V$ from previous chapters might have been kept (and in fact it is still used in the pseudo-code in page 144).

It seems the authors wanted to stress that the update of $V$ happens not only for every trace of $n$ steps, but also at every one of those steps.

However, I think this introduced a technical error, because suppose we just learned from an 10-step episode ($T=10$), using $n = 3$. Then the latest estimate of $v_\pi$ is $V_{T-1} = V_{10 - 1} = V_{9}$. Then at the next episode, the first time $V_{t + n}$ is used to inform a target update, it will be for $\tau = 0$ (from the pseudo-code), which implies $t - n + 1 = 0$, so $t = n - 1$, that is, $V_{t+n}=V_{n-1+n}=V_{2n-1}=V_5$, which is not the most up-to-date estimate $V_9$ of $v_\pi$.

Of course the problem would be easily solved if we simply set the next used estimate $V_{2n + 1}$ to be equal to the last episode's $V_{T-1}$, but to avoid confusion this would have to be explicitly stated somewhere.

So, what is the purpose of the new index for $V$ in Chapter 7, and why is it more important at this particular chapter?

My guess would be that your intuition is correct, and that it's mostly introduced just to clarify exactly which "version" of our value function approximator is going to be used in any particular equation. In previous chapters, which discuss single-step update rules, I guess the authors assumed there was less potential for confusion, and therefore no need to clarify this. Without the clarification, some people might for instance wonder if we should use $V_t$ for our value estimates of an $n$-step return $G_{t+n}$, regardless of how large $n$ is.

  However, I think this introduced a technical error, because suppose we just learned from an 10-step episode ($T=10$), using $n = 3$. Then the latest estimate of $v_\pi$ is $V_{T-1} = V_{10 - 1} = V_{9}$. Then at the next episode, the first time $V_{t + n}$ is used to inform a target update, it will be for $\tau = 0$ (from the pseudo-code), which implies $t - n + 1 = 0$, so $t = n - 1$, that is, $V_{t+n}=V_{n-1+n}=V_{2n-1}=V_5$, which is not the most up-to-date estimate $V_9$ of $v_\pi$.

Once we start considering a situation with more than a single episode, the $V_t$ notation becomes quite confusing. You should read $V_t$ as "the value function approximator that we have available at time $t$ of the current episode". So, if we were to use the symbol $V_0$ within the context of a second episode, that would be identical to what was referred to as $V_T$ in the context of the first episode. The $V_t$ notation can be convenient if we're thinking about our equations with our minds in "math-mode", but becomes highly confusing once we start thinking about practical implementations involving multiple episodes -- this is probably why they did not include it in the pseudocode.

If you really wanted to use the subscript-notation in the pseudocode, you'd have to add an extra term in the subscript that adds up all the durations of all previous episodes. If we then try to work out your example situation, we'd run into another problem though... we'd want to use $V_{t+n+T} = V_{2n-1+T} = V_{15}$ at the first iteration where $\tau = 0$ in the second episode. But, across the two episodes, only $13$ steps have passed, so this does not yet exist! You run into the same issue if you try to work out what happened when $\tau = 0$ in the first episode: applying exactly the same reasoning as in your quote, we would have wanted to use $V_5$ after only $3$ time steps passed in the first episode.

The problem here is that you're trying to use the variable named $t$ in the pseudocode as the subscript for $V$. To get a better idea of what's going on here, let's loop back to the previous page and examine the definition of the $n$-step return:

$$G_{t:t+n} \doteq R_{t+1} + \gamma R_{t+2} + \dots + \gamma^{n-1} R_{t+n} + \gamma^n V_{t+n-1} (S_{t+n}).$$

Ok, we've got that. Now, let's take another look at the update rule in which we use this quantity:

$$V_{t+n}(S_t) \doteq V_{t+n-1}(S_t) + \alpha \left[ G_{t:t+n} - V_{t+n-1} (S_t) \right].$$

Ok. So $V_{t+n-1}$ appears three times in the update rule. Two times explicitly, estimating the value of $S_t$, and once more "hidden" in the definition of $G_{t:t+n}$, where it is used to estimate the value of $S_{t+n}$. Note very carefully what it is that this update rule is doing; it's updating the state estimate of $S_t$. If you now look at the pseudocode again, you'll see a comment on the line where $\tau$ is computed: ($\tau$ is the time whose state's estimate is being updated)!

What this means, is that in the pseudocode, you should be using $\tau$ as the subscript for $V$! If you do that, it'll at least be correct for the first episode. In the pseudocode, the update rule looks like:

$$V(S_{\tau}) \gets V(S_{\tau}) + \dots$$

Plugging in the subscripts from the mathematical definition leads to:

$$V_{\tau + n}(S_{\tau}) \gets V_{\tau + n - 1}(S_{\tau}) + \dots$$

Since the pseudocode defines $\tau = t - n + 1$, we can substitute above:

$$
\begin{aligned}
V_{t - n + 1 + n}(S_{t - n + 1}) &\gets V_{t - n + 1 + n - 1}(S_{t - n + 1}) + \dots\\
V_{t + 1}(S_{t - n + 1}) &\gets V_{t}(S_{t - n + 1}) + \dots
\end{aligned}$$

and now it should make sense again from a practical point of view. At every time step $t$, where $t$ measures number of steps of experience that we have simulated, we simply use the latest value function $V_t$ we have available at that time for bootstrapping in the update rule. When $t + 1 < n$, $S_{t - n + 1}$ is undefined. In these cases, the above update rule doesn't work, which makes sense intuitively because we have not yet progressed far enough into the episode to be capable of compute $n$-step returns.


Is artificial life really life or not? I define Artificial Life as a "simulation" or "copy" of life. However, should it be considered a simulation or copy?
If one had motivation and money, someone could theoretically create evolving computers, with a program that allows mutation OR simply a "simulated" environment with "simulated" organisms.
The computer (or "simulated" organism) would have the ability to reproduce, grow, and take in energy. What if the life evolved to have intelligence? Currently, there are some relatively limited programs that simulate life, but most of them are heavily simplistic. Are they life?
When should something be called life?

If you read Steven Levy's book, Artificial Life,you will find, as I did,

the distinction between biological and "artificial" life blurred. If you think about it, what exactly is "life", anyway? A set of complex systems with emergent behavior capable of evolution and adaptation.

A prototypical biologist may not define life that way. Indeed, he would, not being focused or concerned with the computational aspect, define it in a way that would narrow it down to biological life.

Marvin Minsky mention the concept of luggage words, and I myself came up with the notion of mirage concepts. For the former, that which we don't understand gets lump into the word. For the latter, when you take a "mysterious" concept apart, it vanishes like a mirage does as you get closer.

So what is life? If we look at a "living" organism, we'd all that "life". If we remove a single cell from that organism, we'd still call that "life". But what if we remove a single organelle like, say, a ribosome? Lysosome? Contractile vacuole? Endoplasmic reticulum? Is that still "life"? What if we remove a macromolecule from that? Is that still "life"?

As you see, all becomes very murky, and I do this on purpose to illustrate just how arbitrary the very concept of "life" is.

So I think my definition is a good one, broad enough to encompass both in-silico and the organic versions. It bespeaks to algorithms, robots, viruses -- yes both computer and organic... anything that has complexity and the ability to adapt and evolve.


What is the difference between return and expected return? At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$.

If an agent is following a policy (which in itself is a probability distribution of choosing a next state $S_{t+1}$ from $S_{t}$), the agent wants to find the value at $S_{t}$ by calculating sort of "weighted average" of all the returns from $S_{t}.$ This is called the expected return.

Is my understanding correct?

Formally, the return (also known as the cumulative future discounted reward) can be defined as
$$
G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1},
$$
where $0 \leq \gamma \leq 1$ is the discount factor and $R_{i}$ is the reward at time step $i$. Here $G_t$ and $R_i$ are considered random variables (and r.v.s are usually denoted with capital letters, so I am using the notation used in the book Reinforcement Learning: An Introduction, 2nd edition).
The expected return is defined as
\begin{align}
v^\pi(s)
&= \mathbb{E}\left[G_t \mid S_t = s \right] \\
&= \mathbb{E}\left[\sum_{k=0}^\infty \gamma^k R_{t+k+1} \bigm\vert S_t = s \right]
\end{align}
In other words, the value of a state $s$ (associated with a policy $\pi$) is equal to the expectation of the return $G_t$ given that $S_t = s$, so $v^\pi(s)$ is defined as a conditional expectation. Note also that the expected value is usually defined with respect to a random variable, which is the case. Note also that $S_t$ is a random variable, while $s$ is a realization of this random variable.
A policy is not a probability distribution of choosing the next state. A stochastic policy is a family of a conditional probability distribution over actions given states. There are also deterministic policies. Have a look at this question What is the difference between a stochastic and a deterministic policy? for more details about the definition of stochastic and deterministic policies.

If an agent is following a policy, the agent wants to find the value at $S_{t}$ by calculating a sort of "weighted average" of all the returns from $S_{t}.$ This is called the expected return.

In the case of Monte Carlo Prediction, the value of a state associated with a specific policy, that is, the expected value of the return given a state is approximated with a finite (weighted) average. See e.g. What is the difference between First-Visit Monte-Carlo and Every-Visit Monte-Carlo Policy Evaluation?. Furthermore, note that the expectation of a discrete random variable is defined as a weighted average.


Identifying cars using deep learning I would like to use deep leaning for identifying cars; I want the system to predict wether an object is a car or not. How can I do that knowing that im still a beginner in the Deep Learning field ?
I am considering visual recognition. The system must recognize the car from anything else on the road.

That's definitely possible with Convolutional Neural Networks. You should follow Google's tutorial at:

  https://www.tensorflow.org/tutorials/deep_cnn#cifar-10_model

The tutorial will teach you how to build a deep learning model for classifying cars, trucks etc. It's an easy tutorial to follow.

Your model will build on images pixels. The classifier will know what a car looks like.


What are the current theories on the development of a conscious AI? What are the current theories on the development of a conscious AI? Is anyone even trying to develop a conscious AI?

Is it possible that consciousness is an emergent phenomenon, that is, once we put enough complexity into our system, it will become self-aware?

To answer this question, first we need to know why developing conscious AI is hard.
The main reason is that there is no mathematically or otherwise rigorous definition of consciousness.
Sure you have an idea of consciousness as you experience it and we can talk about philosophical zombies but it isn’t a tangible concept that can be broken down and worked on.
Moreover, the majority of current research in AI is primarily a pragmatic approach in that one is trying to construct a model that can perform well according to some desired cost function. This is a very very big and exciting field and encompasses many research problems and every new finding is based either on mathematical theory or empirical evidence of a new algorithm/model construction/etc.
Because of this, progress is based on and compared against previous progress as it is the scientific method.

So to answer your question, no one is trying to actually make a “conscious” AI because we don’t know what that word means yet, however that doesn’t stop people talking about it.


Comparing the derivation of the Deterministic Policy Gradient Theorem with the standard Policy Gradient Theorem I would like to understand the difference between the standard policy gradient theorem and the deterministic policy gradient theorem. These two theorem are quite different, although the only difference is whether the policy function is deterministic or stochastic.  I summarized the relevant steps of the theorems below. The policy function is $\pi$ which has parameters $\theta$.
Standard Policy Gradient
$$
\begin{aligned}
\dfrac{\partial V}{\partial \theta} &= \dfrac{\partial}{\partial \theta} \left[ \sum_a \pi(a|s) Q(a,s) \right]  \\
&= \sum_a \left[ \dfrac{\partial \pi(a|s)}{\partial \theta}  Q(a,s) + \pi(a|s) \dfrac{\partial Q(a,s)}{\partial \theta} \right] \\
&= \sum_a \left[ \dfrac{\partial \pi(a|s)}{\partial \theta}  Q(a,s) + \pi(a|s) \dfrac{\partial}{\partial \theta} \left[ R + \sum_{s'} \gamma p(s'|s,a) V(s') \right] \right]  \\
&= \sum_a \left[ \dfrac{\partial \pi(a|s)}{\partial \theta}  Q(a,s) + \pi(a|s) \gamma \sum_{s'} p(s'|s,a) \dfrac{\partial V(s') }{\partial \theta} \right]
\end{aligned}
$$
When one now expands next period's value function $V(s')$ again one can eventually reach the final policy gradient:
$$
\dfrac{\partial J}{\partial \theta} = \sum_s \rho(s) \sum_a \dfrac{\pi(a|s)}{\partial \theta}  Q(s,a)
$$
with $\rho$ being the stationary distribution. What I find particularly interesting is that there is no derivative of $R$ with respect to $\theta$ and also not of the probability distribution $p(s'|s,a)$ with respect to $\theta$. The derivation of the deterministic policy gradient theorem is different:
Deterministic Policy Gradient Theorem
$$
\begin{aligned}
\dfrac{\partial V}{\partial \theta} &= \dfrac{\partial}{\partial \theta} Q(\pi(s),s)  \\
&= \dfrac{\partial}{\partial \theta} \left[ R(s, \pi(s)) + \gamma \sum_{s'} p(s'|a,s) V(s') \right] \\
&= \dfrac{R(s, a)}{\partial a}\dfrac{\pi(s)}{\partial \theta} + \dfrac{\partial}{\partial \theta} \left[\gamma \sum_{s'} p(s'|a,s) V(s')   \right]  \\
&= \dfrac{R(s, a)}{\partial a}\dfrac{\pi(s)}{\partial \theta} + \gamma \sum_{s'}  \left[p(s'|\mu(s),s) \dfrac{V(s')}{\partial \theta} + \dfrac{\pi(s)}{\partial \theta} \dfrac{p(s'|s,a)}{\partial a} V(s')   \right]  \\
&= \dfrac{\pi(s)}{\partial \theta} \dfrac{\partial}{\partial a} \left[ R(s, a) +  p(s'|s,a) V(s') \right] +  \gamma  p(s'|\pi(s),s) \dfrac{V(s')}{\partial \theta}  \\
&= \dfrac{\pi(s)}{\partial \theta} \dfrac{\partial Q(s, a)}{\partial a} +  \gamma  p(s'|\pi(s),s) \dfrac{V(s')}{\partial \theta}  \\
\end{aligned}
$$
Again, one can obtain the finaly policy gradient by expanding next period's value function. The policy gradient is:
$$
\dfrac{\partial J}{\partial \theta} = \sum_s \rho(s) \dfrac{\pi(s)}{\partial \theta} \dfrac{\partial Q(s,a))}{\partial a}
$$
In contrast to the standard policy gradient, the equations contain derivatives of the reward function $R$ and the conditional probability $p(s'|s, a,)$ with respect to $a$.
Question
Why do the two theorems differ in their treatment of the derivatives of $R$ and the conditional probability? Does determinism in the policy function make such a difference for the derivatives?

In the policy gradient theorem, we don't need to write $r$ as a function of $a$ because the only time we explicitly 'see' $r$ is when we are taking the expectation with respect to the policy. For the first couple lines of the PG theorem we have
\begin{align}
\nabla v_\pi(s) &= \nabla \left[ \sum_a \pi(a|s) q_\pi (s,a) \right] \;, \\ &= \sum_a \left[ \nabla \pi(a|s) q_\pi(s,a) + \pi(a|s) \nabla\sum_{s',r} p(s',r|s,a)(r+ v_\pi(s')) \right] \; ;
\end{align}
you can see that we are taking expectation of $r$ with respect to the policy, so we don't need to write something like $r(s,\pi(a|s))$ (especially because this notation doesn't really make sense for a stochastic policy). This is why we don't need to take the derivative of $r$ with respect to the policy parameters. Now, the next line of the PG theorem is
$$\nabla v_\pi(s) = \sum_a \left[ \nabla \pi(a|s) q_\pi(s,a) + \pi(a|s)\sum_{s'} p(s'|s,a) \nabla v_\pi(s') \right] \; ;$$
so now we have an equation similar to the bellman equation in terms of the $\nabla v_\pi(s)$'s, so we can unroll this repeatedly meaning we never have to take an explicit derivative of the value function.
For the deterministic gradient, this is a bit different. In general we have
$$v_\pi(s) = \mathbb{E}_\pi[Q(s,a)] = \sum_a \pi(a|s) Q(s,a)\;,$$
so for a deterministic policy (denoted by $\pi(s)$ which represents the action taken in state $s$) this becomes
$$v_\pi(s) = Q(s,\pi(s))$$
because the deterministic policy has 0 probability for all actions except one, where it has probability one.
Now, in the deterministic policy gradient theorem we can write
$$\nabla v_\pi(s) = \nabla Q(s,\pi(s)) = \nabla \left(r(s, \pi(s)) + \sum_{s'} p(s'|s,a)v(s') \right)\;.$$
We have to write $r$ explicitly as a function of $s,a$ now because we are not taking an expectation with respect to the actions because we have a deterministic policy. Now, if you replace where I have written $\nabla$ with the notation you have used for the derivatives you will arrive at the same result and you'll see why you need to use the chain rule, which I believe you understand because your question was more why don't we use the chain rule for the normal policy gradient, which I have hopefully explained -- it is essentially because of how an expectation over the action space works with a deterministic policy vs. a stochastic policy.
Another way to think of this is as follows -- the term you're concerned with is obtained by expanding $\nabla q_\pi(s,a) = \nabla \sum_{s', r}p(s',r|s,a)(r(s,a) + v_\pi(s'))$. Because, by definition of the $Q$ function, we have conditioned on knowing $a,s$ then $a$ is completely independent of the policy in this scenario - we could even condition on an action that the policy would have 0 probability for - thus the derivative of $r$ with respect to the policy parameters is 0.
However, in the deterministic policy gradient we are taking $\nabla q_\pi(s, \pi(s)) = \nabla \left(r(s, \pi(s)) + \sum_{s'} p(s'|s,a) v_\pi(s')\right)$ -- here $r$ clearly depends on the policy parameters because the action taken was the deterministic action given by the policy in the state $s$, thus the derivative wrt the policy parameters is not necessarily 0!


In neural networks, what does the term depth generally mean? Is it

number of units in a layer
number of layers
overall complexity of the network (both 1 and 2)

Very good question, I have heard it refered to all the 3 ways you described. I am not sure if there is a purely objective answer so I will answer with what I understand by "depth" generally:

If speaking of model architecture: the number of layers of the model.
If speaking of CNN layers: the number of filters (not to be confused with the size of the kernel).
If speaking of RNN layers: the number of time steps (temporal "depth").

On the other hand I have never heard about depth regarding to FC layers. As I said the term "depth" is very subjective depending on the speakers. It should not be this way, and maybe theoretically is not, but I speak from what I heard in my experience.


Is there any artificial intelligence that possesses "concentration"? Humans can do multiple tasks at the same (e.g. reading while listening to music), but we memorize information from less focused sources with worse efficiency than we do from our main focus or task.

Do such things exist in the case of artificial intelligence? I doubt, for example, that neural networks have such characteristics, but I may be wrong.

Douglas Hofstadter's CopyCat architecture for solving letter-string analogy problems was deliberately engineered to maintain a semantically-informed notion of 'salience', i.e. given a variety of competing possibilities, tend to maintain interest in the one that is most compelling. Although the salience value of (part of) a solution is ultimately represented numerically, the means by which it determined is broadly intended to correspond (at least functionally) to the way 'selective attention' might operate in human cognition.


What is the meaning of "exploration" in reinforcement and supervised learning? While exploration is an integral part of reinforcement learning (RL), it does not pertain to supervised learning (SL) since the latter is already provided with the data set from the start.
That said, can't hyperparameter optimization (HO) in SL be considered as exploration? The more I think about this the more I'm confused as to what exploration really means. If it means exploring the environment in RL and exploring the model configurations via HO in SL, isn't its end goal "mathematically" identical in both cases?

In reinforcement learning, exploration has a specific meaning, which is in contrast with the meaning of exploitation, hence the so-called exploration-exploitation dilemma (or trade-off). You explore when you decide to visit states that you have not yet visited or to take actions you have not yet taken. On the other hand, you exploit when you decide to take actions that you have already taken and you know how much reward you can get. It's like in life: maybe you like cereals $A$, but you never tried cereals $B$, which could be tastier. What are you going to do: continue to eat cereals $A$ (exploitation) or maybe try once $B$ (exploration)? Maybe cereals $B$ are as tasty as $A$, but, in the long run, $B$ are healthier than $A$.
More concretely, recall that, in RL, the goal is to collect as much reward as you can. Let's suppose that you are in state $s$ and, in the past, when you were in that state $s$, you had already taken the action $a_1$, but not the other actions $a_2, a_3$ and $a_4$. The last time you took action $a_1$, you received a reward of $1$, which is a good thing, but what if you take action $a_2, a_3$ or $a_4$? Maybe you will get a higher reward, for example, $10$, which is better. So, you need to decide whether to choose again action $a_1$ (i.e. whether to exploit your current knowledge) or try another action that may lead to a higher (or smaller) reward (i.e. you explore the environment). The problem with exploration is that you don't know what's going to happen, i.e. you are risking if you already get a nice amount of reward if you take an action already taken, but sometimes exploration is the best thing to do, given that maybe the actions you have taken so far have not led to any good reward.
In hyper-parameter optimization, you do not need to collect any reward, unless you formulate your problem as a reinforcement learning problem (which is possible). The goal is to find the best set of hyper-parameters (e.g. the number of layers and neurons in each layer of the neural network) that performs well, typically, on the validation dataset. Once you have found a set of hyper-parameters, you usually do not talk about exploiting it, in the sense that you will not continually receive any type of reward if you use that set of hyper-parameters, unless you conceptually decide that this is the case, i.e., whenever you use that set of hyper-parameters you are exploiting that model to get good performance on the test sets that you have. You could also say that when you are searching for new sets of hyper-parameters you are exploring the search space, but, again, the distinction between exploitation and exploitation, in this case, is typically not made, but you can well talk about it.
It makes sense to talk about the exploitation-exploration trade-off when there is stochasticity involved, but in the case of the hyper-parameter optimization there may not be such a stochasticity, but it's usually a deterministic search, which you can, if you like, call exploration.


How exactly does nested cross-validation work? I have trouble understanding how nested cross-validation works - I understand the need for two loops (one for selecting the model, and another for training the selected model), but why are they nested?
From what I understood, we need to select the model before training it, which points toward non-nested loops.
Could someone please explain what's wrong (or right?) with my line of reasoning, and also explain nested cross-validation in greater detail? A representative example would be great.

"Selecting the model" in this case refers to selecting the hyperparameters of the model. The reason to use a nested CV is simply to avoid overfitting training data.
Consider the example in the link. First you like to select the best hyperparameters of your svm model by GridSearchCV(). This is done by 4-fold CV. Now the clf.best_score_ will be the mean cross-validated score of the best estimator (the model with the best hyperparameters). However now you used the same data for training and reporting the performance, although you used CV. Keep in mind that the folds are not independent. Therefore the hyperparameters might be too data specific, i.e. your generalization error estimates are too optimistic. Therefore we like to evaluate our final model performance outside / independent of the hyperparameter selection loop / process (the call cross_val_score()).
In the provided plot, you can clearly see that the reported performance by GridSearchCV() is most of the time better than the performance reported by cross_val_score().


Is there research that employs realistic models of neurons? Is there research that employs realistic models of neurons? Usually, the model of a neuron for a neural network is quite simple as opposed to the realistic neuron, which involves hundreds of proteins and millions of molecules (or even greater numbers). Is there research that draws implications from this reality and tries to design realistic models of neurons?

Particularly, recently, Rosehip neuron was discovered. Such neuron can be found only in human brain cells (and in no other species). Are there some implications for neural network design and operation that can be drawn by realistically modelling this Rosehip neuron?

It looks like you really have two questions here. I'll try to answer the first one, and you should think about making a separate question for the second.

There is research into using simulated models of biologically realistic neurons. While there are large projects like the Human Brain Project aimed at simulating human brains, there is also a lot of lower-level AI research. SPAWN is an interesting system that got a lot of press a few years ago, and has continued to be developed since then. It uses realistic neurons to simulate several brain-regions at once, creating a surprisingly general AI system that could perform many types of motor and vision tasks using the same basic design.


Are on-line backpropagation iterations perpendicular to the constraint? Raul Rojas' Neural Networks A Systematic Introduction, section 8.1.2 relates off-line backpropagation and on-line backpropagation with Gauss-Jacobi and Gauss-Seidel methods for finding the intersection of two lines.

What I can't understand is how the iterations of on-line backpropagation are perpendicular to the (current) constraint. More specifically, how is $\frac12(x_1w_1 + x_2w_2 - y)^2$'s gradient, $(x_1,x_2)$, normal to the constraint $x_1w_1 + x_2w_2 = y$?

Answer by Theo Bandit at maths stackexchange

  If you choose two points $(w_1, w_2), (v_1, v_2)$ along this line, then
  $$(x_1, x_2) \cdot ((w_1, w_2) - (v_1, v_2)) = x_1 w_1 + x_2 w_2 - (x_1 v_1 + x_2 v_2) = y - y = 0.$$
  That is, the direction $(x_1, x_2)$ is perpendicular to any vector lying along the line, i.e. $(x_1, x_2)$ is normal to the line.


What will Q-values look like in self-play tic-tac-toe? This corresponds to Exercise 1.1 of RLBook, and a discussion followed from here.
Considering two reward schemes-

Win = +1, Draw = 0, Loss = -1
Win = +1, Draw or Loss = 0

Can we say something about the optimal Q-values?

Chapter 1 of Sutton & Barto, doesn't introduce the full version Q learning, and you are probably not expected to explain the full distribution of values at that stage.

Probably what you are expected to notice is that the maximum Q values out of possible next states - after training/convergence - should represent the agent's best choice of move. What the actual optimal values are depends on how the opponent plays. In self-play it is possible to find optimal play for both players in the game, and thus the Q values represent true optimal play. However what "optimal play" means is dependent on the goals you have set the agent implied by the reward values.

Any move which leads to a guarantee that a player can force a win regardless of what the opponent does, would have a Q value of +1. If the agent or opponent can force a draw at best, then it will have the Q value of the draw, and if the opponent can force a win (i.e. the current agent would lose), then the move will have the Q value of a loss. This happens because the learning process copies best case values backwards from the end game states to earlier game states that lead to them.

In a game with two perfect players, and +1, 0, -1 reward system, then each player will on its turn only see the 0 and -1 moves available. That is because there is no way to force a win in tic-tac-toe, and the perfect opponent will always act to block winning moves. The best choice out of 0 or -1 is 0: each player, when acting under its value estimates, will force a draw. There will be states defined that have a value of +1, but they will never appear as a choice to either player.

What happens if you don't make a difference in rewards between drawing and losing? In the extreme case of having win +1, lose or draw 0 against a perfect opponent, then all of the agent's available Q values will always be 0. The agent would then be faced with no way to choose between defensive plays that force a draw and mistake plays that allow the opponent to win. In turn that means some chance that the opponent will win, even when the agent had learned optimal play.

When two agents learn through self-play using the +1, 0, 0 reward scheme it gets more complicated. That is because the opponent's behaviour is part of the Q value system. Some positions will have more opportunities for the opponent to make mistakes, and score more highly. A mistake that allows an opponent to force a win will actually score worse, because the opponent will not make mistakes once it has a sure route to a +1 score. So even though the agent apparently cannot tell the difference between a loss and a draw, it should still at least partially learn to avoid losses. In fact, without running the experiment, I am not sure whether this would be enough to still learn optimal play.

Intuitively, I think it would be possible for the +1, 0, 0 agent to still learn optimal play, although maybe more slowly than the +1, 0, -1 system, because any situation that gave an opponent a chance of winning would allow it to pick the move with best score, reducing the first agents score for a move that arrived there - and this difference will be backed up to earlier positions. However, the learning would become unstable as described above, as against a perfect opponent the difference disappears as all the best options are draws or losses, and the agent will start to make mistakes again.


Can transformer be better than RNN for online speech recognition? Does transformer have the potential to replace RNN end-to-end models for speech recognition for online speech recognition? This mainly depends on accuracy/latency and deploy cost, not training cost. Can transformer support low latency online use case and have comparable deploy cost and better result than RNN models?

Are there examples that transformer have better accuracy than RNN end-to-end model like RNN-transducer for speech recognition?
  Can transformer be used for online speech recognition which require low speech-end-to-result latency?
  Does transformer have the potential to replace RNN end-to-end models for speech recognition in most cases in the future? This may mainly depends on accuracy and deploy cost, not training cost.

You can check facebook results on wav2letter on all this:

https://ai.facebook.com/blog/online-speech-recognition-with-wav2letteranywhere/

https://research.fb.com/publications/scaling-up-online-speech-recognition-using-convnets/

Transformers definitely have a potential in speech especially when combined with faster computatoin methods (hashing) just like in NLP.

The problem with transformers is that you need a lot of GPUs to train them.


What is a good way to create an artificial self-recognition? Self-Recognition seems to be an item that designers are trying to integrate into artificial intelligence. Is there a generally recognized method of doing this in a machine, and how would one test the capacity - as in a Turing-Test?

Interesting question.  I don't think anybody knows a definite answer, but some rough-sketch ideas seem apparent.  Think about what it means to you to be "self aware".  You'll probably cite the way you "hear" your own thoughts in your head when you think about something.  One can speculate that inside the brain, the various centers that are responsible for hearing, vision, logic, etc. are connected so that as you form a thought, it's being "heard" by the hearing regions, even though it's purely internal instead of actual sound received at the ear.

So in AI terms, it seems likely that self-awareness will somehow involve taking the "thoughts" formed within the AI, and feeding them back into the AI so that it "hears" (or, more broadly, "senses") itself think.

There's this weirdly recursive aspect to all of this, which - interestingly enough - is something Douglas Hofstadter talked about a lot in some of his book, especially GEB.  He was probably onto something.


Disentangled VAE doesn't reconstruct accurate grids I am trying to implement the disentangled VAE model according to this link.
I want to understand the architecture of this model in order to customize it later. As infrastructure, I have a Linux kernel with 4 cores, 8 GB as the memory with only CPU support. but still, the model is taking hours to run.

Can anyone try to run this model and give me feedback? Is there any other simpler implementation of the disentangled VAE in python because I couldn't find any.

First of all , this is a deepmind's paper , so the computational level they operate on is much higher.
Can you share your code files , so that I can see how you interpreted miyosuda's code.
The disentangled representation has to be scaled down so that the model trains , and gets useful results in atleast a few hours.

4GB is too less RAM for training any kind of model. The best temporary solution I can suggest is to perform the python script of distangled VAE on Colab Colab Notebooks

The more permanent solution that can making your model training much easier given your system constraints , is to use GCP or Azure.

How do you plan to customize the architecture once you finish implementing distangled VAE.


Is a trained model in keras is saved with the weights for max accuracy? Does a model trained in keras (tensorflow backend) saves the weights with max accuracy and minimum losses or does it simply saves the weights from the last epoch? If it is the latter then how do I save the best weights?

The weights update by default, they do not get saved anywhere, they are just in the programs memory.

But you can choose to save the weights manually or define a callback to do so. You should checkout the ModelCheckpoint callback. It will give you the power to decide what criterion you want to save the weights on.


Are 2 GT 710 graphics cards enough for deep learning? I want to do some kind of complicated deep learning (specifically, https://github.com/Sentdex/pygta5/), but I don't have a decent graphics card. I am willing to get 2 gt 710s. Are they enough for this purpose?

No, it appears to take large amounts of screenshots to generate the images to be used for training.

Don't waste your money. I would say you could try training on Google's colab if you can upload the training data.


What does "stationary" mean in the context of reinforcement learning? I think I've seen the expressions "stationary data", "stationary dynamics" and "stationary policy", among others, in the context of reinforcement learning. What does it mean? I think stationary policy means that the policy does not depend on time, and only on state. But isn't that a unnecessary distinction? If the policy depends on time and not only on the state, then strictly speaking time should also be part of the state.

A stationary policy is a policy that does not change. Although strictly that is a time-dependent issue, that is not what the distinction refers to in reinforcement learning. It generally means that the policy is not being updated by a learning algorithm.

If you are working with a stationary policy in reinforcement learning (RL), typically that is because you are trying to learn its value function. Many RL techniques - including Monte Carlo, Temporal Difference, Dynamic Programming - can be used to evaluate a given policy, as well as used to search for a better or optimal policy.

Stationary dynamics refers to the environment, and is an assumption that the rules of the environment do not change over time. The rules of the environment are often represented as an MDP model, which consists of all the state transition probabilities and reward distributions. Reinforcement learning algorithms that work online can usually cope and adjust policies to match non-stationary environments, provided the changes do not happen too often, or enough learning/exploring time is allowed between more radical changes. Most RL algorithms have at least some online component, it is also important to keep exploring non-optimal actions in environments with this trait (in order to spot when they may become optimal).

Stationary data is not a RL-specific term, but also relates to the need for an online algorithm, or at least plans for discarding older data and re-training existing models over time. You might have non-stationary data in any ML, including supervised learning - prediction problems that work with data about people and their behaviour often have this issue as population norms change over timescales of months and years.


What is a good convergence criterion for Q-learning in a stochastic environment? I have a stochastic environment and I'm implementing a Q-table for the learning that happens on the environment. The code is shown below. In short, there are ten states (0, 1, 2,...,9), and three actions: 0, 1, and 2. The action 0 does nothing, action 1 subtracts 1 with a probability of 0.7, and action 2 adds 1 with a probability of 0.7. We get a reward of 1 when we are in state 5, and 0 otherwise.
import numpy as np
import matplotlib.pyplot as plt

def reward(s_dash):
    if s_dash == 5:
        return 1
    else:
        return 0
states = range(10)
Q = np.zeros((len(states),3))
Q_previous = np.zeros((len(states),3))
episodes = 2000
trials = 100
alpha = 0.1
decay = 0.995
gamma = 0.9
ls_av = []
ls = []
for episode in range(episodes):
    print(episode)
    s = np.random.choice(states)
    eps = 1
    for i in range(trials):
        eps *= decay
        p = np.random.random()
        if p < eps:
            a = np.random.randint(0,3)
        else:
            a = np.argmax(Q[s, :])

        if a == 0:
            s_dash = s
        elif a == 1:
            if p >= 0.7:
                s_dash = max(s-1, 0)
            else:
                s_dash = s
        else:
            if p >= 0.7:
                s_dash = min(s+1, 9)
            else:
                s_dash = s
        r = reward(s_dash)
        Q[s][a] = (1-alpha)*Q[s][a] + alpha*(r + gamma*np.max(Q[s_dash]))
        s = s_dash
    ls.append(np.max(abs(Q - Q_previous)))
    Q_previous = np.copy(Q)
print(Q)
for i in range(10):
    print(i, np.argmax(Q[i, :]))
plt.plot(ls)
plt.show()

When I plot the absolute value of the maximum change in the Q-table at the end of each episode, I get the following, which indicates that the Q-table is constantly being updated.

However, I see that when I print out the action with the max Q-value for each state, it shows what I expect to be the optimal policy. For each state, the best action is given as shown below:
(0, 2)
(1, 2)
(2, 2)
(3, 2)
(4, 2)
(5, 0)
(6, 1)
(7, 1)
(8, 1)
(9, 1)

My question is: why do I not have convergence in the Q-table? If I had a stochastic environment for which I didn't know before-hand what the optimal policy is, how will I be able to judge if I need to stop training when the Q-table isn't converging?

To obtain guarantees of convergence for Q table values, you need to decay the learning rate, $\alpha$, at a suitable rate. Too fast and convergence will be to inaccurate values. Too slow and convergence never happens.
For sticking with theoretical guarantees, the learning rate decay process should generally follow the rule that $\sum_t \alpha_t = \infty$ but $\sum_t \alpha_t^2 \ne \infty$ - an example of a learning rate schedule that does this is $\alpha_t = \frac{1}{t}$, although in practice that specific choice could lead to very slow convergence.
Choosing a good starting $\alpha$ and a good decay schedule will depend on the problem, and you may want to base it on experience with similar problems. However, it is not that common to need to gurantee convergence of action values in value-based reinforcement learning. In control problems you often care more about have finding an optimal policy than about perfectly accurate action values. Further to that, many interesting control problems are too complex to solve perfectly in tabular form, so you expect some approximation. It seems relatively common just to pick a learning rate for the problem and stick with it.
If you make your learning rate lower, the Q table will converge to more stable Q values, but possibly at the expense of taking longer to converge on the optimal policy.


How to represent the weights of a neural network as binary strings for a genetic algorithm? I want to train my neural network by evolution, that is I want to recombine the weights of the best performing neural networks in each evolution cycle or generation. My initial instinct was to represent weights as they are, which is variable of type double, and either

swap the weights between the two parent network

generate a random number between the two weights

However, what I need is to represent the weights as a binary string, and then, as usual, carry out the crossover on the string.
So, how can I take my array of weights (of floating-point values or doubles) and convert that into binary string? Should the chromosome contain an array of strings where each string represents a single weight?

I would first say consider the advice of Thomas W in the comment above and think about whether you really need to discretize your variables. I'd also question the wisdom of training a reasonably sized network with a GA instead of a dedicated neural net training algorithm that's very likely to exhibit much better performance. However, assuming you really do need to do this, the basic approach is generally to do the following.

Decide on your range and precision. For the sake of simplicity, I'm going to assume you use say $L=16$ bits to represent every weight. You can have some variables encoded at a higher precision than others -- you're in charge of writing the encode/decode functions, so it's entirely up to you, but here, I'm assuming you don't. Your encoding will now be a vector of length $16n$, where $n$ is the number of weights in the network. You can use a string if you like, but it's probably a net loss of efficiency, as mostly what you'll be doing is converting those bits to integers so that you can do arithmetic necessary to decode the encoded weights, so I'd just make it a vector of $16n$ integers.

For the range, you just need the minimum and maximum values you're going to allow. Again, for simplicity, I'm going to say the values range between $-10$ and $10$, denoted as $R_{\min}=-10$ and $R_{\max}=10$.

Your GA then just manipulates the $0$s and $1$s. To evaluate fitness, you have to decode each of those $16$-bit strings into a double between $-10$ and $10$. With $16$ bits, you have $2^{16}=16536$ possible values. The total range of output values is $10-(-10)=20$. $20/16536=0.0012$. Let $0000000000000000$ be the smallest possible string, and map that onto the value $-10$. Now, every increment to that binary string just adds $0.0012$ to the output value.

Putting all this together, you have $w_i = (R_{\max} - R_{\min})/2^L + R_{\min}$. Do that for each block of $16$ bits giving you each weight in the network. Run your network on your training data and use whatever accuracy metric as the fitness function for the GA.

For the GA itself now, you don't really have to do anything special. The GA can just see an arbitrary string of $16n$ bits. You can do crossover and mutation on just the apparently meaningless string of bits.

Your question was asking about going from a vector of weights to a vector of strings, which is the opposite conversion from that described above. Generally, for a GA, you don't need a phenotype->genotype conversion. You just generate genotypes using genetic operators and use the genotype->phenotype conversion to evaluate candidate solutions. However, there are some reasons why you might want to go the other way. For instance, you might want to seed the population with a known set of weights, and so you'd need the ability to convert those weights to binary.
Basically, in that case, just invert that conversion function. The idea is to convert the floating-point weight between $R_{\min}$ and $R_{\max}$ into the integer that's closest to the same distance into the range when mapped to $0$ to $2^{L}-1$. For example, if the weight is $-10$, then it's the smallest possible value. It maps onto the smallest possible value between $0$ and $2^{L}-1$, namely $0$. Similarly, a weight of $0$ maps onto $(2^{L}-1)/2$, because it's halfway between the min and max of both ranges. If you play around with it a bit, you should see that the expression here is given by $\big \lfloor \frac{w_i - R_\min}{R_\max - R_\min} 2^L \big \rfloor$. That gives me an integer in the range $0$ to $2^L$. From there, just do a standard binary encoding (or gray code, whatever) to get into a bit string you can manipulate.


Which humanoid robots have open-source software (or hardware)? Apart from iCub, whose open-source software can be found at https://github.com/robotology/icub-main, what other humanoid robots have open-source and freely accessible software (or hardware)?

There are quite a few ROS (Robot Operating System) based humanoid robots. Not all of them will be open-source down to the firmware level but most will at least be to the control level. I'm not aware of any that have open-source hardware as well.

Some examples include:

Talos
Reem-C
Nao
OP3
Thormang3


How to transfer declarative knowledge into neural networks Humans learn facts about the world like "most A are B" by own experience and by being told so (by other people or texts). The systems and mechanisms of storage and usage of such facts (by an "experience system" and a "declarative system") are presumably quite different and may have to do with "episodic memory" and "semantic memory". Nevertheless at least in the human brain the common currency are synaptic weights, and it would be quite interesting to know how these two systems cooperate.
I assume that neural learning is mainly concerned with "learning by own experience" (= training data + annotations), be it supervised or unsupervised learning. I wonder which approaches there are that allow a neural network to "learn by being told". One brute force approach might be to translate a declarative statement like "most A are B" into a set of synthetic training data, but that's definitely not how it works for humans.

One way to look at intelligence is it's the way to compress the universe. That means we have a short mental representation of meaningful concepts.
For example, if I would say "there is a red swan in your building, it's dangerous and can kill you", you already have concepts of "red", "swan", "danger" and this easy allows you to add the bird to your classifier "should I beware it or not". (and you probably have tried to imagine the red swan by now)
There are similar properties in deep networks when deeper layers could respond to more abstract representation concepts. For example, if you have a convolutional network classifying the faces, the first layers would detect simple shapes like lines and circles, then more complex shapes and then parts of the face like eyes, nose and ears (see example here)
Now, let's imagine you have a classification task "is the thing dangerous or not". You have a trained convolutional model and it has "red" and "swan" abstract features on some level. You could freeze all the preliminary weights and make a single backpropagation with a high learning rate only for the subnetwork which is of interest to you.
That could work in theory, but the key challenge here is that most of the neural network representations are not interpretable at all and you would need to solve this problem before (there is a whole research direction with workshops and conferences on the topic with varying success)


Unclear definition of a "leaf" and diverging UTC values in the Monte Carlo Tree Search I have two questions regarding the Selection and Expansion steps in the Monte Carlo Tree Search Algorithm. In order to state the questions, I recall the algorithm that I believe is the one most commonly associated with the MCTS. It is described as a repeated iteration of the following four steps:

Selection: Start from root R. Choose a leaf node L by iteration of some choice algorithm that determines which child node to choose at each branching. UTC being a prominent choice.
Expansion: Create one or more offspring nodes, unless L is terminal. Choose one of them, say C.
Simulation: Play the game from C, randomly or according to some heuristic.
Backpropagation: Update rewards and number of simulations for each node on the branch R-->C.

When implementing this algorithm by myself I was unclear about the following interpretation of step 1 and 2:
Q1. When expanding the choices at the leaf node L, do I expand all, a few or just one child? If I expand all, then the tree grows exponentially large on each MCTS step, I suspect. When I expand one or a few, then either the selection step itself becomes problematic or the term leaf does. The first problem arises, because after the expansion step the node L is no longer a leaf and can never be chosen again during the selection step and in turn all the children that were not expanded will never be probed. If, however, the node L keeps being a leaf node, contrary to graph-theoretic nomenclature, then during the selection step one would need to check at each node, whether there are non-expanded child-nodes. According to which algorithm should one then choose whether to continue down the tree or expand at this non-leaf "leaf" some more yet unexpanded children?
Q2. Related to the first question, but slightly more in the direction of the exploitation-exploration part of the selection, I am puzzled about the UTC selection step, which again raises issues for each of the above-mentioned expansion methods: In case that a few or all child-nodes are chosen during expansion at the leaf, one is faced with the problem that some of those nodes will not be simulated in that MCTS step and subsequently will have a diverging UTC value $w_i/n_i + c \sqrt{\frac{\ln{N_i}}{n_i}}\to \infty$, since $n_i\to 0$. On the other hand, in case that only one child is chosen, we are facing the issue that no UTC value can be assigned to the "unborn" children on the way. In other words, one cannot use UTC to decide whether to choose a child node according to UTC at each branching or to expand the tree at that node (since all nodes within the tree may have some unexpanded child nodes).

Q1. When expanding the choices at the leaf node L, do I expand all, a few or just one child?

Expanding all nodes or expanding just one node are both possible. There are different advantages and disadvantages. The obvious disadvantage of immediately expanding them all is that your memory usage will grow more quickly. I suppose that the primary advantage is that you no longer need to keep track separately of "actions that are legal but for which I did not yet create child nodes" and "already-created nodes", which I guess might sometimes lead to better computational efficiency (especially when memory isn't a concern for you, for instance if you do relatively few iterations anyway or if you have a huge amount of memory available).
In the more "modern" variants of MCTS that also use trained policy networks (like all the work inspired by AlphaGo / AlphaGo Zero / AlphaZero), it typically makes most sense to expand all children at once, because the trained network will immediately give you policy values for all children anyway, so then you can prime them all at once with those policy head outputs and also immediately start using all of those values.
In the case where you choose to expand only one child at a time, indeed the terminology of "leaf" nodes becomes confusing / incorrect. I never really like the term "leaf" node in the context of MCTS anyway; they're really not leaf nodes (unless they happen to represent terminal game states), they're just nodes for which did not yet choose to instantiate all the child nodes. It's a bit more verbose, but I prefer referring to them as "nodes that have not yet been fully expanded". That would change the description of the Selection phase to something more like:

Selection: Start from root R. Choose a node L that has not yet been fully expanded (or a terminal node) by iteration of some choice algorithm that determines which child node to choose at each branching.

In case that a few or all child-nodes are chosen during expansion at the leaf, one is faced with the problem that some of those nodes will not be simulated in that MCTS step and subsequently will have a diverging UTC value $w_i/n_i + c \sqrt{\frac{\ln{N_i}}{n_i}}\to \infty$, since $n_i\to 0$.

That's true, but I don't see that as a problem. Suppose we're at a non-fully-expanded node; a node where maybe some legal actions already have corresponding child nodes, or maybe none do, but at least some legal actions still don't have corresponding child nodes. We can view these as having a visit count of $0$, which we can view as leading to a UCB1 value of $\infty$. Since UCT picks nodes (or actions) according to an $\arg\max$ over the UCB1 values, we can simply think of this as UCT always preferring to pick actions that we did not yet expand over actions that we've already expanded and created a child node for. This leads to an implementation where we'll only re-visit a node that we've already visited before if we have already visited each of its possible siblings at least once too.


Why is it illegal for Google's autonomous car to drive on the road by itself? If we have well designed autonomous AI vehicles, then why won't the USA government allow it to be witnessed in the public?

The question is based on a false fact: in Michigan, it is currently legal (under certain conditions described here ) for an autonomous car to operate without a driver.

The reason that the federal government has not enacted any direct legislation (although they have enacted guidelines) on autonomous cars is because it is still a developing technology (as great as those Cruise videos look, I wouldn't trust most companies working on the technology to have driverless cars on the road without operators) and that it arguably falls under the state governments jurisdictions. Some companies have appealed to the House to pass national legislation to allow autonomous testing (GM, Toyota, Lyft), but nothing has come of it yet.


Have evolutionary algorithms been used for engineering design? Recently, I've been looking recently into what uses AI - specifically evolutionary algorithms - may have in automating engineering design.  For a long time, there have been algorithms that solve constraint satisfaction problems, and, to me, it makes sense to consider engineering problems as a superset of constraint satisfaction problems.  In spite of this, I haven't been able to find any cases of engineering design being automated other than a couple of cases of genetic algorithms being used to optimise structural members.
So my question is, why can't I find any examples?
The first thing that springs to mind is that I just haven't been looking hard enough - if this is the case, could anyone point me in the right direction?
The other obvious answer is that it isn't a widely researched area - if so, why not?  Is it just due to lack of interest or are there technical hurdles (abstraction, complex logic & reasoning, etc.) that make this a much more difficult problem than computer vision, games, and so on?

Examples of GA implementations:

NASA’s spacecraft evolved antenna
Solution for TSP with a big number of connections. Here is an example with Python and here with C#.
From my experience, GA can be even used for a training of small artificial neural networks, but with less efficiency than with traditional deep learning approaches

Also, I heard about successful projects with using GA for scheduling, packing and  logistics
Therefore, everything, that can be represented in a form of digital chromosomes (can be serialized to and deserialized  back from a sequence of bits) and we can define a function to evaluate its efficiency – fitness or reward function can be optimized by using of GA


How can thought vectors be used outside of an Artificial Neural Network (ANN) context? We hear a lot today about how thought vectors are the Next Big Thing in AI, and how they serve as the underlying representation of thought/knowledge in ANN's.  But how can one use thought vectors in other regimes, especially including symbolic logic / GOFAI?  Could thought vectors be the "substrate" that binds together probabilistic approaches to AI and approaches that are rooted in logic?

I'll take a shot at answering this, though I'm no expert in Neural Nets or Deep Learning.

Given that practical thought vectors (TVs) don't yet exist, and may be impractical or impossible, I think answering your question will require a lot of conjecture and speculation.  So here goes...

For thought vectors to be useful in or outside NNs, the vector values will have to be normalized, probably using the local context of the application problem's 'frame'.  Without a NN to create new baseline vector values (weights) and to normalize them to match each new context, any non-NN mechanistic alternative means of employing TVs will somehow have to fill that void.

Could vector values created by NNs be used by an alternative technique?  Could that technique also normalize them?  Sure.  We're just talking about turing-computable functions performed by NNs.  If NNs aren't magic, then there should exist other means to compute the same results -- creating or editing, or employing TVs.

What might such an alternative to NNs be?  Well, if its to shape vector weights, I suspect it too will have to learn those values through statistical iteration and feedback (as opposed to logical induction, say).  I doubt such a mechanism exists yet, since it'd probably resemble NNs in sufficiently many ways that, thus far, it would have seemed too derivative of NNs to gain acceptance as sufficiently novel.  Of course to be as powerful as deep nets, it too would have to propagate learning weights both forward and backward without incurring much error.  Not an easy thing to accomplish.

Less ambitiously, could TVs be simply interpreted by another technique usefully?  I think so.  I can see several existing techniques, like decision trees or even expert systems, importing thought vectors and being shaped by them, and then function in accordance.  But could these same techniques create or revise TVs usefully?  Beyond a trivial extent, I'm doubtful.  I think TVs are too complex a knowledge representation format for most general learning methods to both use and create/modify them, unless they employ an iterative statistical and feedback-based learning process, like those of NNs, which would allow novel and complex features to be learned and integrated into the vectors.


What are some good alternatives to U-Net for biomedical image segmentation? Soon I will be working on biomedical image segmentation (microscopy images). There will be a small amount of data (a few dozens at best).

Is there a neural network, that can compete with U-Net, in this case?

I've spent the last few hours searching through scientific articles that are dealing with this topic, but haven't found a clear answer and I would like to know what other possibilities are. The best answers I found are that I could consider using ResU-Net (R2U-Net), SegNet, X-Net and backing techniques (article).

Any ideas (with evidence, not necessarily)?

Hey i am working on my Bachelor thesis at the moment and use UNET in combination with a GAN for image segmentation. I spend the last 5 months on that, so on my tests, the new approach of januar 2020, called Multires-UNET is quite a good choice for more texture orientated segmentation. I use the current github implementation. Its quite nice, maybe you notice that you can easyly tweak the number of parameters with "alpha" in the implementation, to scale the multiple Resnets in the Unet structure.

I tried also other segmentation networks like Mask_RCNN with different backbones or tried to construct various types of CAE on my own, but always had to come back to a UNET like structure. Same goes for ResU-NET and R2U, the multires one worked better for my purposes cause i didnt need any kind of LSTM modules.

Some examples which may clarify the difference in performance on a specific task:

Original Image:

Ground-Truth:

Classical UNET++ (Unet with skip-connections)(2.5 million parameters) more parameters (wide) didnt change the result.

Multires-UNET (alpha=1.67, think was about 7 million parameters)

Can you show some of ur microscopy images, how complex is ur task and what do you want to segment ?


Video summarization similar to Summe's TextRank We have the popular TextRank API which given a text, ranks keywords and can apply summarization given a predefined text length.

I am wondering if there is a similar tool for video summarization. Maybe a library, a deep model or ML-based tool that given a video file and a length, it ranks frames, or video scenes/shots. I'd like to generate a short summary of a video with visual features.

From "Deep learning-based video summarization — A detailed exploration" by Surya Remanan:
Video summarization can be considered as the process of distilling a raw video into a more compact form without losing much information. In a general video summarization system, image features of video frames are extracted, and then the most representative frames are selected through analyzing the visual variations among visual features. This is done either by taking a holistic view of the entire video or by identifying the local differentiation among the adjacent frames. Most of those attempts rely on global features such as colour, texture, motion information, etc. Clustering techniques are also used for summarization. Video summarization can be categorized into two forms:

Static video summarization (keyframing) and
Dynamic video summarization (video skimming)

Static video summaries are composed of a set of keyframes extracted from the original video, while dynamic video summaries are composed of a set of shots and are produced taking into account the similarity or domain-specific relationships among all video shots.
Following is an attention-based video summarization model - PyTorch implementation of the ACCV 2018-AIU2018 paper Video Summarization with Attention
There is a video summarisation focused models using reinforcement learning -  Unsupervised video summarization with deep reinforcement learning (Theano)
There is an LSTM - GAN based approach to video summarization - video summarization lstm-gan pytorch implementation
Microsoft Bing Search has come up with a video summarization technique using thumbnails. Intelligent Search: Video summarization using machine learning


Can neural networks be used to find features importance? I am wondering if I can use neural networks to find features importances in similar manner as it can be done for random forests or decision trees and if so, how to do it?

I would like to use it on tabular time series data (not images). The reason why I want to find importances on neural networks not on decision trees is that NNs are more complicated algorithms so using NNs might point out some correlations that are not seen by simple algorithms and I need to know what features are found to be more useful with that complicated correlations.

I am not sure if I made it clear enough, please let me know if I have to explain something more.

This should be possible, considering universal approximation theorem you should be able to build a ann that approximates features that gives the most likely best feature set for a different net to train on. I would us a rnn for with a softmax output layer that ranks features by performance.

You can find a good explanation of softmax here: https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/softmax
basically it will assign probability values for each output node with all of these values adding up to 1.0


What is the most appropriate ML algorithm for creating recommendations I am trying to find the best algorithm to create a list of recommendations for a user based on the interests of all other users.
Say I have a list of of samples:
$samples = [
    ['hot dog', 'big mac', 'whopper'],
    ['hot dog', 'big mac'],
    ['hot dog', 'whopper'],
    ['big mac', 'dave single'],
    ['whopper', 'mcnuggets', 'mcchicken'],
    ['mcchicken', 'original chicken sandwich'],
    ['mcchicken', 'mcrib']
];

And we will say each array in the sample list is unique user's food preferences.
Let's say now I have a user with this food preference:
['hot dog', 'mcchicken']

I want to be able to recommend to this user other foods that other users have in their preferences.
So in the simplest terms, it should return:
['whopper', 'big mac', 'original chicken sandwich', 'mcrib', 'mcnuggets']

Obviously I will also introduce other variables such as how each user rates each item in their preference list and also the percentage of users that need to have that item in order to use their other food items as recommendations.
But I would like to find the best algorithm to start working on it.
At first I thought Apriori might be my best guess, but I wasn't having luck once I introduced multiple items.

You can use Collaborative Filtering, and specifically its memory based approach.
The problem that you have discussed in the question should probably be solved using User-Item collaborative filtering which will calculate similarity between users and then recommend the item. The similarity can be be calculated using cosine similarity or Pearson's similarity formulae.
The best thing about this approach is that no training is required here but if you wish to have very large data, this approach's performance decreases.


Content analysis tools We want to figure out the connection between people, based on their speech.
Assume, that a conversation is a poetry with lines belongs to characters. There are a lot of poetries and the lines are mixed. Now we want to define a conversation to which each line belongs to.
We assume, that people in a conversation use similar words (their dictionaries should be similar). It means that there is a correlation between words of person A and words belongs to person B, and we could detect a connection between the peoples which had a conversation. What are the next steps for content understanding after NLP?
Can some of you advise us about the field of study and tools/libraries, which are dealing with content processing? Maybe, some of you know good articles or online resources, which can help us to dive into this field.

These are really two questions -
1. How to assign lines in a dialogue to particular speakers
2. How to analyse the content of a speaker's utterances

I will attempt to answer the first one.

In discourse analysis (the field that studies conversations), there is the concept of adjacency pairs, linked utterances which typically follow on to each other. A well-formed conversation can be segmented into a series of adjacency pairs, such as greeting/greeting, question/answer/feedback (that's more of an 'adjacency triple'), statement/comment, farewell/farewell. Sometimes it's a bit more tricky, such as question/counter-question/answer/answer:

young person: What beers do you have? (Q)
bartender: How old are you? (Counter-Q)
young person: 21 (A)
bartender: We have a lager or a nice IPA. (A)

I don't know if this has been tried before, but it should be comparatively easy to identify the types of speech acts in a conversation and join them up to form relevant adjacency pairs. This would be a lot more reliable than looking at vocabulary, especially since conversations mostly use higher-frequency words which are part of everyone's active vocabulary.

You would need to take some conversations, classify the individual lines as speech acts (building up an inventory as you do that), and then identify adjacency pairs. This will give you the structure of the conversation, and should also allow you to assign speakers to utterances (assuming a dialogue, ie only two participants). Note that it is not always purely form that determines the speech act: How are you? is a question on the surface, but its function is part of a greeting sequence.

This conversational structure could also assist you in looking at the second part, the content. If you know someone asks questions, and the other participant answers them, then you know exactly where to look for salient utterances that might give you clues as to the topic of the conversation. At least it should enable you to ignore the 'housekeeping' utterances like greetings and farewells etc.

Your second question is hard to answer without knowing more about the purpose of your project. I would rephrase this and ask it as a separate question. People would need to know what kind of conversations they are, and what exactly you want to get out of it.

Oh, and I wouldn't compare conversations with poetry. Completely different things!  :)


How can I use one neural network for both players in Alpha Zero (Connect 4)? First of all, it is great to have found this community!

I am currently implementing my own Alpha Zero clone on Connect4. However, I have a mental barrier I cannot overcome.

How can I use one neural network for both players? I do not understand what the input should be.

Do I just put in the board position ($6 \times 7$) and let's say Player1's pieces on the board are represented as $-1$, empty board as $0$ and Player2's pieces as $1$?

To me, that seems the most efficient. But then, in the backpropagation, I feel like this cannot be working. If I update the same network for both players (which Alpha Zero does), don't I try to optimize Player1 and Player 2 at the same time?

I just can't get my head around it. 2 Neural networks, each for one player is understandable for me. But one network? I don't understand how to backpropagate? Should I just flip my "z" (the result of the game) every time I go one layer backward? Is that all there is to using one network?

I hope I made this clear enough. I am quite confused, I tried my best.

Thank you for reading this!

Let's define your problem from another point of view. Let's say that in this RL problem you have two agents (agent1 and agent2) that compete with each other in order to accomplish their own goal, i.e., wining connect4 game.

Therefore, we could say that from agent1's point of view, he is player1 and the player2 is agent2. The same way, from agent2's point of view, he is player1 and the player2 is agent1.

Here, we can use the same definition as you proposed:

"let's say player1's pieces on the board are represented as -1, empty board as 0 and player2's pieces as 1"

When it is the turn for agent1, he will receive an observation (representing the board configuration) from the environment. In this case, agent1's pieces are represented as -1, empty board as 0 and for player2's (agent2's) pieces as 1. Then, you could use the neural network to compute, e.g., the next action for agent1.

When it is the turn for agent2, he will receive another observation from the environment, but this time you represent its own pieces as -1, while the agent1's (player2's from agent2's point of view) as 1.

With this approach, the neural network will always receives as an input a representation in which the pieces 'it' controls are -1, while its rival's pieces are 1 and empty space 0. From the neural network's point of view, he does not care about which agent is playing as the actual observation is indistinguishable to calculate the next action.

Therefore, you will have to generate two different observations, in which you invert the sign of the ones on it, depending on which turn it is (agent1's or agent2's turn).

For example:


What is the best .net programming language for artificial intelligence programming? I know that every program has some positive and negative points, and I know maybe .net programming languages are not the best for AI programming.

But I prefer .net programming languages because of my experiences and would like to know for an AI program which one is better, C or C++ or C# and or VB ?

Which one of this languages is faster and more stable when running different queries and for self learning ?

To make a summary, i think C++ is the best for AI programming in .net and also C# can be used in some projects, Python as recommended by others is not an option on my view !

because :

It's not a complex language itself and for every single move you need to find a library and import it to your project (most of the library are out of date and or not working with new released Python versions) and that's why people say it is an easy language to learn and use ! (If you start to create library yourself, this language could be the hardest language in the world !)
You do not create a program yourself by using those library for every single option on your project (it's just like a Lego game)
I'm not so sure in this, but i think it's a cheap programming language because i couldn't find any good program created by this language !

If you're talking about pure speed, C will get you there if you really know C and operating systems, etc. C++ is nicer in terms of user friendliness, and won't be much slower. I don't know much about VB but I don't see many benefits.

Please clarify at least generally what the AI program is about. It is extremely difficult to answer the generalized question "What programming language is best for AI"? if you know what I mean. (  I vote python :^]  )

To add: Any programming language of those listed is "stable" if you write things correctly, but C++ and its great IDE's will help you to that point much more nicely than C will. C, to fully utilize it's potential, requires much fiddling with delicate and precise systems. It'll go that little bit faster, as the cost of being less stable in the practical, " uh oh, now I need to troubleshoot this" sense.


Why does DQN require two different networks? I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate action and the second network predicts the target Q values for finding the Bellman error.

Why can we not just make one single network that simply predicts the Q value and use it for both the cases? My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one. Is this the only reason? Am I missing something?

My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one.

It has no real impact on computation time, other than a slight increase (due to extra memory used by two networks). You could cache results of the target network I suppose, but it probably would not be worth it for most environments, and I have not seen an implementation which does that.

  Am I missing something?

It is to do with stability of the Q-learning algorithm when using function approximation (i.e. the neural network). Using a separate target network, updated every so many steps with a copy of the latest learned parameters, helps keep runaway bias from bootstrapping from dominating the system numerically, causing the estimated Q values to diverge.

Imagine one of the data points (at $s, a, r, s'$) causes a currently poor over-estimate for $q(s', a')$ to get worse. Maybe $s', a'$ has not even been visited yet, or the values of $r$ seen so far is higher than average, just by chance. If a sample of $(s, a)$ cropped up multiple times in experience replay, it would get worse again each time, because the update to $q(s,a)$ is based on the TD target $r + \text{max}_{a'} q(s',a')$. Fixing the target network limits the damage that such over-estimates can do, giving the learning network time to converge and lose more of its initial bias.

In this respect, using a separate target network has a very similar purpose to experience replay. It stabilises an algorithm that otherwise has problems converging.

It is also possible to have DQN with "double learning" to address a separate issue: Maximisation bias. In that case you may see DQN implementations with 4 neural networks.


Is reinforcement learning only about determining the value function? I started reading some reinforcement learning literature, and it seems to me that all approaches to solving reinforcement learning problems are about finding the value function (state-value function or action-state value function).
Are there any algorithms or methods that do not try to calculate the value function but try to solve a reinforcement learning problem differently?
My question arose because I was not convinced that there is no better approach than finding the value functions. I am aware that given the value function we can define an optimal policy, but are there not other ways to find such an optimal policy?
Also, is the reason why I don't encounter any non value-based methods that these are just less successful?

There are many algorithms that are not based on finding a value function. The most common ones are policy gradients. These methods attempt to map states to actions through a neural network. They learn the optimal policy directly, not through a value function.

The important part of the image is when Model-Free RL splits into  Policy Optimization (which includes policy gradients) and Q-Learning. Later you can see the two sections coming back together in algorithms that are a mix of both techniques. Even the bottom three methods in policy optimization involve some form of learning a value function. The best and most advanced algorithms use value function learning and policy optimization. The value function is only for training. Then when the agent is tested, it only uses the policy.
The most likely reason you have only heard of value function methods is because policy gradients are more complicated. There are many algorithms more advanced than ones that only use value functions and policy gradients can learn to operate in continuous actions spaces (an action can be between -1 and 1, like when moving a robot arm) while value functions can only operate with discrete action spaces (move 1 right or 1 left).
Summary: Yes, there are other methods that learn the optimal policy without a value function. The best algorithms use both types of reinforcement learning.
The SpinningUp website has a lot of information about reinforcement learning algorithms and implementations. You can learn more about direct policy optimization there. That is also where I got the image from.
This answer is specific to the most common types of Model-Free RL. There are other algorithms related to the RL problem that do not learn value functions, like inverse reinforcement learning and imitation reinforcement learning.


What is a "logit probability"? DeepMind's paper "Mastering the game of Go without human knowledge" states in its "Methods" section on its "Neural network architecture" that the output layer of AlphaGo Zero's policy head is "A fully connected linear layer that outputs a vector of size 19^2+1=362, corresponding to the logit probabilities for all intersections and the pass move" (emphasis mine).  I am self-trained regarding neural networks, and I have never heard of a "logit probability" before this paper.  I have not been able by searching and reading to figure out what it means.  In fact, the Wikipedia page on logit seems to make the term a contradiction.  A logit can be converted into a probability using the equation $p=\frac{e^l}{e^l+1}$, and a probability can be converted into a logit using the equation $l=\ln{\frac{p}{1-p}}$, so the two cannot be the same.  The neural network configuration for Leela Zero, which is supposed to have a nearly identical architecture to that described in the paper, seems to indicate that the fully connected layer described in the above quote needs to be followed with a softmax layer to generate probabilities (though I am absolutely new to Caffe and might not be interpreting the definitions of "p_ip1" and "loss_move" correctly).  The AlphaGo Zero cheat sheet, which is otherwise very helpful, simply echoes the phrase "logit probability" as though this is a well-known concept.  I have seen several websites that refer to "logits" on their own (such as this one), but this is not enough to satisfy me that "logit probability" must mean "a probability generated by passing a logit vector through the softmax function".

What is a logit probability?  What sources can I read to help me understand this concept better?

Indeed I haven't seen the term "logit probability" used in many places other than that specific paper. So, I cannot really comment on why they're using that term / where it comes from / if anyone else uses it, but I can confirm that what they mean by "logit probability" is basically the same thing that is more commonly referred to simply as "logits": they are the raw, unbounded scores of which we generally push a vector through a softmax function to generate a discrete probability distribution that nicely adds up to $1$.

This definition fits the one you linked from wikipedia (although that link only covers the binary case, and AlphaGo Zero would have multinomial logits since it has more than two outputs for the policy head).

In the AlphaGo Zero paper, the described architecture has a "linear output layer" (i.e. no activation function for the outputs, or the identity function as activation function for the outputs, or however you like to describe it) for the policy head. This means that these outputs are essentially unbounded, they could be any real number. We know for sure that these outputs cannot directly be interpreted as probabilities, even if this isn't stated quite explicitly in the paper.

By calling them logits (or logit probabilities for reasons unknown to me), they are essentially implying that these outputs will still be post-processed by a softmax to convert them into a vector that can be interpreted as a discrete probability distribution over the actions, even if they do not explicitly describe a softmax layer as being a part of the network.

It is indeed possible that in Leela Zero they decided to make the softmax operation explicitly a part of the Neural Network architecture. Mathematically they end up doing the same thing... the AlphaGo Zero paper implies (by using the word "logit") that a softmax is used as a "post-processing" step, and in Leela Zero they explicitly make it a part of the Neural Network.

Here are a couple more sources for the reasoning that usage of the word "logit" basically implies usage of a softmax, though indeed they do not cover the term "logit probability":

https://stats.stackexchange.com/a/52836/144483
https://stackoverflow.com/a/34243720/6735980