How to use DQN to handle an imperfect but complete information game? I'm currently having troubles to win against a random bot playing the Schieber Jass game. It is a imperfect card information game. (famous in switzerland https://www.schieber.ch/)

The environement I'm using is on Github https://github.com/murthy10/pyschieber

To get a brief overview of the Schieber Jass I will describe the main characteristics of the game.
The Schieber Jass consists of four players building two teams.
At the beginning every player gets randomly nine cards (there are 36 cards).
Now there are nine rounds and every player has to chose one card every round. Related to the rules of the game the "highest card" wins and the team gets the points.
Hence the goal is to get more points then your opponent team.

There are several more rules but I think you can image how the game should roughly work.

Now I'm trying to apply a DQN approach at the game.

To my attempts:

I let two independent reinforcement player play against two random players
I design the input state as a vector (one hot encoded) with 36 "bits" for every player and repeated this nine times for every card you can play during a game.
The output is a vector of 36 "bits" for every possible card.
If the greedy output of the network suggest an invalid action I take the action with the highest probability of the allowed actions
The reward is +1 for winning, -1 for losing, -0.1 for a invalid action and 0 for an action which doesn't lead to a terminal state

My question:

Would it be helpful to use a LSTM and reduce the input state?
How to handle invalid moves?
Do you have some good ideas for improvements? (like Neural-Fictitious Self-Play or something similar)
Or is this the whole approach absolute nonsense?

Would it be helpful to use a LSTM and reduce the input state?

I'd bet, no. LSTM is more complicated and harder to learn, while the input is 4 * 9 * 36 bits is still rather limited.

However, you may want to aggregate the information somehow, e.g., add additional bits informing about what cards were already played (no matter when). This information is redundant, but by providing it, you may save the network quite some learning.

At the same time, you may want to use symmetries (all colors but trumps are equivalent and the therefore the weights should be the same).

  How to handle invalid moves?

That's simple: There are no invalid moves. The network provides 36 outputs of how much it wants to play a given card. You simply take the one valid card having the greatest output value. You don't try to make the network learn what moves are valid as this is neither needed nor helpful.

  Do you have some good ideas for improvements? (like Neural-Fictitious Self-Play or something similar)

I can't tell. But it should matter at the moment. First make you network clearly beat the random players, then you can look for more. Or start with Self-Play, as you want probably have both for comparison.

  Or is this the whole approach absolute nonsense?

I don't think so, but ... (see below)

  I design the input state as a vector (one hot encoded) with 36 "bits" for every player

This doesn't sound good. Every player has 9 of 36 cards and so should be the encoding. A player doesn't know the cards of other players.

  The reward is +1 for winning, -1 for losing,

In most card games I know, it matters by how much you win (unlike e.g., in Go). Even when it doesn't matter, using this information at the early learning stages is IMHO useful.

  -0.1 for a invalid action and

Drop the invalid action. Just transform anything the network produces to a valid action, add no penalty (as written above).

  ... 0 for an action which doesn't lead to a terminal state

All action but the last one lead to a non-terminal state. You can use some Temporal Difference Learning or use the fact that the game has a small fixed number of moves and reward/punish all actions takes in a the whole game.


Are policy gradient methods good for large discrete action spaces? I have seen this question asked primarily in the context of continuous action spaces.
I have a large action space (~2-4k discrete actions) for my custom environment that I cannot reduce down further:
I am currently trying DQN approaches but was wondering that given the large action space - if policy gradient methods are more appropriate and if they are appropriate for large action spaces that are discrete as in my scenario above. I have seen answers to this question with regard to large continuous action spaces.
Finally - I imagine there isn't a simple answer to this but: Does this effectively mean DQN will not work?

I don't think that (at least from a practical standpoint), there is much difference between continuous action space and discrete action space with >2k discrete actions. Quoting the "Continuous control with Deep RL" paper - which I'd recommend as a starting point for your investigation:

An obvious approach to adapting deep reinforcement learning methods such as DQN to continuous
domains is to to simply discretize the action space. ... Such large action spaces are difficult to explore efficiently, and thus successfully training
DQN-like networks in this context is likely intractable. Additionally, naive discretization of action spaces needlessly throws away information about the structure of the action domain, which may be essential for solving many problems.

The last sentence in the quote above is the most important point for dealing with your problem. The fundamental issue is inability to efficiently explore such a large  action space - so the idea is to use its structure. I'm sure that your >2k discrete action set has a certain structure on it. Like some actions might be "closer" to others. If so, then you (1) can infer some information about "neighbor" actions even if you never took them (2) do some exploration by adding noise to your policy-preferred action.
The Actor-Critic class of algorithms matches perfectly the steps (1) and (2) above. The Critic Q-value network learns about your state-action space and the Actor policy network returns actions that you could smear.

Are policy gradient methods good for large discrete action spaces?

The Actor-Critic class of RL algorithms is a subclass of the Policy Gradient algorithms. So, the answer is yes - it looks like going that way is your best shot at making progress.

Does this effectively mean DQN will not work?

I don't think that there is a strict "it will not work" statement. The problem is that it just would be extremely difficult to make the DQN training stable "by hand". In principle you might be able to construct a neural architecture that captures the state-action space structure. And then figure out how to perform efficient exploration. And then ensure that nothing explodes anywhere. But that's exactly what DDPG and then TRPO/PPO approaches would do for you.


Would a sentient AI try to create a more optimised AI which would eventually overtake AI 1.0? Would AI be a self-propogating iteration in which the previous AI is
destroyed by a more optimised AI child?
Would the AI have branches of it's own AI warning not to create the new AI?

A common concept in AI is "recursive self-improvement." That is, the AI 1.0 would build a version 1.01, which would build a version 1.02, and so on.

This is probably not going to be thought of as the newer version 'destroying' the older version; if an AI can self-modify, it's probably going to be more like going to sleep and waking up smarter, or learning a new mental technique, or so on.

One important point is that even if the AI is not allowed to self-modify, maybe because of a block put in by its programmers, that won't necessarily prevent it from constructing another AI out in the wild, and so an important problem is to figure out how to best generalize the concept of "don't improve yourself" so that we can make AIs that have bounded scope and impact.


Understanding relation between VC Symmetrization Lemma and Generalization Bounds I am new in the field of Machine Learning so I wanted to start of by reading more about mathematics and history behind it.

I am currently reading, in my opinion, a very good and descriptive paper on Statistical Learning Theory - "Statistical Learning Theory: Models, Concepts, and Results". In section 5.5 Generalization bounds, it states that:

  It is sometimes useful to rewrite (17) "the other way round". That is, instead of ﬁxing $\epsilon$ and then computing the probability that the empirical risk deviates from the true risk by more than $\epsilon$, we specify the probability with which we want the bound to hold, and then get a statement which tells us how close we can expect the risk to be to the empirical risk. This can be achieved by setting the right-hand side of (17) equal to some $\delta > 0$, and then solving for $\epsilon$. As a result, we get the statement that with a probability at least $1−\delta$, any function $f \in F$ satisﬁes

Equation (17) is VC Symmetrization lemma to which we applied union bound and then Chernoff bound:

What I fail to understand is the part where we are rewriting (17) "the other way around". I fail to grasp intuitive understanding of relation between (17) and (18), as well as understanding generalization bounds in general.

Could anyone help me with understanding these concepts or at least provide me with additional resources (papers, blog posts, etc.) that can help?

Let $\varepsilon$ in (17) is equal to $\sqrt{\frac{4}{n}\left(\log{(2\mathsf{N}(\mathcal{F},n))}-\log{\delta}\right)}$. We have:

$$
P\left(\sup_{f\in\mathcal{F}}|R(f)-R_{emp}(f)| > \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}\right) \leqslant
2\mathcal{N}(\mathcal{F},n) e^{\frac{-n}{4}\left(\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)\right)} =
2\mathcal{N}(\mathcal{F},n) e^{\log{\delta} - \log{(2\mathcal{N}(\mathcal{F},n))}}
$$

As we know that $e^{\log{n}} = n$ (suppose that base of $\log$ here is $e$), we can write:

$$
P\left(\sup_{f\in\mathcal{F}}|R(f)-R_{emp}(f)| > \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}\right) \leqslant
2\mathcal{N}(\mathcal{F},n) \left(\frac{\delta}{2\mathcal{N}(\mathcal{F},n)}\right)
$$

Hence:

$$
P\left(\sup_{f\in\mathcal{F}}|R(f)-R_{emp}(f)| > \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}\right) \leqslant \delta
$$
As we know that if we have $P(x > a) \leqslant c$, we can conlude that $P( x<a) \geqslant 1-c$, we will have:

$$
P\left(\sup_{f\in\mathcal{F}}|R(f)-R_{emp}(f)| < \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}\right) \geqslant 1- \delta
$$

Now, as this inequality is true for supremum of set $\mathcal{F}$ and $R(f) -R_{emp}(f) \leqslant \varepsilon$ is subset of  $|R(f) -R_{emp}(f)| \leqslant \varepsilon$ (in terms of probability state space), we can say the following inequality is correct for any function $f$ with at least probability of $1-\delta$:
$$
R(f) \leqslant R_{emp}(f) + \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}
$$


Where does entropy enter in Soft Actor-Critic? I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. From SpinningUp-SAC, it looks like the entropy is entering through the value-function, so I'm thinking it enters by the $\log \pi_{\phi}(a_t \mid s_t)$ in the value function?

I'm a little stuck on understanding SAC, can anyone confirm/explain this to me?

Also, side-note question: is being a soft agent equivalent to including entropy in one of the object functions?

In the answer I'll be using notation similar to the one from the SAC paper.
If we look at the standard objective function for policy gradient methods we have
\begin{align}
J_\pi &= V_\pi(s_t)\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t)]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) + V(s_{t+1})]]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) +  \mathbb E_{a_{t+1} \sim \pi(a|s_{t+1})}[ \mathbb E_{s_{t+2} \sim p(s|s_{t+1}, a_{t+1})} [r(s_{t+1}, a_{t+1}) + V(s_{t+2})]]]]\\
&\cdots\\
&= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r( s_t, a_t)]
\end{align}
If you keep unwinding this $V(s_{t+i})$ you will get expected sum of rewards.
We can define soft state value as
\begin{align}
V(s_t) &= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) + \mathcal H(\cdot|s_t)]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) + \mathbb E_{a \sim \pi(a|s_t)}[-\log(\pi(a|s_t))]]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) - \log(\pi(a_t|s_t))]
\end{align}
third equality comes from the fact that $\mathbb E_{a \sim \pi(a|s_t)}[-\log(\pi(a|s_t))]$ is nonrandom so its the same thing as if we are sampling over $\pi$ only once.

In maximum entropy framework objective function would then be
\begin{align}
J_\pi &= V_\pi(s_t)\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) - \log(\pi(a_t|s_t))]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) - \log(\pi(a_t|s_t)) + V(s_{t+1})]]\\
& \cdots\\
&= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r(s_t, a_t) -\log(\pi(a_t|s_t))]\\
&= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r(s_t, a_t) + \mathcal H(\cdot|s_t)]
\end{align}


Train a model using a multi-column text-filled excel sheet I have an excel sheet filled with my own personal appreciations of movies I've watched, and I want to use it to train an AI model so that it can predict if I'll like a specific movie or not, based on the ones I've already seen.

My data is formatted as following (just a sample, the spreadsheet is filled with hundreds of movies):

And I would like to use all the columns to train my model. Because I am going to say if I liked the movie or not, I know it will be Supervised Learning. I already cleaned the data so there's no blank or missing data, but I do not know how to train my model using every column.

If required, I can be more specific on something, just ask and I'll edit the post.

You will need to convert that to something that a neural network can understand.

Movie Name

Is useless. At least you want to judge a movie by its name.

Description

You will need to perform a Tokenization. Grab the x's more common words and convert them on an array.
I recommend you to see these videos from TensorFlow.
https://www.youtube.com/watch?v=fNxaJsNG3-s&t=18s
There you can find the Google Colab Links for the job.

Director

There are two options. A One Hot Array or an integer for every director.
One Hot Array may be better if you don't have an order of similarity of the directors. But will increase the size of the inputs.
If you have an order of similarity of the directors an integer for each director will work fine.

Rating

Nothing to do here. Is ready to go.
You can perform this work directly on excel, but tensorflow has great tools for it.
It’s hard to run the model after you have trained it on excel.
If you are to comfortable on excel or you can install new software on your computer, I’ve made a backpropagation algorithm that runs on excel and give you a formula to paste in a module.
https://github.com/TorrensJoaquin/Multivariant-Nonlinear-Regression-in-VBA-Small-Neural-Network


How to define a "don't care" class in time series classification in Pytorch? This is a theoretical question.
Setup
I have a time series classification task in which I should output a classification of 3 classes for every time stamp t.
All data is labeled per frame.
The problem:
In the data set are more than 3 classes [which are also imbalanced].
My net should see all samples sequentially, because it uses that for historical information.
Thus, I can't just eliminate all irrelevant class samples at preprocessing time.
In case of a prediction on a frame which is labeled differently than those 3 classes, I don't care about the result.

My thoughts:

The net will predict for 3 classes
The net will only learn (pass backward gradient) for valid classes, and just not calculate loss for other classes.

Questions

Is this the way to go for "don't care" classes in classification?
How to calculate loss only for relevant classes in Pytorch?
Should I apply some normalization per batch, or change batch norm layers if dropping variable samples per batch?

I am using nn.CrossEntropyLoss() as my criterion, which has only mean or sum as reductions.
I need to mask the batch so that the reduction will only apply for samples whose label is valid.
I could use reduction='none' and do that manually, or I could do that before the loss and keep using reduction='mean'.
Is there some method to do this using built in Pytorth tools?
Maybe this can be done in the data-fetching phase somehow?

I am looking some standard, vanilla, thumb rule implementation to tackle this. The least fancy the better.

I am aware this is more than a single question.
They are still not separable, as the solution will be unified most likely.

Don't care classes are not advisable since don't care doesn't have any inherent pattern which would hurt your model inadvertently.
You can assign class weights for your classes. CrossEntropyLoss has an argument 'weights'.
BatchNormalization can be left alone since the normalization should happen on the basis of all data and not just some of the data.


Is it possible to run SSD or YOLO object detection on Raspberry Pi 3 for live object detection? Is it possible to run SSD or YOLO object detection on Raspberry Pi 3 for live object detection (2/4frames x second)?
I've tried this SSD implementation but it takes 14 s per frame.
Is there anything I could do to speed up?

No.
YOLO and SSD are based on Nvidia's proprietary CUDA technology which is not available on Raspberry simply because of the GPU vendor is not Nvidia.
Even more, there seems to be no implementation of even OpenCL for the Raspberry's GPU.
What you can do is to try port YOLO's of SSD's CNN core from CUDA to Raspberry GPU's assembler, in the way described in, for example,  https://rpiplayground.wordpress.com/2014/05/03/hacking-the-gpu-for-fun-and-profit-pt-1/


How do English language robots work? By English language robots I mean something like this: http://www.tolearnenglish.com/free/celebs/audreyg.php
I don't know what they called exactly, but interested to know how they work and how can I build something like them? and what subject should I look for it?

English language robots which you mentioned, are called "chatter bots". Chatter Bots are used to communicate with a human and undergo conversations in such a way that the human which is communicating will think that he/she is talking to another human.

There are two types of chatter bots: One is which uses certain rules and pattern matching techniques and the other one is which uses actual artificial intelligence techniques. Of course, the latter one is the most difficult to implement.

This can be understood with an example. The former type of bots which uses pattern matching and rules consist of questions stored in the form,

if question matches 'WHO (IS/ARE) (MEMBERS OF PARLIAMENT/PRESIDENT)?' then
     RETURN Y ... where Y is some predefined answer.

The latter part which uses actual AI techniques, uses various mechanisms to actually understand the question, extract information out of it, process the information into some standard normalized form and then do some inference w.r.t. the facts in the knowledge base. Such methods may use learning algorithms to actually learn question patterns.

To make simple bots (probably of the former type), you can use Artificial Intelligence Markup Language (AIML) which is a XML based language for developing chatter bots like ALICE bot.

If you want to focus more towards latter type of chatter bots, you may have to learn about various AI techniques, like searching, logic, knowledge and inference, learning, etc or sub-domains of AI like Natural Language Processing (NLP). There are various tool kits for NLP like for developing in Python there is nltk, for developing in Java there is Stanford's CoreNLP, and so on.


A mathematical explanation of Attention Mechanism I am trying to understand why attention models are different than just using neural networks. Essentially the optimization of weights or using gates for protecting and controlling cell state (in recurrent networks), should eventually lead to the network focusing on certain parts of the input/source. So what is attention mechanism really adding to the network?

A potential answer in the case of Encoder-Decoder RNNs is:

  The most important distinguishing feature of this approach from the
  basic encoder–decoder is that it does not attempt to encode a whole
  input sentence into a single fixed-length vector. Instead, it encodes
  the input sentence into a sequence of vectors and chooses a subset of
  these vectors adaptively while decoding the translation. This frees a
  neural translation model from having to squash all the information of
  a source sentence, regardless of its length, into a fixed-length
  vector. We show this allows a model to cope better with long
  sentences.
  - Neural Machine Translation by Jointly Learning to Align and Translate

which made sense and the paper says that it worked better for NMT.

A previous study indicated that breaking down the sentence into phrases could lead to better results:

  In this paper, we propose a way to address this issue by automatically
  segmenting an input sentence into phrases that can be easily
  translated by the neural network translation model. Once each segment
  has been independently translated by the neural machine translation
  model, the translated clauses are concatenated to form a final
  translation. Empirical results show a significant improvement in
  translation quality for long sentences.
   - Overcoming the Curse of Sentence Length for Neural Machine
  Translation using Automatic Segmentation

which paved the way for further research resulting in attention models.

I was also going through an article on Attention is not quite all you need where it said something similar:

  An LSTM has to learn to sequentially retain past values together in a
  single internal state across multiple RNN iterations, whereas
  attention can recall past sequence values at any point in a single
  forward pass.

and a more curated blog on the family of attention mechanism gives insight on how different ways have been formulated for implementing the concept: Attention? Attention!

Specifically, I want to know how attention mechanism is formulated for this task (aforementioned) or in general. A detailed mathematical insight would be helpful, probably somewhat on these lines: Understanding Attention in NN mathematically

There's plenty, but keep in mind that these articles do not describe the same approach. They simply have attention shifting automation as part of their approaches and therefore must detect a need for shift and execute it in a way that improves speed, accuracy, reliability or some combination of them.

There is no one dominant attention approach and probably will not be. In fact, the earliest attention mechanism in common use in machines was likely the electromechanical fire alarm. In digital systems, it would be a vacuum tube electric eye driving an intruder alert followed by the first hardware interrupts in transistor microprocessor boards.

The sophistication of hardware interrupts in contemporary computer systems is probably higher than attention mechanisms in neural nets as of this writing, but that may change. Currently the dictionary definition of attention is the only constraint we can place on these newer approaches in artificial networks.

The four working links in the question (one is broken) lead to articles with some mathematical description, so the question half answers itself.
This article, Dynamic relevance: vision-based focus of attention using artificial neural networks,
Shumeet Baluja, Dean Pomerleau, 1995; revised 1996,
(abstract),
has only a few mathematical expressions but is filled with system
theory that could be described mathematically with a little reading
and thought.
Same with Self-Organization of an Artificial Neural Network
Subjected to Attention Shift Impairments and Familiarity
Preference, Characteristics Studied in Autism, Lennart Gustafsson and Andrew P. Papliński, 2004.
This one has clear and comprehensive mathematical treatment: Backpropagation Generalized Delta Rule for the Selective Attention Sigma-if Artificial Neural Network, Maciej Huk, 2012
This one too: Small and Dim Target Detection via Lateral Inhibition Filtering and Artificial Bee Colony Based Selective Visual Attention, Haibin Duan, Yimin Deng, Xiaohua Wang, Chunfang Xu, 2013
There is some here, but the visualizations and concepts are so theoretically relevant that one might not miss the more extensive mathematics: Using chaotic artificial neural networks to model memory in the brain, Zainab Aram, Sajad Jafari, Jun Ma, Julien C. Sprott, Sareh Zendehrouh, Viet-Thanh Pham
Solid math coverage here: Parallel Implementation of Artificial Neural Network Training, Stefano Scanzio, Sandro Cumani, Roberto Gemello, Franco Mana, P. Laface, 2010, (abstract)
Also here: Multiple Object Recognition with Visual Attention, Jimmy Lei Ba, Volodymyr Mnih, Koray Kavukcuoglu, 2015 ICLR
The mathematics for the very popular DiSAN is here: DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding, Tao Shen, Jing Jiang, Tianyi Zhou, Shirui Pan, Guodong Long, Chengqi Zhang, 2018
Mathematics and the algorithm is provided for this unique approach too: Graph Classification using Structural Attention, John Boaz Lee, Ryan Rossi, Xiangnan Kong, 2018

It would be interesting to develop a taxonomy of attention approaches in AI, as that has probably not yet been done. It would take quite a study to see if any of the above bullet items match up with either of the two articles referenced in the question.


Are Cellular Neural Networks one type of Neural Networks? I am researching Cellular Neural Networks and have already read Chua's two articles (1988). In cellular neural networks, a cell is only in relation with its neighbors. So its is easy to use it for real time image processing. Image processing is performed with only 19 numbers (two 3x3 matrix called A and B and one bias value).

I wonder if we can call cellular neural networks neural networks, because there is no learning algorithm.  They are neither supervised nor unsupervised.

If the standards for what is a neural network are held consistent, then cellular neural networks should probably be considered a type of neural network.

That two adjacent layers be fully connected is not considered a requirement of what people consider neural networks.  Some attention networks, most convolution kernels, and several other types of networks are not exhaustively connected.

Exhaustive connectivity is a burden on computation time that is only valuable in cases when function requires it.  In cases where the parameter value for the connection can be shown to be always zero, there is no need for the potential.  In cases where it is rarely nonzero, the benefit of the potential may outweigh the gain of having it.

Many kinds of networks exist that are not exhaustively connected in computer science, and biological neurons are partly connected.  The axons of biological neurons might grow through an adjacent layer and connect to distant one.

In Chua's paper, it was shown that cellular neural networks converge under specific conditions, just like for other neural network types. There is no claim that they learn like humans do, and they don't use the same back propagation and gradient descent that MLPs use, but they do incrementally improve, which is the only common criteria across the many forms of artificial learning networks.

Cellular neural networks do not fit into what is generally thought of when people use the term unsupervised learning, however, they are not supervised and they do exhibit the same incremental improvement that other unsupervised networks exhibit.

Part of the issue with these names is that specialized jargon forms when only one type of a thing becomes popular, assigning the name to something more specific than the words in the name imply.  Cellular neural networks may not be examples of unsupervised learning, but they learn to function better without supervision.


Agent collision avoidance java I am working with a project which is a agent based pedestrian simulation in Java and its is animated with the help of JavaFX. I've tried to read all the social force model papers but my understanding of those articles are none.
So I tried an own approach which got trashed after failing time after time.

My approach was that each agent calculated its surrounding and first calculated the distance to each of the agents on the field and if that distance was below a constant then the agent calculates the angle of that agent which is too close to it and moves accordingly to the calculated angle.

This approach didnt work for me because the "avoidance code" is not efficient enough and the agents just dont know where to go when they meet and just stays in place.

I am asking for guidance to how I can approach this problem in a better way.

double[] check(Vector<Pedestrian> peds, Pedestrian p1){
for (Pedestrian p : peds){
    if (p.getPedestrianId() != this.id){
        double distance = IPedestrian.distance_formula(getTranslateX(), getTranslateY(), p.getTranslateX(), p.getTranslateY());
        if (distance <= DANGER){
            System.out.println("DANGER");
            return IPedestrian.angle(getTranslateX(), getTranslateY(), p.getTranslateX(), p.getTranslateY(), p1);
        }
    }
}
return new double[] {SPEED, 0};

}

public void move(Vector<Pedestrian> peds, Pedestrian p) {
double[] new_steps = this.check(peds, p);
if (side == SideChooser.Left){
    setTranslateX(getTranslateX() + new_steps[0]);
    setTranslateY(getTranslateY() + new_steps[1]);
} else {
    setTranslateX(getTranslateX() - new_steps[0]);
    setTranslateY(getTranslateY() - new_steps[1]);
}

}

Math formulas:

static double distance_formula(double thisX, double thisY, double otherX, double otherY){
return Math.sqrt(Math.pow(otherX - thisX, 2) + Math.pow(otherY - thisY, 2));

}

static double[] angle(double x1, double y1, double x2, double y2, Pedestrian p){
double angle = Math.toDegrees(Math.atan2(y2-y1, x2-x1));
angle += Math.ceil(-angle/360) * 360;

//double angle = Math.toDegrees(Math.atan2(y2-y1, x2-x1));

if (p.getSideChoosen() == SideChooser.Left){//if the pedestrian is from the left side
    if (angle < 45 || angle > 315)//front
        return new double[]{-SPEED/5, 0};

    else if (angle >= 135 || angle <= 225 ) //back
        return new double[]{SPEED*1.4, 0};

    else if (angle >= 45 || angle <= 90)//North-East
        return new double[]{0, SPEED};

    else if (angle > 90 || angle <= 135) //North-West
        return new double[]{SPEED*1.2 , SPEED};

    else if (angle >= 270 || angle <= 315) //South-East
        return new double[]{0, -SPEED};

    else if (angle > 225 || angle <= 270) //South-West
        return new double[]{SPEED*1.2, -SPEED};

    else
        return new double[]{SPEED, 0};
} else {
    if (angle < 45 || angle > 315)//back
        return new double[]{SPEED*1.4, 0};

    else if (angle >= 135 || angle <= 225 ) //front
        return new double[]{-SPEED/5, 0};

    else if (angle >= 45 || angle <= 90)//North-West
        return new double[]{SPEED*1.2, -SPEED};

    else if (angle > 90 || angle <= 135) //North-East
        return new double[]{0 , -SPEED};

    else if (angle >= 270 || angle <= 315) //South-West
        return new double[]{SPEED*1.2, SPEED};

    else if (angle > 225 || angle <= 270) //South-East
        return new double[]{0, SPEED};

    else
        return new double[]{SPEED, 0};
}

}

A very efficient approach to what you are trying to do is velocity obstacles.

Assuming two agents use constant velocity motion vectors, a velocity obstacle models a geometric region in which if the endpoint of the velocity vector for agent 1 falls, it will collide with agent 2 (and vice versa). Hence you can predict what velocity vectors will lead to collisions between the agents and choose velocity vectors that do not collide.

There are very good examples and tutorials here:
http://gamma.cs.unc.edu/RVO/


Why would adding all the possible embeddings be "worse" than using 1D-convolutions? Suppose we are using word2vec and have embeddings of individual words $w_1, \dots, w_{10}$. Let's say we wanted to analyze $2$ grams or $3$ grams.
Why would adding all the possible embeddings, $\binom{10}{2}$ or $\binom{10}{3}$, be "worse" than using 1D-convolutions?

N-grams are defined as sets of n contiguous words. We use n-grams because they are more useful than random combinations of words across the sentence. Intuitively, combinations of nearby words have more semantic meaning than combinations of distant words.
Also, using all possible combinations of n embeddings would take much longer, especially since (1D) convolutions are such efficient operations.


Given specific rewards, how can I calculate the returns for each time step? Let's use Excercise 3.8 from Sutton, Barto - Introduction to RL:

  Suppose $\gamma = 0.5$ and following sequence of rewards is received
  $R_1=-1$ , $R_2=2$ , $R_3=6$ , $R_4=3$ , $R_5=2$ , with $T=5$ . What
  are $G_0, G_1, ..., G_5?$

There isn't $G_5$ because $R_5$ is last reward. Am I understanding it right?

So:

$G_4 = 2$

$G_3 = 3 + 0.5*2 = 4$

$G_2 = 6+0.5*4 = 8$

$G_1 = 2+0.5*8 = 6$

$G_0 = -1 +0.5*6 = 2$

Perfect.

To back up your intuition about there not being a $G_5$, refer to the definition of discounted return in the periodic case (3.11).
$$G_t \doteq \sum_{k=t+1}^T \gamma^{k-t-1} R_k$$

You'll see that $G_5$ would be written as a sum with no terms in it, since $T=5$.


The real approach of creating AI for a fighting robot in an MMO I have a MMO game where I have players.

I wanted to invent something new to the game, and add player-bots to make the game be singleplayable as-well.

The AI I want to add is simply only for fighting other players or other player-bots that he sees around at his level.

So I thought of implementing my fighting-strategy, exactly how I play, to the bot which is basically using if statements and randoms.

For example, when the opponent has low health, and the bot has enough special attack power, he will use this chance and use his special attack power in order to try to knock the opponent down, or if the bot has low health he will eat in time but not too much because there is a point in risking fights, if you eat too much your opponent will do too.

Or for example if the bot detects the opponent player is eating too much and gains health, he will do the same.

So I told this idea of implementation to one of my friends and he simply responded with: This is not AI, its simply just a set of conditions, it does not have any heuristic functions.

For that type of a game, what are some ideas to create a real AI to achieve these conditions?
Basically the AI should know what to do in order to beat the opponent, based on the opponent's data such as current health, Armour and weapons and level, if he risks his health or not and so on.

I am a beginner and it really interests me to do it in the right way.

I would set up a list of goals for your bot. These could be 'maintain a minimum level of health', 'knock out human player', 'block way to location X', etc. This obviously depends on the domain of your MMO.

Then you can use a planner to achieve these goals in the game. You define a set of actions with preconditions and effects, set the current goal, and the planner will work out a list of actions for the bot to achieve the goal. You can easily express your actions (and the domain) in PDDL.

Examples for actions would be 'move to location X', 'eat X', 'attack player X'. A precondition of 'attack player X' could be 'health(X) is low', and an effect could be 'health(X) is reduced by Y'. There are different ways of expressing these depending on the planner's capabilities.

The beauty of this is that you don't actually have to explicitly code any behaviour. You describe the domain, and tell the bot what it should achieve, and what capabilities it has. The actual behaviour then emerges out of that description. If the bot only attacks a player if the player has lower health, then observing the player eat (and thus up their health) could result in the bot eating (to push its own health above the player's so that it can attack) — but you have not told the bot directly to do that.

For a starting point, go to http://education.planning.domains/ for a list of resources.

If you only have a few actions available, it might appear predictable to a human user, but with a variety of goals and actions, this will quickly become more complex and seem more 'intelligent'.

Update: Here is a link to a paper, Applying Goal-Oriented Action Planning to Games, which describes how this can be applied in a game.


How to evaluate output of unlayered NN? I used to work with 'traditional' layered neural network and I evaluated the output given certain inputs by processing layer-by-layer.
With NEAT, a neural network may assume any topology and they are no longer layered, so how to evaluate the output? I understand time-steps must be taken into account, but how?
Should I keep the inputs until all hidden neurons are processed and output is produced? Should I wait for output to stabilize?

The networks in NEAT are still implicitly layered. There are neurons that need to be evaluated before other neurons can be evaluated and so this gives us our layers.

If you don't know the structure of your network then you can use Kahn's algorithm to find an arbitrary (by arbitrary I just mean one of the possible partially ordered sets) ordering of the nodes in the network. Then you evaluate your neurons in the order given to you by Kahn's algorithm. This works because the ordering of your network (which is a directed acyclic graph) is a partially ordered set.


Is there a way to translate the concept of batch size into reinforcement learning? I am using a neural network as my function approximator for reinforcement learning. In order to get it to train well, I need to choose a good learning rate. Hand-picking one is difficult, so I read up on methods of programmatically choosing a learning rate. I came across this blog post, Finding Good Learning Rate and The One Cycle Policy, about finding cyclical learning rate and finding good bounds for learning rates.
All the articles about this method talk about measuring loss across batches in the data. However, as I understand it, in Reinforcement Learning tasks do not really have any "batches", they just have episodes that can be generated by an environment as many times as one wants, which also gives rewards that are then used to optimize the network.
Is there a way to translate the concept of batch size into reinforcement learning, or a way to use this method of cyclical learning rates with reinforcement learning?

Potentially.

If you do offline reinforcement learning, you're basically learning to approximate a function by sampling input/output pairs, rather than episode-by-episode. Here, your batch size could be set exactly as in an ordinary supervised learning problem.

If you do online learning, then it's not clear to me that the techniques used to set the learning rate in supervised learning can be directly applied though.

Both approaches are well covered in the RL chapter of Russell & Norvig (17? 18?).


Auto-regression - Reduce error in prediction I am trying to develop a time series model using autoregression. The data set is like as follows
INDEX MAXIMA
  0   0.743
  1   0.837
  2   0.838
  4   0.896
  5   1.014
  6   1.003
  7   1.01
  8   1.101
  9   1.097

The Maxima point is given is the largest points on each curve. Basically, I have to perform multi-step forecasting (at least 9 steps ahead). I've done it using the recursive approach. but the accuracy of the prediction getting worse as it reaches the end.

Predicted Result

PYTHON CODE
Using the AR model from stats model
# fit model for MAX VALUE
  model = AR(data)
  model_fit = model.fit()
  yhat_max = model_fit.predict(len(data),len(data]))

For obtaining an accurate prediction, What changes should be done in the approach? or Do I have to change the model?
Any kind of help is appreciated.

The predictions tend to move towards the mean of the series as one predicts for longer horizons. Also, in general, optimal long range forecast is the process mean.

In other words, the past of the process contains no information on the development of the process in the distant future.

And, this might be the reason that you are getting poor forecasts.

  What changes should be done in the approach? or Do I have to change the model?

You might want to move to ARIMA models. See how they perform.

If you have some other time series that might act as explanatory variables then you might want to augment the dataset and use the ARIMAX model.

Third option would be to try out RNN if you have lots of data. You can also try hybrid models like ARIMA and RNN together.


What are examples of techniques to prevent bias in artificial intelligence systems? I recently read an article about how artificial intelligence replicates human stereotypes when applied to biased datasets.
What are examples of techniques to prevent bias (and stereotypes) in artificial intelligence (in particular, machine learning) systems?

It's important to note that ultimately, the statistical methods we currently use in ML research are just that: statistical methods. So when they show some "bad behaviour" it's not because of problems with the statistical methods, but with the data we give it. But if the data we give it are as "genuine and unfiltered" as it gets, then it probably shows something about us.

From a cognitive science perspective, its probably the case that the same heuristics and biases that creates stereotypes are also the ones that make us powerful agents(note the similarity between categories and stereotypes), so at least at this moment its unclear how we can segregate desired from undesired behaviour.

To combine the points mode above, it seems we can only either: 1. Remove "bad content" by curating the data by hand or by some metric that we don't know of yet 2. Accept that our methods will produce AI as "bad as we are" cause that's what we are, and let it operate under the knowledge that it might produce undesired behavior sometimes.

Unless we have some crazy new theory of mind that we can begin to analyze this in a more rigour manner, it seems like there is no clear cut solution.


Is it a sign of overfitting when validation_loss dips and then goes up with increasingly bigger swings? I am experimenting with a ConvNet to categorize images taken with a depth camera. So far I have 4 sets of 15 images each. So 4 labels. The original images are 680x880 16-bit grayscale. They are scaled down before feeding it to the ImageDataGenerator to 68x88 RGB (each color channel with equal value). I am using the ImageDataGenerator (IDG) to create more variance on the sets. (The IDG does not seem to be able to handle 16-bit grayscale images, nor 8-bit images well, so hence I converted them to RGB).

I estimate the images to be low on features, compared to regular RGB images, because it represents depth. To get a feel for the images, here are a few down scaled examples:

I let it train 4.096 epochs, to see how that would go.

This is the result of the model and validation loss.

You can see that in the early epochs the validation (test / orange line) loss dips, and then goes up and starts to show big swings. Is this a sign of overfitting?

Here is a zoomed in image of the early epochs.

The model loss (train / blue line) reached relatively low values with an accuracy of 1.000. Training again shows repeatedly the same kind of graphs.
Here are the last epochs.

Epoch 4087/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.1137 - accuracy: 0.9286 - val_loss: 216.2349 - val_accuracy: 0.7812
Epoch 4088/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0364 - accuracy: 0.9643 - val_loss: 234.9622 - val_accuracy: 0.7812
Epoch 4089/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0041 - accuracy: 1.0000 - val_loss: 232.9797 - val_accuracy: 0.7812
Epoch 4090/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0091 - accuracy: 1.0000 - val_loss: 238.7082 - val_accuracy: 0.7812
Epoch 4091/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0248 - accuracy: 1.0000 - val_loss: 232.4937 - val_accuracy: 0.7812
Epoch 4092/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0335 - accuracy: 0.9643 - val_loss: 273.6542 - val_accuracy: 0.7812
Epoch 4093/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0196 - accuracy: 1.0000 - val_loss: 258.2848 - val_accuracy: 0.7812
Epoch 4094/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0382 - accuracy: 0.9643 - val_loss: 226.6226 - val_accuracy: 0.7812
Epoch 4095/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0018 - accuracy: 1.0000 - val_loss: 226.2943 - val_accuracy: 0.7812
Epoch 4096/4096
7/7 [==============================] - 0s 11ms/step - loss: 0.0201 - accuracy: 1.0000 - val_loss: 207.3653 - val_accuracy: 0.7812

Not sure if it is required to know the architecture of the neural network to judge whether this is overfitting on this data set. Anyway, here is the setup.

kernelSize = 3
kernel = (kernelSize, kernelSize)

model = Sequential()
model.add(Conv2D(16, kernel_size=kernel, padding='same', input_shape=inputShape, activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(32, kernel_size=kernel, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, kernel_size=kernel, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(32, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(nr_of_classes, activation='softmax'))

sgd = tf.keras.optimizers.SGD(lr=learning_rate, decay=1e-6, momentum=0.4, nesterov=True)
model.compile(loss='categorical_crossentropy',
            optimizer=sgd,
            metrics=['accuracy'])

Yes this looks a lot like overfitting. The clue is in the low and slowly decreasing training loss compared to the large increases in validation loss.

One simple fix would be to stop training around epoch 50, taking the best cross validation result to select the most general network at that point. However, anything that works to improve stable generalisation could help here - more training data, more regularisation, simpler model.

The tricky part is finding the best combination for generalisation. Typically if you overuse regularisation to make the NN completely stable, it will lose some accuracy, so you need to run multiple experiments and measure things carefully. As you have a small dataset here and seem capable of running 1000's of epochs, I would suggest k-fold cross validation for improved measurement of cross validation loss.


Problem extracting features from convolutional layer where the dimensions are big for feature maps I have trained a convolutional neural network on images to detect emotions. Now I need to use the same network to extract features from the images and use them to train an LSTM. The problem is: the dimensions of the top layers are: [None, 4, 4, 512] or [None, 4, 4, 1024]. Therefore, extracting features from this layer will result in a 4 x 4 x 512 = 8192 or 4 x 4 x 1024 = 16384 dimensional vector for each image. Clearly, this is not what I want.
Therefore, I would like to know what to do in this case and how to extract features that are of reasonable size. Should I apply global average pooling to the activation or what?
Any help is much appreciated!

I think it's okay to let CNN extract a big number of feature maps, then you can reduce the features from it. To reduce features from CNN result, you can do feature selection or feature extraction. Some people prefer to use feature selection and select the most significant feature maps, for examples:

Action unit selective feature maps in deep networks for facial expression recognition
Convolutional Neural Network Simplification Based on Feature Maps Selection

Hope it helps.


How to handle proper names or variable names in word2vec? The input in word2vec is known word (spellings), each tagged by its ID.

But if you process real text, there can be not only dictionary words but also proper nouns like human names, trade marks, file names , etc, how to make an input for that?
Is you consider some input where items are variables, like the meaning of input would be x = something, and after some time you acces to x value and define some other stuff with it. That would be format for this input, and will this approach work at all?

Word2vec works on the concept of  typical word co-occurrences. This means that it will work well only for words that occur frequently in the dataset. So proper nouns will not play any role in training the model. You can keep the proper nouns as they are or use only the words the occur more frequently than some threshold value based on the size of your dataset.

Once you use the value stored in variable x for something, and then change the stored value, it will not reflect anywhere unless you us use the variable x again somewhere in the program.

# Example
x = "something something"
print(x + "...")

# Result
something something ...

# Changing x
x = "new value"

# This new value of x will not reflect anywhere in the program
# Unless you use the variable x again.


Can you explain me this CNN architecture? I am starting to get my head around convolutional neural networks, and I have been working with the CIFAR-10 dataset and some research papers that used it. In one of these papers, they mention a network architecture notation for a CNN, and I am not sure how to interpret that exactly in terms of how many layers are there and how many neurons in each.
This is an image of their structure notation.

Can some give me an explanation as to what exactly this structure looks like?

In the CIFAR-10 dataset, each image is $32 \times 32$ pixels, represented by 3072 integers indicating the red, green, blue values for each pixel.
Does that not mean that my input layer has to be of size 3072? Or is there some way to group the inputs into matrices and then feed them into the network?

While it would certainly help if the link to the paper could also be posted, I will give it a shot based on what I understand from this picture.

1) For any convolutional layer, there are few important things to configure, namely, the kernel (or, filter) size, number of kernels, stride. Padding is also important but it is generally defined to be zero unless mentioned otherwise. Let us consider the picture block-by-block.

The first block contains 3 convolutional layers: (i) 2 conv layers with 96 filters each and the size of each filter is $ 3 \times 3$ (and stride $=1$ by default since it is not mentioned), and (ii) another conv layer with same configurations as above but with stride $=2$.

The second block is pretty much the same as the previous except the number of filters is increased to 192 for each layer that is defined.

The only considerable change in the third block is the introduction of $ 1 \times 1$ convolutional filters instead of $3 \times 3$.

And finally, a global average pooling layer is used (instead of a fully connected layer).

2) As for your analysis, it is exactly the case in fully connected layers, wherein the number of units in the input layer must match the vectorized dimensions of the input data. But, in the case of CNN, we give the images directly as the input to the network. The whole idea of a CNN is to understand the spatial structure of the data by analyzing patches of the image at a time (which is what the filter size defines). This PyTorch tutorial should give an idea as to how exactly the input is given to CNN.


Is there any network/paper used to analyse music scores? As I am curious on music theory I would like to know that If is there any such network that analyse like labeling chords, or doing a roman numeral analysis.
Like an example below:

Source
It does not seem to be a difficult task.
Some other examples are given here[external link]
Also I am curious that If it is a possible task for AI to accomplish.

I have not come across music labeling algorithms but upon a google scholar search, I found a couple of papers that aim to do quite the same task.
In general, if you have a labeled dataset then you can take an approach of a general speech recognition model. It should work fine for music labeling too, but you might need to tweak certain parameters.


What is an appropriate fitness function for a simulated self-driving car? I have been working for ages on a neuro-evolution AI program, where cars learn how to race around a track.  Presently, I have a rudimentary fitness function that awards points for every degree traveled in the CW direction about the center of the window (removes points in CCW dir) and removes a certain amount of points for every collision that occurs. Cars also lose points for every moment they stay still and/or are colliding with something. My end goal is to create cars that can complete a track full of obstacles, faster than a human controlled car.

Is there a better fitness function that would result in more efficient cars that

make better use of their sensors,
are efficient in getting around the track (don't weave like a drunk driver), and
are faster in general (cars are too cautious, but this is a race!)

?

Half of the population seems to just spin around.

I have fully implemented my neuro-evolution program. Here's my implementation. However, you'll find that it isn't perfect.

How should I alter my fitness function to generate better driving cars?

Currently, the cars will only turn left or right if the outer sensors are activated.  However, if the sensors directly in front are activated, the neural network is configured in such a way that "ignores" this signal. So, the cars crash head-on into obstacles but avoid obstacles in their periphery.  I think this is due to the fact that the fitness function (which gives points for the displacement in 3 second time period) is too generous and removes the incentive for cars to avoid obstacles.  I've tried altering the punishment for collisions and the reward for driving in the right direction but it still isn't performing the way I would like it to.

Here are some screenshots

You're correct that it is related to the fitness function, but only indirectly.  Recall that a function is the mathematical embodiment of a conceptual relationship.  The model is a motion model.  Proximity is not simply applying the distance formula $D = \sqrt{(x_2 - x_2)^2 + (y_2 - y_2)^2}$.

Proximity as we "feel it" when driving a car is based on the risks we see in the trajectories of ourselves and the other moving objects along with the stationary objects.  Mathematically, this is an integral over time of a distribution of probable locations.

Look for work in the aeronautics industry about collision avoidance.  The mathematics and algorithms are well developed for antiaircraft weaponry (where the fitness function is the inverse of the one you want) and air traffic control, where the fitness is like yours except with an altitude dimension.


Question on identifiability in the “Dueling Network Architectures for Deep Reinforcement Learning” paper I have two questions on the Dueling DQN paper. First, I have an issue on understanding the identifiability that Dueling DQN paper mentions:

Here is my question: If we have given Q-values $Q(s, a; \theta)$ for all actions, I assume we can get value for state $s$ by:
$$V(s) = \frac {1} {|Q|} \sum_{a \in \mathcal{Q}} Q(s, a; \theta)$$
and the advantage by:
$$A(s,a) = Q(s, a; \theta) - V(s), ~~~ \forall ~a ~in ~\mathcal{A}(s)$$
in which $\mathcal{A}(s)$ is the action space for state $s$. If this is correct, why do we need to have two heads in the network to obtain value and advantage separately?
and then obtain Q-value using
$$Q(s, a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \left( A(s, a; \theta, \alpha) - \max_{a' \in | \mathcal{A} |} A(s, a'; \theta, \alpha) \right). \tag{8}$$
or
$$Q(s, a; \theta, \alpha, \beta) = V (s; \theta, \beta) + \left( A(s, a; \theta, \alpha) − \frac {1} {|A|} \sum_{a' \in \mathcal{A}} A(s, a'; \theta, \alpha) \right). \tag{9}$$
Am I missing something?
My second question is why Dueling DQN does not use the target network as it is used in the DQN paper?

Regarding your first question, $$V^{\pi}(s) = \sum_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$ so recovering the value function from Q really depends on what policy $\pi$ you are using. Hence, you can't really recover the value function $V(s)$ from the $Q(s,a)$ values without knowing your policy distribution for state $s$.
However, you can recover $Q^{\pi}(s,a)$ values if we know $V^{\pi}(s)$ and $A^{\pi}(s,a)$. This is because
$$A^{\pi}(s,a) = V^{\pi}(s,a) - Q^{\pi}(s,a)$$
by definition of advantage. And this is why you need 2 heads to recover the $Q$ values from the Value and Advantage functions. In the original paper, the author's do not use this direct equation to recover $Q^{\pi}(s,a)$ values due to "identifability" issue and the fact that both $V^{\pi}(s)$ and $Q^{\pi}(s,a)$ are only estimates.
Regarding your second question, I believe the author's appllied the Duelling architecture on Double Deep Q Networks, which is an improvement over the single DQN used by Minh et al in learning atari. I do think that you can still use a a target network as in the single DQN case if you wanted to.


What does the depth of a decision tree depend on? In these notes, we have the following statement

The depth of a learned decision tree can be larger than the number of training examples used to create the tree

This statement is false, according to the same notes, where it is written

False: Each split of the tree must correspond to at least one training example, therefore, if there are $n$ training examples, a path in the tree can have length at most $n$
Note: There is a pathological situation in which the depth of a learned decision tree can be larger than number of training examples $n$ - if the number of features is larger than $n$ and there exist training examples which have same feature values but different labels.

I had written on my notes that the depth of a decision tree only depends on the number of features of the training set and not on the number of training samples. So, what does the depth of the decision tree depend on?

While this sounds obvious, the depth of the tree depends on how your algorithm builds the tree. For a fixed dataset $\mathcal{D}$, there're many algorithms, such as ID3, C4.5, CART, etc. (and their variants) to build your tree. For the most part, these algorithms recursively partition the dataset, so it's never possible to get a tree larger than $|\mathcal{D}|$. Large/deep trees are also prone to overfitting and are computationally expensive, so these algorithms typically prune the tree so it's much smaller than $|\mathcal{D}|$.
In fact,
Kearns and Mansour showed that under the weak learning assumption (i.e. at each node, there's a split that classifies the data at this node better than a random classifier, by $\gamma$) to achieve $\epsilon$ training error, it suffices to make $(1/\epsilon)^{\mathcal{O}(\log(1/\epsilon)/\gamma^2)}$ splits (and thus depth is upper bounded by this).
But of course, you can always cook up trees of arbitrary depth...


Is there a good and easy paper to code policy gradient algorithms (REINFORCE) from scratch? I am interested in learning about policy gradient algorithms and REINFORCE. Can you suggest a good and easy paper that I can use to code them from scratch?

this is maybe not a paper but this article is dope.

I got 2 link in below to help you understand about policy gradient algorithms especially Reinforce Learning. Both article have a good explanation about that algorithms and not only an explanation, also contain a good example about that. They really put their effort into it so i'm sure if you've read it and really understand it, you can code from scratch. Hope this is help you.

If you think this answer is helping you, i will grateful if you make my answer as accepted. Just kidding, have a nice day....

(no i'm not kidding)

https://towardsdatascience.com/policy-gradients-in-a-nutshell-8b72f9743c5d by Sanyam Kapoor on Towardsdatascience.com

https://medium.com/@jonathan_hui/rl-policy-gradients-explained-9b13b688b146 by Jonathan Hui on Medium.com


How to fill NaNs in Cross-Validation? I have been searching this but did not find the answer, so sorry if this is a duplicated question.
I was working with cross-validation, where some doubts came to my mind, and I am not sure about which is the correct answer.
Lets say I have a mixed dataset, with numerical and categorical features. I want to perform a K-Fold Cross-Validation with it, with a K=10. Some of this numerical features are missing, so I decided that I will replace those NaNs with the average of that feature.
My steps are the following ones:

Read entire dataset
Perform One Hot Encoding to categorical features.
Divide my data into different folds. Lets say that I will use 90% for training, 10% for validating.
For every different combination of folds, I replace the missing values from the traininig and validating sets separately. This means, in one hand I get the average of the missing values of the training part, and on the other hand the average of the missing values of the validating part.
Normalize the data of the training and validating sets between [0, 1] separately, like I did before.
Train the correspondant model.

So lets put a simple example of a dataset of 20 rows with N columns. Once I do steps 1 and 2, in the first iteration I will select the 18 first rows as a training set, and the last two rows as validating set.
I fill the missing values of the 9 first rows with the average of those 18 rows. Then the same for the 2 last rows.
Then, again, normalize in the same way, separately. And do this for every combination of folds.
I am doing it like this, because otherwise, from my understanding, is that you are training your model with a biased data. You should not have access to the validation data, thus you should not be able to do the average with those numbers. Hence I am using only the numbers of the training part. If I do the average with the entire dataset, this will make my model to overfitting.
I am not so sure about the normalization step, as I do not really think this will have the same impact. But here I do not really know...
Is this aproach correct? Or should I do the average and normalization with the entire dataset? Why?
Kind regards.

I would do the exact same thing as you are describing! One of the main reasons that you would want to do cross-validation is to prevent that your model is unable to generalize later. Therefore, you take out a random small subset which will be your new small validation set and do all the 'operations' on that which you are also doing on your training set. This way, you can check that these 'operations' are thus also generalizable (if they work for all different cross-validations).


How does the forget layer of an LSTM work? Can someone explain the mathematical intuition behind the forget layer of an LSTM?

So as far as I understand it, the cell state is essentially long term memory embedding (correct me if I'm wrong), but I'm also assuming it's a matrix. Then the forget vector is calculated by concatenating the previous hidden state and the current input and adding the bias to it, then putting that through a sigmoid function that outputs a vector then that gets multiplied by the cell state matrix.

How does a concatenation of the hidden state of the previous input and the current input with the bias help with what to forget?

Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special characteristic of a sigmoid that creates a vector of important embeddings?

I'd really like to understand the theory behind calculating the cell states and hidden states. Most people just tell me to treat it like a black box, but I think that, in order to have a successful application of LSTMs to a problem, I need to know what's going on under the hood. If anyone has any resources that are good for learning the theory behind why cell state and hidden state calculation extract key features in short and long term memory I'd love to read it.

TL;DR Here is a beautiful explanation with diagrams: source.
To address:

the cell state is essentially long term memory embedding (correct me if I'm wrong)

The embedding can be long or short term and it is a vector.
To answer:

Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special characteristic of a sigmoid that creates a vector of important embeddings?

Excerpt from source:

The sigmoid layer outputs numbers between zero and one, describing how much of each component should be let through. A value of zero means “let nothing through,” while a value of one means “let everything through!”

Formally the forget vector is:
$$f_t = \sigma(W_f\cdot[h_{t-1},x_t],+b_f)$$
So we see that it is actually a linear operation followed by a non-linear operation, which restricts the values to be between 0 and 1. This is followed by an element-wise operation on the previous cell state. That is we "gate"/"filter" the previous cell state doing:
$$C_{t-1}\odot f_t,$$
where $\odot$ is element-wise multiplication.
Thus we see that the forget gate really is acting like a gate: It either lets values pass through or it pushes them toward zero. So, it's purpose is to assign what values to forget/remember.
To answer:

How does a concatenation of the hidden state of the previous input and the current input with the bias help with what to forget?

This concatenation takes into account the "rolling" hidden state and the current input. That is, the linear operation results in a new vector which can be seen as a "consideration embedding" of the current input and a "summary" of past inputs and cell states. The sigmoid then converts this "consideration embedding" into the forget vector.
In summary, the forget gate has the primary purpose of forming a vector of values between zero. This vector results by considering the current input and the previous hidden state. The vector is used to forget/remember parts of the previous cell state via element-wise multiplication. So, no, it is not a black box, it is a highly considerate non-linear operation through time.
Additional Intuition - More Mathematical
Recall that multiplying a matrix on the right by a column vector produces a vector, which is a linear combination of the columns of the matrix. So each column of $W_f$ can be seen as a column of "toggles" that push up to 1 or pull down to 0 (once we apply sigmoid). That is, the columns of $W_f$ form a sort of "basis" for a "toggle space."
When we concatenate to form $[h_{t-1},x_t]$ and then multiply we see that the term $W_f\cdot[h_{t-1},x_t]$ gives consideration to the hidden state and the input. That is, both $h_{t-1}$ and $x_t$ have there hand in toggling the gate.
The LSTM has learned from the data how to make this toggling "meaningful." Below is a simple pictographic example.

That is, $W_f\cdot[h_{t-1},x_t]$ will be a vector in the column space of $W_f$ (ie the "toggle space"). Thus, the concatenated vector will "point" to the best "toggle" in the column ("toggle") space of $W_f$. This "toggle" is then transformed into a "gate" in the "gate space."
So intuitively, the range of the forget component is a sort of "gate space" where the domain is all vectors of form $[h_{t-1},x_t]$. Formally:
$$f_t:[h_{t-1},x_t]\mapsto g_t$$
That is to say that the forget gate learns the best mapping from the domain of concatenated hidden/input's to the range of possible gates.
Important Note
The LSTM is only one of many types of forget-gate mappings. The main takeaway is that the LSTM works empirically for many applications.
That is a forget gate is an instance of a general "gate mapping":
$$f_t:v_t \mapsto g_t,$$
where $v_t$ could be any number of vectors resulting from any number of concatenations.
Conclusion
Some data from the past is irrelevant to the current time step. We need a way to correctly "forget" irrelevant information. One way to "forget" is to use a forget gate mapping (like the one in LSTMs). Then we need to optimize the parameters of that forget gate. One implementation is used in the common LSTM architecture. Finally, the concatenation of the previous hidden state provides contextual information to the current input which is very helpful with forgetting.
More Mathematical Intuition
Here is another view of what is happening in the LSTM from a dynamic point of view: video.


Should the input to the negative log likelihood loss function be probabilities? I am trying to train a supervised model where the output from the model is output of a linear function $WX + b$. Kindly note that I'm not using any softmax or $\log$ softmax on the result of the linear. I am using negative log-likelihood loss function, which takes the input as the linear output from the model and the true labels. I am getting decent accuracy by doing this, but I have read that the input to negative log-likelihood function must be probabilities. Am I doing something wrong?

This seems pretty reasonable to me. You can optimize for any function that is proportionate to the negative log-likelihood. Conventionally, we assume that the likelihood of piece of data under a linear model is proportionate to some sort of gaussian function of the difference between the predicted value and the observed value. If you're a Bayesian you'd say this is a probability. If you're a hardcore frequentist, you might quibble about that, but it's still a number between 0 and 1.

If you take the log of this likelihood function however, you'll get a quadratic function of the difference, with some scalers that you don't need to worry about. So you ought to minimize:

$$\sum_y (y-\hat{y})^2$$

This is not a probability, but since it is proportionate to the log of the original likelihood function (which was, in some sense), minimizing it will also minimize the original function.

Hope that helps!


How does general image background removal AI work? I'm well aware of the inner workings of CNN models for object detection, and although I've not worked on a semantic segmentation problem I can imagine how it works.
With these types of models, we need to say "segment out the humans", or "segment out the X". But what about when I say something like "segment out the subject of this photo, whatever it happens to be". For example, see this service: https://removal.ai/
Without too much imagination I might guess that they apply a multiclass segmentation model and just show any foreground pixels, no matter what class they belong to. So we'd hope that the subject is in one of the classes that the model was trained for, and that there are no other class instances in the image that shouldn't be captured. But is there a more general way?

In image segmentation the target is actually an image, with the same dimensions as the input, where each pixel has a label depending on which class it represents. It is not uncommon for such a dataset to have a "background" class that essentially consists of the pixels not belonging to any other class. If not you can always group together classes typically associated with background (e.g. "sky", "cloud", "grass", "mountain", etc.) to form the class "background". Likewise you could group all other possible classes of interest (e.g. "person", "car", "horse", etc.) into the class "foreground". With this dataset you could train an image segmentation model that predicts if a pixel belongs to the background or the foreground, without actually classifying it into a "person" or a "car".
So suppose you want to make your own removal.ai, you could:

find one or more diverse image segmentation datasets (it needs to be diverse so that it will work on any generic photo uploaded to the site)
check all the unique classes in the labels
group all classes associated with backgrounds into class 0 (i.e. "background" class)
group all classes associated with foregrounds into class 1 (i.e. "foreground" class)
train an image segmentation model with these two classes
profit


Why are autonomous cars categorized as AI? How are autonomous cars related to artificial intelligence? I would presume that artificial intelligence is when we are able to copy the human state of mind and perform tasks in the same way. But isn't an autonomous car just rule-based machines that operates due to its environment? They are not self-aware, and they cannot choose a good way to act in a never before experienced situation.

I know that many people often mention autonomous cars when speaking about AI, but I am not really convinced that these are related. Either I have a too strict understanding of what AI is or

There is a neat definition of artificial intelligence, which circumvents the problem of defining "intelligence" and which I would ascribe to McCarthy, the founder of the field, although I can only find it now in this book by H. Simon:

"… having to do with finding ways to do intelligent tasks, to do tasks which, if they were done by human beings, would call for our human intelligence."

So, at its core we call the automation of every task AI, that can only be done by the human mind. At the time people thought that a computer able to play chess would also be intelligent in other ways. When this turned out to be false, the term AI was split into "narrow or weak AI", i.e. a program able to do one task of the human mind, and "general or strong AI", a program that can do all the tasks of the human mind.

Self-driving cars are narrow AI.

Note, that all these definitions don't specify whether these programs copy the way the human mind works or whether they come to the same result via completely different algorithms.


How to represent Tic-Tac-Toe vs Checkers or Chess for a Neural Network I've been reading a lot about TD-Gammon recently as I'm exploring options for AI in a video-game I'm making. The video game is a turn-based positional sort of game, i.e. a "units", or game piece's, position will greatly impact it's usefulness in that board state.

To work my way towards this, I thought it prudent to implement a Neural Network for a few different games first.

The idea I like is encoding the board state for the Neural Network with a single output neuron which gives that board states relative strength compared to other board states. As I understand, this is how TD-Gammon worked.

However, when I look at other people's code and examples/tutorials, there seems to be a lot of variance in the way they represent the board-state. Even for something as simple as tic-tac-toe.

So; specifically for tic-tac-toe, which is a better, or what is the correct representation for the  board state? I have seen:

9 input neurons, one for each square. A 0 indicating a free-space, -1 the opponent and 1 yourself.
9 input neurons, but using different values such as 0 for the opponent, 0.5 for free and 1 for yourself?
Could you use larger values? LIke 0, 1 and 2?
27 input neurons. The first 3 being square 1, the next 3 being square 2 etc. Every neuron is 1 or 0. The first of the set of three indicates whether this square is free or not; the second indicating whether the square is occupied by your opponent or not. In the end, only one in every 3 neurons will have a 1, the other two will have a 0.
18 input neurons. The first being 1 for the X player, the second being 1 for the O player and both being 0 for a blank

Then; when branching into games where the specific pieces abilities come into play, like in chess, how would you represent this?

Would it be as simple as using higher input values for more valuable pieces? I.e. -20 for an opponents Queen and +20 for your own queen? Or would you need something more complex where you define 10+ values for each square, one for each unit-type and player combination?

When you are working with neural networks, as long as the data is there, the network is usually able to learn how to process it into a useful result, you usually also want to keep the amount of weights to a minimum. When you you use extra weights, it will take longer to train the network because you need to tune even more values for an optimal network. So, for tic-tac-toe, any of your solutions involving 9 inputs should work just fine. Also, it helps if you keep the inputs between 0 and 1 if you are using log sigmoid, and -1 and 1 if you are using hyperbolic tangent for your activation function. You can probably easily figure out what to use for other activation functions. You can take your data and transform it into another dataset with values within a specific range through a process called range normalization. For chess, you can simply encode every piece in several different ways, and it will probably not make that much of a difference. The general rule of thumb is you want to minimize the amount of weights while still giving the most possible variables to the network.


Do images submitted to a segmentation network after training need to be the same size as the training images? I have not seen this explicitly stated anywhere so I was curious. Say I have network trained to meet my segmentation needs using 250x250 images. After this training is complete and I wish to submit images in production for segmentation, do those production submitted images need to be 250x250 as well or can they be any reasonable size?

If they must be resized to 250x250 for segmentation, is it possible to scale up the segmentation regions to apply to a larger image? If so what is the name of that technique so I can research it a bit more.

Yes, you will need data that has the same dimensionality as your training set. Otherwise the model will either discard excess data, or will supplement missing data with a random number. This will likely lead to very poor results.

I would suggest to rescale your images to have the same pixel density as those you used for training (250x250) rather than scaling your model.

Methods for rescaling images and maintaining details are very well developed. Many exist and you can use existing tools such as Photoshop to change the pixel density of an image.


GA rule discovery fitness function I'm attempting to develop a genetic algorithm capable of discovering classification rules for a given data set, a number of papers make use of the Confidence (precision) and Coverage of a rule to define its fitness.

However I'm not sure my understanding of the equations is correct.

For example confidence is:

conf = |P & D| / |P|

And is defined as follows; "In classification problems, confidence measure is defined as the ratio of the number of examples in P that are correctly classified as decision class of D and the number of examples in P."

Is this saying, the total number of occurrences of the attributes in a given rule P which occur in rules which have been classified as class D, by the number of attributes in P ?

Where an example of a rule containing two attributes would be as follows:

(martial_status = married  &  age > 30)

It seems a number of papers define it differently which has led to my confusion, if anyone is able to confirm my understanding or provide an some insight that'd be great.

Edit:

The research paper I've been following can be found here.

The confidence equation you are referring to is the definition of precision in the Classification/pattern-recongition/information-retrieval contexts. You can visually understand the equation with the help of the following figure from the wikipedia page:

 P     : Refers to the set of samples in your dataset. (Selected elements)
|P|    : Refers to the number of samples in your dataset.
 D     : Refers to the set of correct class labels.(aka. Ground Truth).
|P & D|: Refers to the number of samples in the dataset that the classifier correctly labeled.

I hope with this understanding you can implement the fitness function for your Genetic Algorithm. If you want more help in defining the fitness function, then you should probably add details about your approach or links to the research paper you are trying to follow.


Cost function increasing with SGD In Deep Learning by Goodfellow et al., I came across the following line on the chapter on Stochastic Gradient Descent (pg. 287):

  The main question is how to set $\epsilon_0$. If it is too large, the
  learning curve will show violent oscillations, with the cost function
  often increasing significantly.

I'm slightly confused why the loss function would increase at all. My understanding of gradient descent is that given parameters $\theta$ and a loss function $\ell (\vec{\theta})$, the gradient update is performed as follows:

$$\vec{\theta}_{t+1} = \vec{\theta}_{t} - \epsilon \nabla_{\vec{\theta}}\ell (\vec{\theta})$$

The loss function is guaranteed to monotonically decrease because the parameters are updated in the negative direction of the gradient. I would assume the same holds for SGD, but clearly it doesn't. With a high learning rate $\epsilon$, how would the loss function increase in its value? Is my interpretation incorrect or does SGD have different theoretical guarantees than vanilla gradient descent?

The learning rate used in SGD and in most convergence strategies used in artificial network designs is much like the use of voltage dividers in electronic feedback. The differential equations for control theory, developed by Norbert Wiener and others, demonstrated that insufficiently attenuated negative feedback will cause a circuit to destabilize.

Back-propagation with gradient descent is conceptually similar. The main difference is that artificial networks use discrete examples or batches of examples rather than continuous signals. Back-propagation is a strategy to distribute negative feedback throughout the layers of an artificial network using the chain rule, but that is also common in electronic control circuitry.

It is hypothesized, and the evidence is accumulating in support, that more sophisticated strategies to achieve stability using feedback exist in the neural networks of the brain and in the gut of animals. That is likely, and part of the reason arrays of activation functions are called artificially neural.

Those that have studied the mathematics field of chaos understand that discrete systems can fall into oscillation with insufficiently attenuated negative feedback too. In a worse case, a chaotic pattern will emerge at what is called a bifurcation point. Those that work with convergence in machine learning are always working against factors that lead to divergence. However, too much attenuation, which corresponds to too low a learning rate will be highly stable and extremely costly in terms of computing resources, time to converge, or both.

All of this was established in theory in the late nineteenth and early twentieth century. Goodfellow is explaining it to university level students in the context of deep learning, but it is not specific to SGD. In fact SGD is less likely to diverge or go into chaotic states than what the question is calling vanilla gradient descent.

From a mathematicians stoic point of view, calling chaos violent is anthropomorphic, placing human qualities on non-human things. They might point out that some forms of chaos are more stable than constants. There are many forms of mental, metabolic, molecular, and biospheric stasis that have what chaoticians call attractors, and never converge yet maintain a statistical adjacency to their attractor in phase space. For example, using the above list, emotion, glucose, electron shells, and populations show this type of non-constant stability.

That does not help the person using a multilayer-perceptron or convolution kernel hoping to get some ideal set of parameters during training to deploy into a working system in the field. What is desired is reliable, relatively fast, and accurate convergence results.

What practitioners try to avoid is a cost function that diverges. Keep in mind that cost, error, and loss functions are terms used in various contexts in various articles but all represent disparity from some ideal. The function is the mathematical model for that ideal or optimum, not to be confused with the use of the term model in other aspects of AI. Models are used in many places. In SGD, increases in the cost function are indications of instability and reduced likelihood that convergence will occur.

This is the interesting part. Since the cost function represents a surface and the curvature of the surface has curvature (otherwise it could be solved with linear algebra technique without a network) it may not be monotonic. In other words, one cannot determine the mismatch of output values and corresponding labels from the result of the function. The information flow is constrained to be unidirectional.

This means that the existence of local minima can interfere with the search for the global minimum. Mounds, ridges, saddle points, mazes of troughs, and circular ridges may separate the current state of network parameters from those that represent the optimal state sought.

Therein lies the caveat. The cost surface has curvature.

The statement in the question, because of this, is not generally true.

  The loss function is guaranteed to monotonically decrease because the parameters are updated in the negative direction of the gradient.

It is true only in specific cases identified by the theory. In the general case, the cost is not guaranteed to decrease and the algorithm is not guaranteed to converge unless a number of constraints are maintained, few of which normally are.

If not sufficiently attenuated, the next estimation to radically overshoot or undershoot the global minimum. To see how this works, look into chaos simulations using the logistic function. As the next estimate amplification increases, the system first goes into oscillation and then into modes of chaos. In the case of artificial networks, the values in the algorithm used can also exceed the range of IEEE floats and throw an exception.

If one studies the theoretical guarantees in the literature and some of the overview theory such as the PAC (probably approximately correct) Learning Framework, and then try to find a business, marketing, defense, industrial, robotic, or financial application for AI that meets all the constraints required to benefit from the theoretical guarantees, the challenge will become clear. A few easy problems exist. Most require expertise, further investigation, proofs of concept, architecture and design skills, and significant development effort to produce satisfactory results.

Once the results are published with the appropriate references to theoretical foundations and the various design choices presented with hindsight, it looks much easier than it was.


How to choose evaluation functions for features, when network effects are in place (multi-agent systems)? So, I have this huge amount of data, which has 7 vector features (float from 0 to 1). I am trying to build a kind of recommendation system, with a twist (it uses agents and negotiations and narratives; narratives meaning, that there will be temporal and partial order causal link dynamics, or "short term memory").

There is network effects in the data, since the more agents I have, the more potential there will be for match making; in simulation phase I will be just using randomness, instead of real data, but I believe the real dataset will also be more or less random: of course I think there will be nice clusters of correlations for the vectors, but I also want to understand the more general aspects of the problem, instead of just this specific use-case.

The agents will be generating a list of artifacts in co-operation and also evaluate how the generated list of artifacts match their preferences. The agents will be recommending items from their own list (and some times from a list of their related agents) and after each suggestion, there is negotiation.

If there would not be network effect in the data, I would use simple heuristics, which would measure the diff of each feature. However, in network effect world this does not work very well, because it will make many agents unable to co-operate.

I was thinking of something, where some features are preferred features by the agents in different circumstances (depending on the other agents), this will enable some elasticity to the system and IMO avoid the simple problems, of which many recommendation systems suffer (lack of context awareness; humans do not like the likes of other people, we have flexible preferences, which depend upon our mood).

Since this sounds like a very generic problem, it seems like I have just discovered a problem, which has been elegantly solved by someone else from the academia years ago, but I just don't know the name for this concept.

Instead of depending on strict evaluation criteria (only accept diff X per feature), how to choose evaluation function to make the agents more co-operative? I was thinking something, where diff has to be narrow for at least one feature, but may be big for three worst matching features; so that there would be essence of few features and diversity of several.

EDIT: As I have spent some more time with this problem, I have a hunch about which issues might be related:

In real world networks, which depend upon energy efficient co-operation and signaling, scale-free networks seem to emerge; effective Agent-Based Model in network effect scenario should probably form co-operation networks, which follows some kind of Fitness Function.
Due to scale-free networks, it should be assumed, that the Agents using essential features (which are common in data) should be ranked higher and Agents using diverse features (which are rare in data) should be ranked lower; in scale-free networks (like Facebook) it has been observed, that all but 3% of nodes form edges to more popular nodes: this makes signaling efficient.
As a reference, in deep learning Capsule Networks have had some state-of-the-art success with this similar hierarchical "recommendation for upper tiers" mentality (there are much more lower layer features, than upper layer features).

So, I have this idea, that when we use Agent-Based Models for solving problems with network effects, the evaluation function should evaluate Agents skill of passing the problem for more popular Agent, which could delegate it to the best Agent or attempt to solve it on it's own.

In a sense it seems like I am thinking some kind of "Deep Agent-Based Modeling", where it is okay to have network effects on the lowest layer (which would only evaluate the matching of the generated artifacts and their own preferences); the co-operation problem should be solved by a "Coach" layer, which would attempt to prevent situations, where some agents can not play well together.

The evaluation function of the "Coach" layer should be such, that it minimizes the amount of negotiation actions (energy efficient co-operation) required per artifact list generated and also minimizes the signaling distance regarding diversity (less popular artifacts / unorthodox solutions). The "Coach" layer should also be able to reorganize the lists contained by the Agents. The signaling distance can be easily evaluated in my use case, since the dataset has easily available popularity ranking per list artifact list item; the shorter the distance from popular artifact list to the least popular artifacts is, the better.

In practice this means, that I actually want to build composable Agents, which have lists of items (or other Agents), action space for generating artifacts from those items, evaluation functions and costs of specific actions, aggregateable popularity rank (which helps to evaluate delegation). All Agents would have generic default action space, which would help in solving the signaling problem; if the diameter of the co-operation network is too long (not within the scale-free network topology), the system should attempt to create a new layer or reorganize the existing layers (or do both).

All this might be computationally very expensive; however we tested a theory about building a "multi-dimensional" merge-sort algorithm by using entanglement / super dense encoding from quantum information theory and it would seem like it could work and be able to create a hash map, which would allow the Agents to reshape the scale-free network efficiently.

I might build some kind of demo out of this, since all this seems really interesting to me and complements nicely my past hobby projects.

I will not accept this answer until I get some verification, that this is a good answer. However, I will post this in hopes, that someone has already done similar work, gets inspired and can provide a better answer. This answer will probably be edited later, when I get to a more formal level with this.


How important is the choice of the initial state? Is it crucial to always have the same initial (starting) state for Reinforcement Learning, for example, for Q-learning or DQN?
Or it can vary?

The initial state can vary during in both training and use, and how you decide to do this makes very little difference to Q-learning. The important factor is whether all state/action pairs relevant to optimal behaviour can be reached. As there is already randomness in any exploring policy, and in many environments as part of state transitions and reward functions, adding some more at the start is not an issue.

More formally, you can take any existing environment with states $S_1, S_2, S_3 ... S_n$ plus defined actions and rewards. Add a special fixed start state $S_{0}$ with one special action $A_{0}$ allowed. Make the state transition matrix following that action any distribution you like over the other states, and reward $0$. It is clearly a valid MDP, and is identical to your original MDP in terms of value and policy functions for states $S_1, S_2, S_3 ... S_n$. For all intents and purposes to the agent (which gets no meaningful policy choice in $S_0$), it starts in the original MDP in some randomly chosen state.

That is as long as the variations are not made to deliberately exclude some of the state space during training that it would need later, or otherwise used to "attack" the agent and prevent it learning. Q-learning proofs of convergence assume that all state/action pairs are reachable an infinite number of times in the limit of infinite training time. Of course in practice this is never achieved, but clearly if you excluded some important state from ever being seen at the start by choosing to start in a way that it is never reachable, then the agent would never learn about it.

You can use a highly variable start state for training to your advantage. It is a form of exploration, and truly exploring starts (that could start from any state/action pair, so you would want to randomise the first action choice) allow you to skip any further exploration and still guarantee convergence to an optimal policy - i.e. you could learn Q values on-policy using a deterministic policy. This will not necessarily make things converge faster, but does go to demonstrate that random start states are not a barrier to RL.

There are a couple of minor/partial exceptions to "do what you like":

If the environment is otherwise highly deterministic, adding a random start may increase the difficulty in optimising it, as more states will be reachable, even with an optimal policy, therefore the state space over which optimal behaviour needs to be discovered may be larger and take longer to learn.
Policy Gradient methods, like REINFORCE, optimise the policy towards maximising return from a known distribution of start states. So you can vary start state, but you should really stick to a standard start procedure, including a fixed distribution of allowed start states.
Similarly, if you want to report a standard score for your agent, it is common to quote the expected return from the start states. Again that doesn't mean you have a fixed start state, but you should use a fixed distribution of start states to assess your agent.

Many standard environments, like Open AI's Gym environments CartPole and LunarLander, include some amount of randomness in the start state, so that the agent has to solve a more general problem than just generating a fixed sequence of actions that always works from the beginning.


Video Analysis: Providing a success score for a of a student carrying out a specific task I have an AI/ML challenge in relation to video analysis and am unsure where to start.
I am investigating an application that will grade students performance of carrying out a task, based on analysis of a video of them carrying out the task.
The problem has not been sufficiently defined yet but to get a high level idea, imagine  video showing close up of a trainee doctor performing stitches to close a wound. the AI model would be trained using many videos of someone performing the stitches correctly and score the trainee on a number of criteria.
Most frameworks will allow detection of objects but taking a video of a person carrying out a task and assessing their success using an AI/ML model feels a step above regular object analysis.
Assumption is we will create the training material, having professionals video themselves carrying out the task successfully, which will also be graded by other professionals to provide a rubric of scores.
I understand this is not something that can be simply answered but an idea of where to start would be very helpful.

are there specific areas of AI i should investigate?
are there frameworks that can actually do this ( i have not found any)?

Appreciate any advice.
Thank you

You have not described exactly what the tasks will be, but there are some open source libraries for real time pose tracking. For example, OpenPose is one that can be configured to track the body, the hands and the face. However, this is only going to give you predicted pose information for each frame. If the subjects are meant to be doing specific tasks, e.g. picking up objects and moving them, OpenPose won't help and you may need to do further image recognition/analysis or object tracking in order to interpret what is going on.