How to implement word2vec using Tensorflow 2.0 keras API? Since Keras API as defined as layers, how would it be used to implement the word2vec?

Your question is really broad.. But there are many manuals and tutorials which describe how to do it step by step.

My personal favorite would be:

Practical Text Classification With Python and Keras - It has a very detailed explanation of every step of the implementation while remaining practical.

You can also try:

A Word2Vec Keras tutorial - This one focuses more on understanding how word2vec embedding works, but it also contains an implementation.


How to avoid falling into the "local minima" trap? How do I avoid my gradient descent algorithm into falling into the "local minima" trap while backpropogating on my neural network?

Are there any methods which help me avoid it?

There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include:

Probabalistically accepting worse solutions in the hope that this
will jump out of the current basin (like Metropolis-Hastings acceptance in Simulated Annealing).
Maintaining a list of recently-encountered states (or attributes thereof) and not returning
to a recently-encountered one (like Tabu Search).
Performing a random walk of a length determined by the current state of the search (an explicit 'Diversification strategy', e.g. as used in 'Reactive Tabu Search').

See the excellent (and free online) book 'Essentials of Metaheuristics' by Sean Luke for more details on these kind of techniques and some rules of thumb about when and how to use them.


Understanding CNN in a few sentences I don't know if this is the right place to ask this question. If it is not, please tell me and I remove it.

I've just started to learn CNN and I'm trying to understand what they do and how they do it.

I have written some sentences to describe it:

Let's say that CNN is a mathematical function that adjusts its values based on the result obtained and the desired output.

The values are the filters of the convolutional layers (in other
types of networks would be the weights).
To adjust these values there is a backpropagation method as in all networks.

The result obtained is an image of the same size as the original.
In this image you can see the delimited area.
The goal of the network is to learn these filters.
The overfitting may be because the network has learned where the pixels you are looking for are located.
The filters have as input a pixel of the image and return a 1 or a 0.

My doubt is:

In your own words, Have I forgotten something?

NOTE:

This is only one question. The six points above are affirmative sentences, not questions.

There is only one question mark, and it is on my question.

I have to clarify this because someone has closed my previous question because she/he thinks there were more than one question.

To address your description point by point,

There are many types of CNN architectures. You appear to be describing a fully-convolutional neural network built only from convolutional layers which is a very specialized type of network (typically used for segmentation tasks).

Most networks will include other layer types which have trainable parameters too, i.e. batch normalization layers or densely connected layers.
Yes, the training process operates on the same concept of error backpropagation.

If the model is performing a segmentation task (or any task which requires a 1-to-1 mapping of individually classified pixels), then yes its typically the same size.
The raw output of a fully-CNN is typically a probability map where each pixel is the probability between 0 and 1 of it being a member of the positive class. This is normally thresholded at 0.5 to obtain a binary mask which delimits the area of interest.
In essence, yes. But it may be more accurate to say that the goal is to minimise the loss function (reduce the error) which is done by learning optimal filter weights.
Overfitting is not about pixel location. In fact, CNNs are translationally invariant which means that the location of the feature has no impact on the output. There are some types of CNNs which take feature location into account but thats still not standard. Overfitting generally happens when a filter is an exact fit to a pixel patch, for example, the filter is no longer just looking for vertical lines but the exact pixel values of a section of vertical line in a specific image.
No, each convolution of a 3x3 filter takes 3x3 pixels and it is convolved over the whole image which means that the filter output will likely be around the same size of the image depending on stride length, padding and dilation. Have a look at one of my other answers for more details.


What are some (deep) reinforcement learning books for beginners? What are some books on reinforcement learning (RL) and deep RL for beginners?
I'm looking for something as friendly as the head first series, that breaks down every single thing.

Reinforcement Learning: An Introduction by Richard Sutton and Andrew Barto is undoubtedly one of the best books, to begin with. Despite its age, the book is still the canonical introduction to reinforcement learning. It does require some patience, but I think it's very approachable and rigorous at the same time!


Is my GRU model under-fitting given this plot of the training and validation loss? I was running my gated recurrent unit (GRU) model. I wanted to get an opinion if my loss and validation loss graph is good or not, since I'm new to this and don't really know if that is considered underfitting or not

When ever you are buliding a ML Model don't take accuracy seriously(Mistake done by Netflix that cost them alot), you should try to get the hit scores as they will help you to know how many times your model worked on real world users.However, if your model must have to measure the accuracy try it with the RMSE score as it will penalise you more for being more out of the Line. Here is the link for more information on it RMSE
Its hard to predict if its overfitting or underfitting as your graph is vague(for example what does graph lines representing). However, you can solve underfitting by following steps:
1) Increase the size or number of parameters in the ML model.
2)  Increase the complexity or type of the model.
3)  Increasing the training time until cost function in ML is minimised.

For overfitting you can try Regularization methods like weight decay provide an easy way to control overfitting for large neural network models. A modern recommendation for regularization is to use early stopping with dropout and a weight constraint.


Monte Carlo epsilon-greedy Policy Iteration: monotonic improvement for all cases or for the expected value? I was going through university slides and this particular slide is trying to prove that in a Monte Carlo Policy Iteration algorithm using an epsilon-greedy policy, the state Values (V-Values) are monotonically improving.

My question is about the first line of computation.

Isn't this actually the formula for the expected value of Q? It is calculating a probability of occurrence following the policy times actual Q values, then doing the summation.

If that is the case, could you help me understand the relationship between the expected value of Q and the expected value of V ?

Also, if above is true, in a real world scenario, depending on how many episodes we sample and on stochasticity, does it mean that the V values of the new policy could be worse than the V values of the old policy ?

I think this equation answer your question:
$$ q_{\pi^{i}}(s,\pi^{i+1}(s)) = \mathbf{E}[q_{\pi^{i}}(s,\pi^{i+1}(s))] = \sum_{a \in A}\pi^{i+1}(a|s)q_{\pi^{i}}(s,a)$$

value of the Q while taking action from policy $\pi^{i+1}$ and thereafter following the policy $\pi^{i}$ is equal to the expected q value while taking action from policy $\pi^{i+1}$ and thereafter following the policy $\pi^{i}$. And for the second part of your question the answer is:

$$ V_{\pi^{i}}(s) = q_{\pi^{i}}(s,\pi^{i}(s))$$

state Value function following the policy $\pi^{i}$ is the same as action-value function while taking action from policy $\pi^{i}$ and thereafter following the policy $\pi^{i}$.


How are neural networks built in practice? I am curious to know how neural networks are built in practice.
Are they hand coded using weight matrices, activation functions etc OR are there ways to build the NN by mentioning the number of layers, number of neurons in each layer, activation to be used, etc as parameters?
Similar question on training, once built is there a ‘fit’ method or does the training need to be hand coded?
Any reference for understanding these basics will be of great help.

A "software library" is a codebase that includes many commonly used functions/algorithms, you do not need to write the same function again if you "import" the library.
In practice, practitioners/researchers don't often code the nitty gritty details of well-used algorithms, they just use existing libraries such as sklearn or TensorFlow that already has an existing implementation.
In the case of sklearn, ignoring some details, you can create a muti-layer perceptron in this line of code:
>>> clf = MLPClassifier(activation = 'relu', alpha=1e-5, hidden_layer_sizes((5,2))

...in which clf is an "object" that refers to the neural network, MLPClassifier is a function provided by sklearn that "creates" a neural net depending on the parameters provided.
In this case, the parameters provided are that the activation function is ReLU, where the network has two layers, with the first having 5 nodes, and the second has 2 nodes.
All other parameters that are not explicitly provided are set to a default value, in this case, the learning rate, among others.
It should be noted that this is not complete yet, we still have not completed training. Luckily, this is as simple as:
>>> clf.fit(X, y)

Where X and y are some array-shaped data/target set.
I recommend sklearn as i think its more beginner friendly.
https://scikit-learn.org/stable/getting_started.html


How to detect vanishing gradients? Can vanishing gradients be detected by the change in distribution (or lack thereof) of my convolution's kernel weights throughout the training epochs? And if so how?
For example, if only 25% of my kernel's weights ever change throughout the epochs, does that imply an issue with vanishing gradients?
Here are my histograms and distributions, is it possible to tell whether my model suffers from vanishing gradients from these images? (some middle hidden layers omitted for brevity)

Vanishing Gradients can be detected from the kernel weights distribution. All you have to look for is whether the weights are dying down to 0.
If only 25% of your kernel weights are changing that does not imply a vanishing gradient, it might be a factor, but there can be a variety of reasons, such as poor data, loss function used to the optimizer, etc. Kernel's weight not changing only points out that the model is not learning well.
From the histograms, only the conv_2d_2 layer shows any form of vanishing gradients since the numbers are pretty small. But even that seems to be picking after 600 epochs. Usually, a good indicator is to have the mean of the weights in a layer to be closer to 0 and the standard deviation to be closer to 1. So, if this is maintained to a good extent during training, you're good to go.


What is the difference between a Bayesian Network and a Markov Chain? I am trying to understand the difference between a Bayesian Network and a Markov Chain.

When I search for this one the web, the unanimous solution seems to be that a Bayesian Network is directional (i.e. it's a DAG) and a Markov Chain is not directional.

However, often a Markov Chain example is overtime, where the weather today is impacting the weather tomorrow, but the weather tomorrow is not (obviously) impacting the weather today. So I am quite confused how is a Markov Chain not directional?

I seem to be missing something here. Can someone please help me understand?

The main difference between a Bayesian network and a Markov chain is not that a Markov Chain is not directional, it is that the graph of the Bayesian network is not trivial whereas the graph of a Markov chain would be somewhat trivial, as all the previous $k$ nodes would just point to the current node. To illustrate further why this would be trivial, we let each node represent a random variable $X_i$. Then the nodes representing $X_i$ for $ t-k \leq i < t$ would be connected by a directed edge to $X_t$. That is, the edges $(X_i, X_t) \in E$ for $ t-k \leq i < t$ and where $E$ is the set of edges of the graph.

To illustrate this please see the examples below.

Assume that we have a $k$th order Markov chain, then by definition we have $\forall t > k$ $\mathbb{P}(X_t = x | X_0,...,X_{t-1}) = \mathbb{P}(X_t = x | X_{t-k},...,X_{t-1})$.

The main difference between the above definition and the definition of a Bayesian Network is that due to the direction of the graph we can have different dependencies for each $X_t$. Consider the Bayesian Network in the Figure below

We would get that $\mathbb{P}(X_4 = x| X_1, X_2, X_3) = \mathbb{P}(X_4 = x | X_2, X_3)$ and $\mathbb{P}(X_5 = x| X_1, X_2, X_3, X_4) = \mathbb{P}(X_5 = x | X_3)$.

So, the past events that the current random variable depends on don't have to have the same 'structure' in a Bayesian Network as in a Markov Chain.


Back-of-the-envelope machine learning (specifically neural networks) calculations There is a popular story regarding the back-of-the-envelope calculation performed by a British physicist named G. I. Taylor. He used dimensional analysis to estimate the power released by the explosion of a nuclear bomb, simply by analyzing a picture that was released in a magazine at the time.

I believe many of you know some nice back-of-the-envelope calculations performed in machine learning (more specifically neural networks). Can you please share them?

I have one to share. This is no formula, but a general thing I have noticed.

The number of neurons + neurons should be proportionate, in some way, to the complexity of the classification.

Although this is fairly basic and widely known, it has helped me in many times to consider one thing: how many at a minimum does it need?


What algorithm should I use to classify documents? I'd like to build a program that would learn to automatically classify documents. The principle would be that, for each new document I add to the system, it would automatically infer in which category to classify the document. If it doesn't know, I would have to manually enter the category. For each hint I give to the system, the system would learn to refine its knowledge of document kinds. Something similar to face recognition in Picasa, but for documents.

More specifically, the documents would be invoices, and I want to classify them by vendors. Documents could be extracted as text, as image, or both.

Is there some know algorithms for this kind of job?

Up to now, I could think at two possible ways I could do it:

For images, I could add all the images of a given kind together, and record the pixels that are the most common to all images, to create a mask. For a new image, I would compare this mask with the image to determine how similar it is.
For text, I could record the list of words or sentences that are similar to all documents of a given kind.
Finally, I could do a combination of both techniques, for example by converting a PDF document to an image, or an image to text by OCR techniques.

I'm just wondering if I'm approaching the problem the right way. Especially about storing just enough information in the database.

Text approach:

Use LDA (Latent Dirichlet Allocation). LDA is unsupervised. Feed it in corpuses of text from the various documents (i.e. OCR them and feed LDA the results of OCR). It will then cluster them based on the contents of the text (with or without stop words - at your discretion). If possible, you could do a supervised approach of using a bag-of-words and any classifier such as an SVM or Random Forest.

Image Approach:

Use a CNN (convolutional neural network) and train it on images of the various vendors. If you don't have this class discrimination, and can't get it, then use an unsupervised approach such as an autoencoder and then cluster the points in the lower-dimensional autoencoder feature space.


What is the difference between multi-agent and multi-modal systems? The Wikipedia definitions are as follows

Multi-agent systems - A multi-agent system is a computerized system composed of multiple interacting intelligent agents.

Multi-modal interaction - Multimodal interaction provides the user with multiple modes of interacting with a system.

Doesn't providing a user with multiple modes of interacting with a system, assuming all modalities interact with each other to give final output (some sort of fusion mechanism for example), make it a multi-agent system?

If not, what is the difference between multi-modal and multi-agent systems and, monolithic and uni-modal systems?

An agent is a concept, which can have slightly different meanings, abilities or instantiations depending on the context. However, given the purpose of this website, I will use and refer to the definition of agent commonly used in artificial intelligence.

  An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.

For more details regarding the definition of an agent in AI, see my answer to the question What is an agent in Artificial Intelligence?.

A multi-agent system is a system composed of multiple agents that interact with an environment. See Multi-Agent Systems: A survey (2018) fore a more exhaustive overview of the field.

Multimodal interaction (MI) refers to the interaction with a system (e.g. a computer) using multiple modalities (e.g. speech or gestures). For example, we usually can interact with a laptop using a keyboard and a touchpad (or mouse), so the keyboard and the touchpad are the two different modalities that are used to interact with the computer. MI could thus be considered a sub-field of human-computer interaction.

Conceptually, an agent could be associated with each modality provided by a multimodal system, so a system that provides multimodal interaction could indeed be a multi-agent system. See, for example, A Multi-Agent based Multimodal System Adaptive to the User’s Interaction Context (2011).


How to achieve recognition of postures and gestures? I'd like to investigate the possibility of achieving similar recognition as it's in Honda's ASIMO robotp.22 which can interpret the positioning and movement of a hand, including postures and gestures based on visual information.

Here is the example of an application of such a recognition system.

Image source: ASIMO Featuring Intelligence Technology - Technical Information (PDF)

So, basically, the recognition should detect an indicated location (posture recognition) or respond to a wave (gesture recognition), like a Google car does it (by determining certain patterns).

Is it known how ASIMO does it, or what would be the closest alternative for postures and gestures recognition to achieve the same results?

Just to add some discourse; this is actually an incredibly complex task, as gestures (aka kinematics) function as an auxiliary language that can completely change the meaning of a sentence or even a single word. I recently did a dissertation on the converse (generating the correct gesture from a specific social context & linguistic cues). The factors that go into the production of a particular gesture include the relationship between the two communicators (especially romantic connotations), the social scenario, the physical context, the linguistic context (the ongoing conversation, if any), a whole lot of personal factors (our gesture use is essentially a hybrid of important individuals around us e.g. friends & family, and this is layered under the individual's psychological state). Then the whole thing is flipped around again when you look at how gestures are used completely differently in different cultures (look up gestures that swear words in other cultures for example!). There are a number of models for gesture production but none of them captures the complexity of the topic.
Now, that may seem like a whole lot of fluff that is not wholly relevant to your question, but my point is that ASIMO isn't actually very 'clever at this. AFAIK (I have heard from a visualization guy that this is how he thinks they do it) they use conventional (but optimized) image recognition techniques trained on a corpus of data to achieve recognition of particular movements. One would assume that the dataset consists of a series of videos/images of gestures labelled with that particular gesture (as interpreted by a human), which can then be treated as a machine learning problem. The issue with this is that it does not capture ANY of the issues I mentioned above. Now if we return to the current best interpretation of gesture that we have (that it is essentially auxiliary language in its own right), ASIMO isn't recognizing any element of language beyond the immediately recognizable type, 'Emblems'.
'Emblems' are gestures that have a direct verbal translation, for example in English-based cultures, forming a circle with your thumb and index finger translates directly to 'OK'. ASIMO is therefore missing out on a huge part of the non-verbal dictionary (illustrators, affect displays, regulators and adapters are not considered!), and even the part that it is accessing is based on particular individuals' interpretations of said emblems (e.g. someone has sat down and said that this particular movement is this gesture which means this), which as we discussed before is highly personal and contextual. I do not mean this in criticism of Honda; truth be told, gesture recognition and production are in my opinion one of the most interesting problems in AI (even if it's not the most useful) as it is a compound of incredibly complex NLP, visualization and social modelling problems!
Hopefully, I've provided some information on how ASIMO works in this context, but also on why ASIMO's current process is flawed when we look at the wider picture.


What is a convolutional neural network? Given that this question has not yet been asked on this site, although similar questions have already been asked in the past (e.g. here or here), what is essentially a convolutional neural network (CNN)? Why are they heavily used in image recognition applications of machine learning?

(Of course, similar questions have been asked in the past and there are many sites, papers, video lessons, online that explain how CNNs work, but I think it's still a good idea to have a reference answer that hopefully will give you the main ideas behind CNNs.)
A convolutional neural network (CNN) is a neural network that performs the convolution (or cross-correlation) operation typically followed by some downsampling (aka pooling) operations.
The convolution operation comes from the mathematical equivalent operation, which is an operation that takes as inputs two functions $f$ and $h$ and produces another function $g$ as output, which is often denoted as $f \circledast h = g$, where $\circledast$ is the convolution operation (or operator).
In image processing, the convolution is used to process images in multiples ways. For example, it is used to remove noise (e.g. the convolution of a noisy image with a Gaussian kernel produces a smoother image) or to compute derivatives of the image, which can then be used e.g. to detect edges or corners, which are usually the main features of an image. For example, the Harris corner detector makes use of the partial derivatives (in the $x$ and $y$ direction) of the image to find corners (or interest points) in the image.
In the context of CNNs, $f$ is the image (in the case of the first layer of the CNN) or a so-called feature map (in the case of hidden layers), $h$ is the kernel (aka filter) and $g$ is also a feature map. (In this answer, I explain these concepts, including how an image can be viewed as a function, more in detail, so I suggest that you read it).
Here's an illustrative example of how the convolution works.

where the $\color{blue}{\text{blue}}$ grid is the image, the $\color{gray}{\text{gray}}$ grid is the kernel and $\color{green}{\text{green}}$ grid is the feature map (i.e. the output of the convolution between the image and the kernel). The white squares are around the images represent the padding that is added around the image so that the convolution operation produces a feature map of a specific dimension.
Essentially, the convolution is a series of dot products between the kernel and different patches (or parts) of the image. The convolution can even be represented as matrix multiplication, so the name convolution shouldn't scare you anymore, if you are familiar with dot products and matrix multiplications.
As opposed to many operations in image processing where the kernels are typically fixed, in the context of CNNs, the kernels are learnable parameters, i.e. they change depending on the loss function, so they are supposed to represent functions that when convolved with their respective input functions are useful to extract meaningful information (i.e. features) to solve the task the CNN is being trained to solve. For this reason, the convolution is often thought of as an operation that extracts features from images. In fact, the output of the convolution, in the context of CNNs, is often called feature map. Moreover, a CNN is typically thought of as a data-driven feature extractor for the same reason.
There are many different variations of the standard convolution operation. For example, there are transposed convolutions, dilated convolutions, etc., which are used to solve slightly different problems. For example, the dilated convolution can be used to increase the receptive field of an element of a feature map (which is typically a desirable property for several reasons). There are also upsampling operations which are particularly useful in the context of image segmentation. These different convolutions and their arithmetic are explained very well in the paper A guide to convolution arithmetic for deep learning by Vincent Dumoulin and Francesco Visin. Here's also the associated Github repository that contains all the images in this paper that show how the convolution operations work (from which I also took the gif above).
To conclude, CNNs are very useful to process images and extract features from them because they use convolution operations (and downsampling and upsampling operations). They can be used for image (or object) classification, object detection (i.e. object localization with a bounding box + object classification), image segmentation (including semantic segmentation and instance segmentation), and possibly many other tasks where you need to learn a function that takes as input images and needs to extract information from those images to get someone high-level (but also low-level) output (e.g. the name of the object in the image), given some training data.


What are the differences between SARSA and Q-learning? From Sutton and Barto's book Reinforcement Learning (Adaptive Computation and Machine Learning series), are the following definitions:

To aid my learning of RL and gain an intuition, I'm focusing on the differences between some algorithms. I've selected Sarsa (on-policy TD control) for estimating Q ≈ q * and Q-learning (off-policy TD control) for estimating π ≈ π *.

For conciseness I'll refer to Sarsa (on-policy TD control) for estimating Q ≈ q * and Q-learning (off-policy TD control) for estimating π ≈ π * as Sarsa and Q-learning respectively.

Are my following assertions correct?

The primary differences are how the Q values are updated.

Sarsa Q value update:
$ Q ( S, A ) ← Q ( S, A ) + α [ R + \gamma Q ( S ′ , A ′ ) − Q ( S, A ) ] $

Q-learning Q value update:
$ Q ( S, A ) ← Q ( S, A ) + α [ R + \gamma \max_a Q ( S ′ , a ) − Q ( S, A ) ] $

Sarsa, in performing the td update subtracts the discounted Q value of the next state and action, S', A' from the Q value of the current state and action S, A.
Q-learning, on the other hand, takes the discounted difference between the max action value for the Q value of the next state and current action S', a.  Within the Q-learning episode loop the $a$ value is not updated, is an update made to $a$ during Q-learning?

Sarsa, unlike Q-learning, the current action is assigned to the next action at the end of each episode step. Q-learning does not assign the current action to the next action at the end of each episode step

Sarsa, unlike Q-learning, does not include the arg max as part of the update to Q value.

Sarsa and Q learning in choosing the initial action for each episode both use a "policy derived from Q", as an example, the epsilon greedy policy is given in the algorithm definition. But any policy could be used here instead of epsilon greedy?
Q learning does not utilise the next state-action pair in performing the td update, it just utilises the next state and current action, this is given in the algorithm definition as  $ Q ( S ′ , a ) $ what is $a$ in this case ?

The main difference between the two is that Q-learning is an off policy algorithm. That is, we learn about an policy that is different to the one we choose to make actions. To see this, lets look at the update rule.

Q-Learning

$$Q(s,a) = Q(s,a) + \alpha (R_{t+1} + \gamma \max_aQ(s',a) - Q(s,a))$$

SARSA

$$Q(s,a) = Q(s,a) + \alpha (R_{t+1} + \gamma Q(s',a') - Q(s,a))$$

In SARSA we chose our $a'$ according to what our policy tells us to do when we are in state $s'$, so the policy we are learning about is also the policy that we choose to make our actions.

In Q-learning, we learn about the greedy policy whilst following some other policy, such as $\epsilon$-greedy. This is because when we transition into state $s'$ our TD-target becomes the maximum Q-value for whichever state we end up in, $s'$, where the max is taken over the actions.

Once we have actually updated our Q-function and we need to choose an action to take in $s'$, we do so from the policy we are using to generate our actions from -- thus we are learning about the greedy policy whilst following some other policy, hence off policy. In SARSA when we move into $s'$ our TD-target is chosen by the Q-value for the state we transition into and then the action we would choose based on our policy.

  Within the Q-learning episode loop the $a$ value is not updated, is an update made to $a$ during Q-learning?

It will be, because the policy we use to choose our actions is ensured to explore sufficiently around all of the state-action pairs and so it is guaranteed to be encountered at some point.

  Sarsa, unlike Q-learning, does not include the arg max as part of the update to Q value.

It is not an $\arg \max$, it is a $\max$. This is defined as
$$\max_x f(x) = \{f(x) | \forall y\; : f(y) \leq f(x) \}$$

  Sarsa, unlike Q-learning, the current action is assigned to the next action at the end of each episode step. Q-learning does not assign the current action to the next action at the end of each episode step

Kind of - the action that you chose for your TD-target in SARSA becomes the next action that you consider in the next step of the episode. This is natural because essentially you are in state $s$, you take action $a$ and observe a new state $s'$, at which point you can use your policy to see which action you will take, call this $a'$, and then perform the SARSA update, and then execute that action in the environment.

  Sarsa and Q learning in choosing the initial action for each episode both use a "policy derived from Q", as an example, the epsilon greedy policy is given in the algorithm definition. But any policy could be used here instead of epsilon greedy?

Yes, any policy can be used although you want to choose a policy that allows sufficient exploration of the state-space.

  Q learning does not utilise the next state-action pair in performing
  the td update, it just utilises the next state and current action,
  this is given in the algorithm definition as $Q(S',a)$ what is $a$ in
  this case ?

in the algorithm it actually has $\max_a Q(S',a)$, which if you refer back to my earlier definition of what the $\max$ operator does, should answer this question.


In sequence-to-sequence, why is the output of the decoder used as its input? The basic seq-2-seq model consists of 2 parts: a recurrent encoder that compresses a sequence to a vector and decoder that unrolls the vector into the output sequence:

Why is the output, w, x, y, z of the decoder used as its input? Shouldn't the hidden state of the RNN from the previous timestamps be enough?

In seq2seq they model the joint distribution of whatever char/word sequence by decomposing it into time-forward conditionals:
$\begin{align*}
p(w_1,w_2,...,w_n) =& \ p(w_1)*p(w_2|w_1) * \ ... \ * p(w_n|w_1,...,w_{n-1}) \\
 =& \ p(w_1)*\prod_{i=2}^{n}p(w_i|w_{<i})
\end{align*}
$

This can be sampled by samping each of the conditional is ascending order. So thats exactly what theyre trying to imitate. You want the second output dependant on the sampled first output, not its distribution.

This is why the hidden state is NOT good for modeling this setup because it is a latent representation of the distribution, not a sample of the distribution.

Note: In training they use ground truth as input by default because its working under the assumption the model shouldve predicted the correct word and if it didnt the gradient of the word/char level loss will reflect that (this is called teacher forcing and has a multitude of pitfalls)


What is the goal of the model and is the training data relevant to that? The model that we develop in artificial intelligence.What is its purpose,and what training data is relevant to it.

Model means you can say a Prototype we make regarding to our task.
As we first train our model on some observed or you can say bench-marked data ; called as TRaining phase of model.

Then we apply that model to our problem (test data) you can say in order to evaluate how  much well you have trained your model.
Training data we use related to our task or use bench marked data for any model of AI.
hope you find this answer useful .


What is different in each head of a multi-head attention mechanism? I have a difficult time understanding the "multi-head" notion in the original transformer paper. What makes the learning in each head unique? Why doesn't the neural network learn the same set of parameters for each attention head? Is it because we break query, key and value vectors into smaller dimensions and feed each portion to a different head?

The reason each head is different is because they each learn a different set of weight matrices $\{ W_i^Q, W_i^K, W_i^V \}$ where $i$ is the index of the head. To clarify, the input to each attention head is the same. For attention head $i$:
$Q_i(x) = x W_i^Q \\
K_i(x) = x W_i^K \\
V_i(x) = x W_i^V \\
attention_i(x) = softmax \left(\frac{Q_i(x) K_i(x)^T}{\sqrt{d_k}} \right) V_i(x)$.
Notice that the input to each head is $x$ (either the semantic + positional embedding of the decoder input for the first decoder layer, or the output of the previous decoder layer). More info
The question as to why gradient descent learns each set of weight matrices $\{ W_i^Q, W_i^K, W_i^V \}$ to be different across each attention head is very similar to "Is there anything that ensures that convolutional filters end up the same?", so maybe you might find the answer there helpful for you:

No, nothing really prevents the weights from being different. In practice though they end up almost always different because it makes the model more expressive (i.e. more powerful), so gradient descent learns to do that. If a model has n features, but 2 of them are the same, then the model effectively has n−1 features, which is a less expressive model than that of n features, and therefore usually has a larger loss function.


How to alpha-beta pruning to expectiminimax I have this problem above and I'm trying to think of how to apply alpha-beta pruning to the above. The algorithm states that on you're opponents turn the (expecti turn) you just return the lowest value, but does that mean you apply the probabilities to those values? So for the far left you'd get 2 as the largest value then multiply that by 0.5, but then that set's $\\beta$ in the expecti node to $0.5*2=1$ and when it goes into the branch to the right it's comparing values without the probabilities applied to it when updating $\beta$.

alpha-beta-prunning algorithm is using for improve performance and reject the options, which not consist the condition - or it's possible to set some factors of probability to take into consideration or not.
This algorithm work on tree structure - and if there are a lot of levels (10-20) - It allows you to eliminate paths - which will logically not be used - saving memory and computing resources.
In this particular case - for finding the minimum value it works like this:
First branch:

Go to B
Go to D - and there is 2 and 3 - so return the min 2
Go From B to E - and choose 5 - the minimal value in B points is actually 3 - so there isn't need for checking the next - cause everything below E, will be higher than D (3)

Second branch:

Go to
Go to F - and check 0 and 1
If C have 1 - than not necessary to go to G - cause 1 is the smaller.

The given sequence is a simplification - and in the case of this algorithm there are also layers - min & max

Source:
https://www.javatpoint.com/ai-alpha-beta-pruning#:~:text=Key%20points%20about%20alpha%2Dbeta,values%20to%20the%20child%20nodes.

Implementation, however, is a more complex matter - unless we do copy and paste :-)
https://gist.github.com/exallium/1446104/5109388cfc21578f555dcac0ba54da680326af7b


Measuring Soft Skill How can we attach numerical value to the measurement of soft skill? Is there any book, site or materials that can teach me how to do this?

I want to know if there is any way we can measure soft skill as listed  here and say a person  has (over 100) 70% or 60% in critical thinking ability or his critical thinking ability is over 72% and be accurate with the score.

I spent some time on the Future of Work in UK government. If you want to look at the impact of AI on work you need to have good definitions of soft skills that you can measure and you can track progress against.

So the definitions sometimes need to change and decomposed. Take "Coaching" from that list: there are now bots (such as Wysa.ai) that coach you. You can measure coaching as a skill by the effectiveness of the participants achieving goals.

Things really get murky when you talk about "Creativity" -- I wrote a piece on AI impact of creativity in a similar vein.


How to train a logical XOR with reinforcement learning? After reading an excellent BLOG post Deep Reinforcement Learning: Pong from Pixels and playing with the code a little, I've tried to do something simple: use the same code to train a logical XOR gate.

But no matter how I've tuned hyperparameters, the reinforced version does not converge (gets stuck around -10). What am I doing wrong? Isn't it possible to use Policy Gradients, in this case, for some reason?

The setup is simple:

3 inputs (1 for bias=1, x, and y), 3 neurons in the hidden layer and 1 output.
The game is passing all 4 combinations of x,y to the RNN step-by-step, and after 4 steps giving a reward of +1 if all 4 answers were correct, and -1 if at least one was wrong.
The episode is 20 games

The code (forked from original and with minimal modifications) is here: https://gist.github.com/Dimagog/de9d2b2489f377eba6aa8da141f09bc2

P.S. Almost the same code trains XOR gate with supervised learning in no time (2 sec).

Reinforcement learning is used when we know the outcome we want, but not how to get there which is why you won't see a lot of people using it for classification (because we already know the optimal policy, which is just to output the class label). You knew that already, just getting it out of the way for future readers!

As you say, your policy model is fine - a fully connected model that is just deep enough to learn XOR.  I think the reward gradient is a little shallow - when I give a reward of +1 for "3 out 4" correct and +2 for "4 out of 4", then convergence happens (but very slowly).


How does the DQN loss from td_targets against q_values make sense? Why td_loss is calculated from (td_targets against q_values)?

Why I am lost is because:

q_values is just the probability of action. It does not have a reward and discount.
td_targets does have rewards + discounts * next_q_values. Somemore next_q_values is next state.

How both td_targets and q_values can minus (or Huber or MSE) to get lost work?

td_error = valid_mask * (td_targets - q_values)
td_loss = valid_mask * td_errors_loss_fn(td_targets, q_values)

td_loss = valid_mask * td_errors_loss_fn(td_targets, q_values)

First of all, DQN is off-policy learning. That means, you are following the behavior policy(epsilon greedy policy) but still learning about the optimal policy or target policy(greedy policy). Td_target in DQN is the estimation of our current state's optimal action-value function independent of the policy we are following (since we are picking next state's action-value from target policy) and q_values(as you referred) is what you get following the behavior policy. While using this kind of update, you are improving both the behavior policy and the target policy.


Is fuzzy logic connected to neural networks? Fuzzy logic is typically used in control theory and engineering applications, but is it connected fundamentally to classification systems?

Once I have a trained neural network (multiple inputs, one output), I have a nonlinear function that will turn a set of inputs  into a number that will estimate how close my set of given inputs are to the trained set.

Since my output number characterizes "closeness" to the training set as a continuous number, isn't this kind of inherently some sort of fuzzy classifier?

Is there a deep connection here in the logic, or am I missing something?

They are unrelated.

There is a possibility of interpreting fuzzy values as probabilities, but strictly speaking they are different: fuzzy values are vague, while probabilities reflect likelihood (see Wikipedia entry for Fuzzy Logic)

While rolling a particular number on a six-side die has a probability of $1 \over 6$, a roll can actually only ever have one outcome.

A fuzzy value "quite old" can simultaneously be member of a number of fuzzy sets with different degrees of membership, eg "young" with 0.001, "adolescent" with 0.1, "old" with 0.4, "ancient" with 0.7. Unless it is "defuzzified", it is simultaneously contained in all the sets.

Defuzzyfication is a way of interpreting the result of a series of fuzzy operations and finding the set that best matches, but it is not a clearly defined process such as picking a random number according to a set of probabilities (or rolling the die).

I am not sure that the sum of all fuzzy set membership values of any given fuzzy value has to add up to 1.0; whereas this condition has to hold for probabilities.

[EDIT: to clarify - probabilities are not a set; I refer here to all possible outcomes of a random event which have a certain probability of being realised. The sum of all possible event probabilities has to be 1.0]

One alternative interpretation for your application could be the confidence that the input set is identical to the training set. Which could be a fuzzy value if you wanted to do something else with it, eg by combining it with other fuzzy variables.


Should intelligent AI be granted the same rights as humans? One of the most crucial questions we as a species and as intelligent beings will have to address lies with the rights we plan to grant to AI.

  This question is intended to see if a compromise can be found between conservative anthropocentrism and post-human fundamentalism: a response should take into account principles from both perspectives.

Should, and therefore will, AI be granted the same rights as humans or should such systems have different rights (if any at all) ?

Some Background

This question applies both to human-brain based AI (from whole brain emulations to less exact replication) and AI from scratch.

Murray Shanahan, in his book The Technological Singularity, outlines a potential use of AI that could be considered immoral: ruthless parallelization: we could make identical parallel copies of AI to achieve tasks more effectively and even terminate less succesful copies.

Reconciling these two philosophies (conservative anthropocentrism and post-human fundamentalism), should such use of AI be accepted or should certain limitations - i.e. rights - be created for AI?

This question is not related to Would an AI with human intelligence have the same rights as a human under current legal frameworks? for the following reasons:

The other question specifies "current legal frameworks"
This question is looking for a specific response relating to two fields of thought
This question highlights specific cases to analyse and is therefore expects less of a general response and more of a precise analysis

I'll attempt to analyze a couple of different perspectives.
1. It is artificial
Synonyms: insincere, feigned, false.
There is the idea that any "intelligence" created by humanity is not actually intelligent and, by definition, it is not possible. If you look at the structure of the human brain and compare it to anything humans have created thus far, none of the computers come close to the power of the brain. Sure they can hold data, or recognize images, but they cannot do everything the human brain can do as fast as the brain can do it with as little space as the brain occupies.
Hypothetically if a computer could do that, how do we determine its intelligence? The word artificial defines that the intelligence is not sincere or real. This means that even if humanity creates something that appears intelligent, it has simply become more complex. It is a better fake, but it is still fake. Any money not printed by the government is by definition counterfeit. Even if someone finds a way to make an exact duplicate, that doesn't mean that the money is legal tender.
2. Misuse of power
If an AI is given rights and chooses to exercise those rights in a way that agrees with its creator's views, possibly through loyalty to its creator, or through hidden motives, then anyone with the capabilities to create such an AI would become extremely powerful by advancing their own beliefs through the creation of more AIs. This might also lead to the ruthless parallelization that you mentioned, but with (even more) selfish goals in mind.
If this were not the case, and an AI could be created to be neutral with free will and uncontrollable by humans, then perhaps an AI could be given rights. But I do not believe this would ever be the case. With great power comes great responsibility. Even with free will, a true AI would most likely end up serving humanity, because humans have control of the plugs and the electricity, the Internet, the software, and the hardware. The social implications of this for the AI are not promising. It's not even just the ongoing control of these resources that is the issue. Whoever creates the software and hardware for the AI would have special knowledge. If fine adjustments were made, specific individuals would undoubtedly hold sole control of the AI, as adjustments could be made to the code in such a way that the AI behaves the same except under specific circumstances, and then when something goes wrong (assuming the AI has its own rights), then the AI would be blamed rather than the programmers who were responsible.
3. Anthropocentrism
In order for humanity to get away from anthropocentrism, we would have to become less selfish when it comes to humans, first. Until we can solve every existing social problem within humanity, there is no reason to believe that we could cease thinking of humanity as more important than created machines. After all, supposing there were an almighty God that created humanity, wouldn't the humans always be beneath God, never to be equals? We can't fully understand our own biology. If an AI were created, would it be able to understand its makings in the same way its creators would? Being the creator would give humanity a sense of megalomania. I do not think that we would relinquish our dominion over our own technological creations. That is as unlikely to happen as the wealthiest of humanity willingly giving the entirety of their money, power, and assets to the poorest of humanity. Greed prevents it.
4. Post-human fundamentalism
Humans worship technology with their attention, their time, and their culture. Some movies show technologically advanced robots suppressing mankind to the point of near-extinction. If this were the case and humanity were in danger of being surpassed by its technology, humanity would not stand idly and watch its extinction at the hands of its creation. Though people may believe superior technology could be created, in the event we reached such a point humanity would fight to prove the opposite, as our survival instincts would take over.
5. A balance?
Personally, I do not think the technology itself it actually possible, though people may be deceived into thinking such an accomplishment has been achieved. If the technology were completed, I still think that anthropocentrism will always lead, because if humanity is the creator, humanity will do its best to ensure it retains control of all technological resources, not simply due to fear of being made obsolete, but also because absolute power corrupts absolutely. Humanity does not have a good historical record when it comes to morality. There is always a poor class of people. If wealth were distributed equally, some people would become lazy. There is always injustice somewhere in the world, and until we can fix it (I think we cannot), then we will never be able to handle the creation of true AI. I hope and think that it will never be created.


Can digital computers understand infinity? As a human being, we can think infinity. In principle, if we have enough resources (time etc.), we can count infinitely many things (including abstract, like numbers, or real).

For example, at least, we can take into account integers. We can think, principally, and "understand" infinitely many numbers that are displayed on the screen. Nowadays, we are trying to design artificial intelligence which is capable at least human being. However, I am stuck with infinity. I try to find a way how can teach a model (deep or not) to understand infinity. I define "understanding' in a functional approach. For example, If a computer can differentiate 10 different numbers or things, it means that it really understand these different things somehow. This is the basic straight forward approach to "understanding".

As I mentioned before, humans understand infinity because they are capable, at least, counting infinite integers, in principle. From this point of view, if I want to create a model, the model is actually a function in an abstract sense, this model must differentiate infinitely many numbers. Since computers are digital machines which have limited capacity to model such an infinite function, how can I create a model that differentiates infinitely many integers?

For example, we can take a deep learning vision model that recognizes numbers on the card. This model must assign a number to each different card to differentiate each integer. Since there exist infinite numbers of integer, how can the model assign different number to each integer, like a human being, on the digital computers? If it cannot differentiate infinite things, how does it understand infinity?

If I take into account real numbers, the problem becomes much harder.

What is the point that I am missing? Are there any resources that focus on the subject?

I think this is a fairly common misconception about AI and computers, especially among laypeople. There are several things to unpack here.
Let's suppose that there's something special about infinity (or about continuous concepts) that makes them especially difficult for AI. For this to be true, it must both be the case that humans can understand these concepts while they remain alien to machines, and that there exist other concepts that are not like infinity that both humans and machines can understand. What I'm going to show in this answer is that wanting both of these things leads to a contradiction.
The root of this misunderstanding is the problem of what it means to understand. Understanding is a vague term in everyday life, and that vague nature contributes to this misconception.
If by understanding, we mean that a computer has the conscious experience of a concept, then we quickly become trapped in metaphysics. There is a long running, and essentially open debate about whether computers can "understand" anything in this sense, and even at times, about whether humans can! You might as well ask whether a computer can "understand" that 2+2=4. Therefore, if there's something special about understanding infinity, it cannot be related to "understanding" in the sense of subjective experience.
So, let's suppose that by "understand", we have some more specific definition in mind. Something that would make a concept like infinity more complicated for a computer to "understand" than a concept like arithmetic. Our more concrete definition for "understanding" must relate to some objectively measurable capacity or ability related to the concept (otherwise, we're back in the land of subjective experience). Let's consider what capacity or ability might we pick that would make infinity a special concept, understood by humans and not machines, unlike say, arithmetic.
We might say that a computer (or a person) understands a concept if it can provide a correct definition of that concept. However, if even one human understands infinity by this definition, then it should be easy for them to write down the definition. Once the definition is written down, a computer program can output it. Now the computer "understands" infinity too. This definition doesn't work for our purposes.
We might say that an entity understands a concept if it can apply the concept correctly. Again, if even the one person understands how to apply the concept of infinity correctly, then we only need to record the rules they are using to reason about the concept, and we can write a program that reproduces the behavior of this system of rules. Infinity is actually very well characterized as a concept, captured in ideas like Aleph Numbers. It is not impractical to encode these systems of rules in a computer, at least up to the level that any human understands them. Therefore, computers can "understand" infinity up to the same level of understanding as humans by this definition as well. So this definition doesn't work for our purposes.
We might say that an entity "understands" a concept if it can logically relate that concept to arbitrary new ideas. This is probably the strongest definition, but we would need to be pretty careful here: very few humans (proportionately) have a deep understanding of a concept like infinity. Even fewer can readily relate it to arbitrary new concepts. Further, algorithms like the General Problem Solver can, in principal, derive any logical consequences from a given body of facts, given enough time. Perhaps under this definition computers understand infinity better than most humans, and there is certainly no reason to suppose that our existing algorithms will not further improve this capability over time. This definition does not seem to meet our requirements either.
Finally, we might say that an entity "understands" a concept if it can generate examples of it. For example, I can generate examples of problems in arithmetic, and their solutions. Under this definition, I probably do not "understand" infinity, because I cannot actually point to or create any concrete thing in the real world that is definitely infinite. I cannot, for instance, actually write down an infinitely long list of numbers, merely formulas that express ways to create ever longer lists by investing ever more effort in writing them out. A computer ought to be at least as good as me at this. This definition also does not work.
This is not an exhaustive list of possible definitions of "understands", but we have covered "understands" as I understand it pretty well. Under every definition of understanding, there isn't anything special about infinity that separates it from other mathematical concepts.
So the upshot is that, either you decide a computer doesn't "understand" anything at all, or there's no particularly good reason to suppose that infinity is harder to understand than other logical concepts. If you disagree, you need to provide a concrete definition of "understanding" that does separate understanding of infinity from other concepts, and that doesn't depend on subjective experiences (unless you want to claim your particular metaphysical views are universally correct, but that's a hard argument to make).
Infinity has a sort of semi-mystical status among the lay public, but it's really just like any other mathematical system of rules: if we can write down the rules by which infinity operates, a computer can do them as well as a human can (or better).


What is the difference between tree search and graph search? I have read various answers to this question at different places, but I am still missing something.

What I have understood is that a graph search holds a closed list, with all expanded nodes, so they don't get explored again. However, if you apply breadth-first-search or uniformed-cost search at a search tree, you do the same. You have to keep the expanded nodes in memory.

There is always a lot of confusion about this concept, because the naming is misleading, given that both tree and graph searches produce a tree (from which you can derive a path) while exploring the search space, which is usually represented as a graph.
Differences
Firstly, we have to understand that the underlying problem (or search space) is almost always represented as a graph (although the underlying graph may not contain cycles, so it may represent a tree). So, the difference is not whether the problem is a tree (a special kind of graph), or a general graph!
The distinction is, instead, how we are traversing the search space (represented as a graph) to search for our goal state and whether we are using an additional list (called the closed list) or not.
So, the basic differences are

In the case of a graph search, we use a list, called the closed list (also called explored set), to keep track of the nodes that have already been visited and expanded, so that they are not visited and expanded again.

In the case of a tree search, we do not keep this closed list. Consequently, the same node can be visited multiple (or even infinitely many) times, which means that the produced tree (by the tree search) may contain the same node multiple times.

Advantages and disadvantages
The advantage of graph search obviously is that, if we finish the search of a node, we will never search it again. On the other hand, the tree search can visit the same node multiple times.
The disadvantage of graph search is that it uses more memory (which we may or may not have) than tree search. This matters because graph search actually has exponential memory requirements in the worst case, making it impractical without either a really good search heuristic or an extremely simple problem.
So, there is a trade-off between space and time when using graph search as opposed to tree search (or vice-versa).
Conclusion
So, the difference between tree search and graph search is not that tree search works on trees while graph search works on graphs! Both can work on trees or graphs (but, given that graphs are a generalization of trees, we can simply say that both work on graphs, either trees or not) and both produce a tree!
Notes
The definitions of tree search and graph search given above are based on the definitions given in section 3.3 (page 77) of the book Artificial Intelligence: A Modern Approach (3rd edition), by Stuart J. Russell and Peter Norvig, which is the de-facto standard book in artificial intelligence, so these definitions are applicable in the context of artificial intelligence (which is also the context of this site). Here you have the pseudocode of tree and graph searches (provided by P. Norvig).
However, note that, occasionally, people may use the term tree search to refer to a tree traversal, which is used to refer to a search in a search tree (e.g., a binary search tree or a red-black tree), which is a tree (i.e. a graph without cycles) that maintains a certain order of its elements. This is another reason for having different definitions of a tree search and to think that a tree search works only on trees.


Multi label Classification using Keras I am trying to build a Multi label classification model, having dataset with different input numerical values and specific label...

Eg:

Value Label

35   X

35.8 X

29   Y

29.8 Y

39   AA

41   CB

So depending on input numerical value the model should specify its label....please note that the input values won't necessarily follow exact dataset values....eg dataset has 35 and 34.8 as input values with X as label. So if model has 35.4 as input label, the X should be output label. Bottom line is that the output label is based on range of input values instead of fixed one..

Can anyone help me with quick solution (example Jupyter notebook will be highly appreciated)

For a simple multi layer perceptron, you can refer to here:
https://www.kaggle.com/fchollet/simple-deep-mlp-with-keras

This is a great resource for kerad multi input label classification. Also, here is a few reminders for implementing such classification model.

One hot encoding

In the sample data you provided, it seems like you are using raw numbers as input. This will create unnecessary complications for the model. A better approach will be to encode them into a vector with 0 in all except one index with 1. The index will be the numerical value you are encoding. For example, 1 will be 01000000... Till end of your range of input, and 2 will be 001000 and so on. If you have decimal values, then this approach isn't for you. However you should still scale down your input to something smaller like 0-1.

output encoding

The output seems to be text, which cannot be done and also complicates the problem. Instead you can also use one hot encoding for the output. For example, let A be 0, B be 1, C be 2... AA be 27.... Until your list ends. And then use one hot encoding to encode them to vector of 0 and 1.

Hope I can help you.


How can I encode states where the environment consists of multiple identical elements, but each is characterised by different features? I am quite new to Deep Reinforcement Learning, and I'm trying to define states in a Reinforcement Learning problem. The environment consists of multiple identical elements, and each one of them is characterized by different features of the same type. In other words, let us say we have $e_0$, $e_1$, and $e_2$. Then, suppose that each one is characterized by features $f_0$ and $f_1$, where $f_0$ belongs to $[0, 1]$, and $f_1$ belongs to $\{0, 1, 2, 3, 4, 5\}$. Then, $e_0$ will have some value for the features $f_0$ and $f_1$, and the same goes for $e_1$ and $e_2$.
How can I encode such states?
Can I simply vectorize such state by concatenating the different features of each element obtaining $[f_{0e_0}, f_{1e_0}, f_{0e_1}, f_{1e_1}, f_{0e_2}, f_{1e_2}]$, or should I use a convolutional architecture of some sort?

For general advice about state representation, you could check my answer to How to define states in reinforcement learning? - this does not cover your specific issue, but may help with other details such as whether to one-hot-encode and/or scale your discrete features. Also you will want to assess whether any state vector you construct actually contains enough data for reinforcement learning to work.

Assuming that is all good, then your approach here:

  Can I simply vectorize such state by concatenating the different features of each element obtaining $[f_{0e_0}, f_{1e_0}, f_{0e_1}, f_{1e_1}, f_{0e_2}, f_{1e_2}]$

should work with a generic feed-forward neural network with two or more layers. As this is simple to construct, then it is a reasonable place to start just considering development time. If nothing else it can be a benchmark against other representations and model architectures that you may want to try.

  should I use a convolutional architecture of some sort?

That will depend on any structure inherent in your environment. The interesting thing about your description of the environment is the repetition of meaning in the values. Each entity $e_i$ appears multiple times, and the units of feature $f_{0e_0}$ will be the same as $f_{0e_1}$.

An unfortunate consequence of using a flattened vector is that the information about these repeated entities and their units is lost to the system. Any important details due to the entities being essentially the same of this must be "rediscovered" by the approximator training with many examples. Although neural networks can generalise function results, this does not apply to generalising across different parts of the input unless you choose an architecture designed to help with that.

Typically you would use a RNN architecture (such as LSTM) if key information is in the sequence order of entities, or a CNN architecture if key information is in local patterns between neighbouring entities. By "key information" I mean whether useful relationship to a value function or optimal policy depends on those factors. This might be difficult to assess for some environments with mixed data types and complex meanings for the entity ids - so you may have to experiment with multiple architectures to figure out what factor is more important. However, it is also possible you have some insight based on known good policies, so can head more directly to a promising feature representation and architecture.

Learning may benefit if instead of ordering your entities arbitrarily by identity in the vector, that you sort them according to some factor that is relevant to the goals of your agent. I have used this is in a colour matching environment where an agent had to hit a target with a "current" colour selection whilst avoiding incorrect colours - sorting target entities by whether they matched colour and then by angular distance to the agent's aim enabled the agent to learn its task far more efficiently. This sorting may help independently of architecture choice - i.e. you don't need to be using a RNN or CNN to see a benefit with this approach.

Another thing worth considering, but probably not possible here because of the continuous feature $f_0$, is whether you can invert the representation so that instead of being based on entities with properties, it is a view over property space populated with entities in certain positions. A CNN processed over a small enough "property space" instead of "entity space" may well perform better - this is why board game RL tends to treat the board positions as defining feature vector indexes which represent which entities are present on them, which is a very sparse structure that can represent far more states than ever occur in real games. The over-specified space works because it can be matched to architectures that generalise efficiently over it.

In general, you need to consider different types of symmetry within the system - especially what is meaningful about the id number you have assigned to each entity. If the entity id is not meaningful, or is secondary to some other factor, you have some flexibility in changing the representation to make it meaningful and use architectures that take advantage of how you have injected that domain knowledge into the q function approximator. If the id is meaningful, think how could you best represent the difference between entities with different ids in your environment?


How much data do I need to collect? I'm trying to optimize a combination of 8 cards with 64 card characters. No repeats and order doesn't matter.

n!/(n!(n-r)!) = 4,426,165,368 combinations

I have everything set up, including the data scrapper. But I don't know how many games my machine needs to learn from to start seeing patterns. For example, in 14,000 games analyzed, only 834 decks had a certain character.

Analyzing the next 7 cards from 834 decks is 621,216,192

So I guess I need more data before reliable patterns emerge... but how much data? Thank you and god bless

Sorry but no one can give you a number.

As stated already in the comment by Manngo, the general rule: is as much as you can. I have also seen the 10x number being thrown around, see here. According to this "rule", you need  roughly 10 times as many examples as there are degrees of freedom in your model.

How much data you will need is something you can only answers to through empirical investigation. The amount of data you need to collect is influenced by the complexity of the data, the dimensionality of the data and the algorithm you intend to train(although this can be controlled somewhat through regulirazation, see this good blog post by Jake Vanderplas for more information).

This question  might be of interest to you.


Is a genetic algorithm an example of artificial intelligence? Since human intelligence presumably is a function of a natural genetic algorithm in nature, is using a genetic algorithm in a computer an example of artificial intelligence? If not, how do they differ? Or perhaps some are and some are not expressing artificial intelligence depending upon the scale of the algorithm and what it evolves into?

An ability that is commonly attributed to intelligence is problem solving.
Another one is learning (improving itself from experience).
Artificial intelligence can be defined as "replicating intelligence, or parts of it, at least in appearance, inside a computer" (dodging the definition of intelligence itself).
Genetic algorithms are computational problem solving tools that find and improve solutions (they learn).

Thus, genetic algorithms are a kind of artificial intelligence.

Regarding scale, I don't see it as an important factor for defining G.A. as A.I or not. The same way we can simply classify different living forms as more or less intelligent instead of just saying intelligent or not intelligent.

Finally, let's just make an important distinction: our brains are the product of natural selection, but the brains themselves don't use the same principle in order to achieve intelligence.


Can Q-learning be used to find the shortest distance from each source to destination? Is it possible to form a table that will have simply the shortest distance from each source to destination using q learning?

If not, suggest any other learning algorithm.

Welcome to AI.SE Adarsh!

This is fairly simple to do with Q-Learning.

If you assign a reward equal to -1 for each location other than the goals, and 0 for the goals, then Q-learning with no discount factor will learn the length of the shortest path from each location to the goal. Alternatively, you can use a discounting factor, and then invert the equation on the learned values of the Q function to obtain the number of steps.

Hope this helps!


How is neural architecture search performed? I have come across something that IBM offers called neural Architecture search. You feed it a data set and it outputs an initial neural Architecture that you can train.

How is neural architecture search (NAS) performed? Do they use heuristics or is this meta machine learning?

If you have any papers on NAS, I would appreciate if you can provide a link to them.

You could say that NAS fits into the domain of Meta Learning or Meta Machine learning.

I've pulled the NAS papers from my notes, this is a collection of papers/lectures that I personally found very interesting. It's sorted in rough chronological descending order, and *** means influential / must read.

Quoc V. Le and Barret Zoph are to good authors on the topic.

The Evolved Transformer
Exploring Randomly Wired Neural Networks for Image Recognition
GRAPH HYPERNETWORKS FOR NEURAL ARCHITECTURE SEARCH
Backprop Evolution
Progressive Neural Architecture Search
*** DARTS: Differentiable Architecture Search
*** Efficient Neural Architecture Search via Parameter Sharing - ENAS
*** Progressive Neural Architecture Search
AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
Automatic Machine Learning - Prof. Frank Hutter
Google Brain - Neural Architecture Search - Quoc Le
*** Regularized Evolution for Image Classifier Architecture Search
Autostacker: A Compositional Evolutionary Learning System
Generating Neural Networks with Neural Networks
Finding Competitive Network Architectures Within a Day Using UCT
Neuroevolution: A different kind of deep learning
Evolving Deep Neural Networks
Pieter Abbeel: Deep Learning-to-Learn Robotic Control
*** SMASH: One-Shot Model Architecture Search through HyperNetworks


Creating a self learning Mario Kart game AI? I will be undertaking a project over the next year to create a self learning AI to play a racing game, currently the game will be Mario Kart 64.

I have a few questions which will hopefully help me get started:

What aspects of AI would be most applicable to creating a self learning game AI for a racing game (Q-Learning, NEAT etc)
Could a ANN or NEAT that has learned to play Mario Kart 64 be used to learn to play another racing game?
What books/material should i read up on to undertake this project?
What other considerations should i take throughout this project?

Thank you for your help!

What aspects of AI would be most applicable to creating a self learning game AI for a racing game (Q-Learning, NEAT etc)

In general, you are looking at a problem that involves sequential decision making, in a machine learning context.

If you are wanting to build an agent that can learn by receiving screen images, then NEAT cannot scale to that complexity directly. Although there might be clever combinations of deep learning and evolutionary algorithms you could apply, the most heavily explored and likely successful solution will be found in Deep Reinforcement Learning. Algorithms like DQN, A3C, A2C, PPO . . . there are dozens to consider, but all are based around agents using samples of experience to update functions that measure either the "value" of acting in a certain way (a policy) or estimate the best policy directly.

  Could a ANN or NEAT that has learned to play Mario Kart 64 be used to learn to play another racing game?

Within limits, yes. You will have built a system that takes pixel inputs, and outputs controller messages. If you re-start training from scratch on any compatible N64 game (with same screen resolution and same controller outputs) there is a chance it could learn to play that new game well. As other driving games on the N64 are a subset of all games, and more similar to each other than, say, a scrolling shooter or adventure game, then an agent that can successfully learn one will likely learn another too.

It is unlikely that a Mario Kart agent will immediately be good at another game without re-training. The visual and control differences will probably be too much. You could try though. An interesting experiment would be to take your trained agent, or some part of it, and see if starting with that improves learning time on a new game. This is called Transfer Learning.

  What books/material should i read up on to undertake this project?

You will need at least the first parts of an introduction to Reinforcement Learning. If your goal is to aim to just make the agent, then you can skip theory-heavy parts, but within limits the more theory you understand, the easier it will be to change code features towards getting something working. I can suggest the following:

The book Sutton & Barto, Reinforcement Learning: An Introduction - this covers theory behind basic RL, and the online draft is free
A course of lectures on Reinforcement Learning from UCL, by David Silver. The video lectures cover same topics as the Sutton & Barto book.
The book Deep Reinforcement Learning Hands-On, which you will need to buy. As the title suggests, it covers more practical aspects of building RL systems. It uses Python and PyTorch. As RL has become more popular in recent years, there are plenty of other similar books - I have no reason to recommend this one over them, but I am reading it at the moment, and the quality is adequate.

  What other considerations should i take throughout this project?

Before you get started, you should know this could be an ambitious project, requiring a lot of compute time using currently-available toolkits. You need to think ahead a little, as you will be faced with these decisions:

Is it more important to you to make a working bot on this problem, or more important to understand the underlying theories so that you understand what is going on. You will need to do at least a little of both, but you can go strongly in either direction:

There are enough pre-built learning agents available on the internet that you might have a successful project by learning just enough to wire up a copy of the game, and then letting the learning agent do its job. Would this be enough for you, would you feel that you had solved your problem?
There is plenty of educational material about RL and how the maths behind it works. If you are interested in understanding that, and perhaps generating your own ideas for improvements to current algorithms, then you need to study that material harder. However, this may lead you away from your Mario Kart player goal, and you may never actually solve it, because far simpler problems that require less computing power are still actually interesting academically.

In addition, I can think of the following:

Building your Mario Kart player is a long-term goal. You will need to start with simpler agents in order to understand the techniques that you hope to apply to Mario Kart.
You may need more compute power to solve this game than you have available.
You will need to solve the issue of automating control of a N64 emulator. There is at least an existing emulator Mupen64Plus - I do not know whether it will be adequate for you, but at least one person has attempted to wrap this for automated learning, in the gym-mupen64plus project.


What does the formula $1-\sum_i(e_i-a_i)^2$ mean in this NEAT Python API? I have looked at the documentation for the NEAT Python API found here, where it's written

The error for each genome is $1-\sum_i(e_i-a_i)^2$

I have not yet learned calculus, so I can't understand this formula. So, can someone please explain what the calculation means?

$$1-\sum_i(e_i-a_i)^2$$
$\sum$ - there just means sum. It is the greek letter for S.  You can rewrite the above formula as
$$1 -[(e_1 - a_1)^2+(e_2-a_2)^2+(e_3-a_3)^2+\ldots ]$$
$\sum$ just helps us avoid writing dozens of $+$ signs. Read more here.
What they are doing here is taking the difference of expected value $e_1$ and the actual value $a_1$ for the 1st example, and so on. The difference can be positive ($e_1 > a_1$) or negative ($e_1 < a_1$), so usually we square the difference to make it positive number.
The rest is there in the docs. Try putting in concrete imagined values for $a_i$ and $e_i$.


Using AI to enhance customer service I'm trying to find out how AI can help with efficient customer service, in fact call routing to the right agent. My usecase is given context of a query from a customer and agents' expertise, how can we do the matching?

Generally, how is this problem solved? What sub-topic within AI is suitable for this problems? Classification, recommender systems, ...? Any pointers to open-source projects would be very helpful.

This sounds to me like a use case for a chatbot. You would have different intents reflecting the types of user queries that your system can respond to. The intent matching can be done by pattern matching, machine learning (classification), or a combination of the two (hybrid). You can then use the chatbot to ask clarification questions or elicit more information to identify which live agent would be the best person to take over the call. Essentially each live agent would have a list of intents plus added information (such as geographical area etc) which you then compare against the data from the caller to find the best match.

If this is a voice call you'd need to put an ASR system at the front of the pipeline. Chatbots can usually do live-agent handover to then pass control to a human agent at any time in the conversation.

[Disclaimer: I work for a company that operates in exactly that area and whose system works as described above]


Are sentences from the same document independent and identically distributed? I am trying to build an LSTM model to generate Shakspeare-like poems. I have data set $\{s_1, s_2, \dots, s_m \}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1, w_2, \dots, w_n \}$.

I am wondering:  Are different $s_i$ ($i=1, \dots, m$) independent and identically distributed samples (IID)? Are $w_i$ ($i=1, \dots, n$) within each sentence the IID?

The sentences coming from the same document, author, etc., are unlikely to be independent, that is, the occurrence of a sentence $s_i$ in a certain document $d$ is likely correlated with the occurrence of another sentence $s_j$. If they are not independent, they can also not be independent and identically distributed (which is a stronger condition). The same can be said for words in the same sentence.


What's the commercial usage of "image captioning"? If "image captioning" is utilized to make a commercial product, what application fields will need this technique? And what is the level of required performance for this technique to be usable?

If "image captioning" is utilized to make a commercial product, what application fields will need this technique?

There are several important use case categories for image captioning, but most are components in larger systems, web traffic control strategies, SaaS, IaaS, IoT, and virtual reality systems, not as much for inclusion in downloadable applications or software sold as a product. These are a few examples.

Provision of captions to reduce the number of user operations (keying and selection) required to post a movie or image to improve posting volume of sticky media and therefore improve the position of a web site or page in terms of human interest
Provision of captions to provide HTML header and alt attribute content to improve search engine scoring of page for search terms related to the content of the movie or image
Testing to see if the content of an image submitted for posting matches the criteria intended by the owner or stake holder for the posting space on the basis of caption rather than directly through CNN categorization

The first two are usually monetized in ways such as these.

Improvements in online purchase daily volume

Acquisition of contact lists with consumer or business interests as fields in the list for marketing purposes

Additional draw of web traffic to enhance ad impression revenue

What is the level of required performance can this technique be usable?

Performance is in terms of these system qualities.

Cost of computing resources per caption generated
Accuracy of caption generated (which may be quantified in more than one way)
Reliability of generation (since the AI component should produce a rating of confidence with the caption string or produce a null or empty string based on a hyper-parameter confidence threshold

The quantification of cost, accuracy, and reliability is business dependent. Some may have a nasty negative effect on the business if the caption is wrong or missing. Others use cases may not.
In some cases the average revenue generated by the caption's presence is already known and known to be small, which requires that the CNN run time and computing resource requirements must be kept below that. In other cases, a Fortune 500 company CTO said, "Do it and send me the budget needed." In such cases the budget may be, for all practical purposes, unconstrained, not that a system that wastes time and resources is ever desirable, even if only for energy conservation reasons.


Value Iteration failing to converge to optimal value function in Sutton-Barto's Gambler problem In Example 4.3:Gambler's Problem of Sutton and Barto's book whose code is given here.
In this code the value function array is initialized as np.zeros(states) where states $\in[0,100]$ and the value function for optimal policy which is returned after solving it with value iteration is same as the one given in the book, but, if we only change the initialization of the value function in the code, suppose to np.ones(states) then the optimal value function returned changes too, which means that the value iteration algorithm converges in both the cases but to different optimal value functions,but two different optimal value function is impossible in a MDP. So why is the value iteration algorithm not converging to optimal value function?
PS: If we change the initialization of value function array to -1*np.random.rand(states), then the converged optimal value function also contains negative numbers which should be impossible as rewards>=0, hence value iteration fails to converge to optimal value function.

So, naturally, if you've observed something that contradicts the theoretical properties of Value Iteration, something's wrong, right?
Well, the code you've linked, as it is, is fine. It works as intended when all the values are initialized to zero. HOWEVER, my guess is that you're the one introducing an (admittedly very subtle) error. I think you're changing this:
state_value = np.zeros(GOAL + 1)
state_value[GOAL] = 1.0

for this:
state_value = np.ones(GOAL + 1)
state_value[GOAL] = 1.0

So, you see, this is wrong. And the reason why it's wrong is that both GOAL (which is 100 in the example) and 0 must have an immutable and fixed values, because they're terminal states, and their values are not subject to estimation. The value for GOAL is 1.0, as you can see in the original code. If you want initial values other than 0, then you must do this:
state_value = np.ones(GOAL + 1)
state_value[GOAL] = 1.0
state_value[0] = 0

In the first case (changing the initial values to 1) what you were seeing was, essentially, an "I don't care policy". Whatever you do, you'll end with a value of 1. In the second case, with the random values, you saw the classic effects of "garbage in, garbage out".


What are some best practices when trying to design a reward function? Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward function, and how should you avoid them? What information from your problem should you take into consideration when going about it?
Let us presume that our environment is fully observable MDP.

Designing reward functions
Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three outcomes: win (good), loss (bad), or draw (neutral). So, you could reward the agent with $+1$ if it wins the game, $-1$ if it loses, and $0$ if it draws (or for any other situation).
However, in certain cases, the specification of the reward function can be a difficult task [1, 2, 3] because there are many (often unknown) factors that could affect the performance of the RL agent. For example, consider the driving task, i.e. you want to teach an agent to drive e.g. a car. In this scenario, there are so many factors that affect the behavior of a driver. How can we incorporate and combine these factors in a reward function? How do we deal with unknown factors?
So, often, designing a reward function is a trial-and-error and engineering process (so there is no magic formula that tells you how to design a reward function in all cases). More precisely, you define an initial reward function based on your knowledge of the problem, you observe how the agent performs, then tweak the reward function to achieve greater performance (for example, in terms of observable behavior, so not in terms of the collected reward; otherwise, this would be an easy problem: you could just design a reward function that gives infinite reward to the agent in all situations!). For example, if you have trained an RL agent to play chess, maybe you observed that the agent took a lot of time to converge (i.e. find the best policy to play the game), so you could design a new reward function that penalizes the agent for every non-win move (maybe it will hurry up!).
Of course, this trial-and-error approach is not ideal, and it can sometimes be impractical (because maybe it takes a lot of time to train the agent) and lead to misspecified reward signals.
Misspecification of rewards
It is well known that the misspecification of the reward function can have unintended and even dangerous consequences [5]. To overcome the misspecification of rewards or improve the reward functions, you have some options, such as

Learning from demonstrations (aka apprenticeship learning), i.e. do not specify the reward function directly, but let the RL agent imitate another agent's behavior, either to

learn the policy directly (known as imitation learning [8]), or
learn a reward function first to later learn the policy (known as inverse reinforcement learning [1] or sometimes known as reward learning)

Incorporate human feedback [9] in the RL algorithms (in an interactive manner)

Transfer the information in the policy learned in another but similar environment to your environment (i.e. use some kind of transfer learning for RL [10])

Of course, these solutions or approaches can also have their shortcomings. For example, interactive human feedback can be tedious.
Reward shaping
Regarding the common pitfalls, although reward shaping (i.e. augment the natural reward function with more rewards) is often suggested as a way to improve the convergence of RL algorithms, [4] states that reward shaping (and progress estimators) should be used cautiously.  If you want to perform reward shaping, you should probably be using potential-based reward shaping (which is guaranteed not to change the optimal policy).
Further reading
The MathWorks' article Define Reward Signals discusses continuous and discrete reward functions (this is also discussed in [4]), and addresses some of their advantages and disadvantages.
Last but not least, the 2nd edition of the RL bible contains a section (17.4 Designing Reward Signals) completely dedicated to this topic.
Another similar question was also asked here.


What is "conditioning" on a feature? On page 98 of Jet Substructure at the Large Hadron Collider: A Review of Recent Advances in Theory and Machine Learning the author writes;

  Redacted phase space: Studying the distribution of inputs and the
  network performance after conditioning on standard physically-inspired
  features can help to visualize what new information the network is
  using from the jet. Training the network on inputs that have been
  conditioned on specific values of known features can also be useful
  for this purpose.

I cannot find other references to conditioning features. What does that mean?

This is conditioning in the sense of conditional probability. The idea is that the authors have some "standard physically-inspired features". They are splitting the data up into bins based on the values of these features, and then training a model for each bin. They are then examining the differences between the models. Usually this is done to learn something about the benefits of using the different features, and about the relationships between features and outputs.


Is creating dataset only by augmentation a bad practice? I wonder if creating data set only by augmentation base images is a bad practice.
I mean the situation when you have to train net to predict really simple patterns, for example printed-like digits. And all digits from specific group looks basically the same, for example all one's look the same and so on. The only difference is rotation/translation etc. in the image.
Is it bad way to create data set by taking digit image and randomly rotate, translate and maybe erode/dilate it?
My intuition tells me that something's wrong with that approach, but I cannot find any reason why it should be wrong.

Data augmentation is usually rotating, cropping and translating images. And this makes sense if your network could meet these kind of images.
If I take a digit recognition like LeNet, it is useless to complicate the task of the network by forcing it to learn rotated digits, which could lead to a more complex architecture and training and less accuracy in the task. Another example I could think of is human pose recognition (openpose project). As we humans usually stands with our feet on the bottom of the image and the head on top, openpose project didn't use rotation of images on the dataset.
So I would say data augmentation is a great tool (especially when we lack data), but I would only use it when the augmented data could be met when performing the task. If the digits are always oriented and placed on the middle of the image, it doesn't make much sense to use translations and rotations on the data set to augment it unless we really lack data.
In your example, it does make sense to me to create the dataset using data augmentation.


Can DQN announce it has things in its hand in a card game? More informations on the card game I'm talking about are in my last question here: DQN input representation for a card game

So I was thinking about the output of the q neural network and, aside from which card to play, I was wondering if the agent can announce things.

Imagine you have the current hand: 2, 4, 11, 2 (The twos are different card type).
When you're playing the game and you get dealt a hand like this, you have to announce that you have the same number twice (called Ronda) or thrice (called Tringa) before anyone plays a card on the table. Lying about it gets you a penalty.

Could a DQN handle this? I don't know if adding "Announcing a Ronda/Tringa" as an action would actually help. I mean, can this be modeled for the  NN or should I just automate this and spare the agent having to announce it everytime.

The simplest thing to do when you make you first implementation of the agent, is to automate decisions like this, in order to keep representations and decisions simple.

However, if you want to explore tactics surrounding declaration, then I think the following applies:

There should be an initial round of actions where the agent may get to decide whether or not to declare a Ronda, based on the cards it holds. These will be different action choices to playing cards, so you would need to alter your action representation to include those choices. Only allow action choices which are valid, so if it is not valid to declare a Ronda or a Tringa when a player does not have one, then the player does not get to make that choice.
You may want to add a state feature "has a Ronda" and "has a Tringa" for the agent's player, to help with the action decision.
You should also add a state feature for each player according to whether they declared a Ronda or a Tringa.
Rather than have the agent learn to detect a lie, given your comment that all cards are played so it is easy to tell (there are only 8 cards total in play by the end), then I would just assume lies are automatically found out and include that in the game engine. In other words, the penalty is always paid.
The interesting question is then whether withholding the declaration can give a tactical advantage when the round is being played (because your opponent knows less about your hand), and whether that advantage offsets the inevitable penalty. This might not be true in your card game, but could be true in games with similar choices.

  Could a DQN handle this?

A DQN is maybe going to struggle with partial information in this game. The opponent's cards are hidden from the network, but could have a non-random influence on the opponent's choices of action. It is possible that you will need to investigate agents that can solve POMDPs to get the best player.

I don't know for certain though. It depends on how much tactical advantage there is in having concealed cards, or how much that is just luck that plays out much the same whether you know the opponent's cards or not. If there is strategy, and ways to determine/guess what the opponent holds based on your cards and their actions so far, this is more like a POMDP.


How to verify classification model trained on classification dataset on a detection dataset for classification purpose? I am working on a problem that involves two tasks - detection and classification. There is no single dataset for both tasks. I am training two models, separate on detection dataset and another on classification dataset. I use the images from the detection dataset as input and get classification predictions on top of detected bounding boxes.
Dataset description :

Classification - Image of the single object (E.g. Car) in the center with a classification label.
Detection - Image with multiple objects (E.g. 4 Cars) with bounding box annotations.

Task - Detect objects(e.g. cars) from detection datasets and classify them into various categories.
How do I verify whether the classification model trained on the classification dataset is working on images from detection dataset? (In terms of classification accuracy)
I cannot manually label the images from the detection dataset for individual class labels. (Need expert domain knowledge)
How do I verify my classification model?
Is there any technique to do this ? Like domain transfer or any weakly-supervised method ?

To verify the accuracy of the classification stage, you will need labeled images with a single car.
To train and verify accuracy of the detection stage and full system, you can:

in the datasets with images with multiple cars, manually, mark the image rectangles that contains one car.
from previous, split the image in one or more ones, each one containing a single car.
pass each one of the previous image with a single car to the classification stage (that means assume classification has 100% accuracy). Record its outputs (labeled cars).
now, from output of steps 1) and 3), you can produce labeled images with multiple cars. Use it to train the detector and verify full system accuracy.