What is the definition of the hinge loss function? I came across the hinge loss function for training a neural network model, but I did not know the analytical form for the same.
I can write the mean squared error loss function (which is more often used for regression) as
$$\sum\limits_{i=1}^{N}(y_i - \hat{y_i})^2$$
where $y_i$ is the desired output in the dataset, $\hat{y_i}$ is the actual output by the model, and $N$ is the total number of instances in our dataset.
Similarly, what is the (basic) expression for hinge loss function?

The hinge loss/error function is the typical loss function used for binary classification (but it can also be extended to multi-class classification) in the context of support vector machines, although it can also be used in the context of neural networks, as described here.
The hinge loss function is defined as follows
$$
\ell(y) = \max(0, 1-t \cdot y) \tag{1}\label{1},
$$
where

$t = \{-1, 1\}$ is the label (so, if your labels are in the set $\{0, 1 \}$, you will have to first map them to $\{-1, 1\}$)
$y$ is the output of the classifier (e.g. in the context of the linear SVM, $y=\mathbf{w} \cdot \mathbf{x}+b$, where $\mathbf{w}$ and $b$ are the parameter of the hyper-plane)

This means that the loss in equation \ref{1} is always non-negative. If you're familiar with the ReLU, this loss should look familiar to you. In fact, their plots are very similar.
For more details, you probably should start with the related Wikipedia article, then maybe one of the many machine learning books that covers support vector machines, for example, Pattern Recognition and Machine Learning (2006) by  Christopher Bishop, chapter 7 (page 325).


How to deal with the addition of a new state to the environment during training? Let's say we have a dynamic environment: a new state gets added after 2000 episodes have been done. So, we leave room for exploration, so that it can discover the new state.
When it gets to that new state, it has no idea of the Q values, and, since we're past 2000 episodes, our exploration rate is very low. What happens if try to exploit when all Q values are 0?

There are several ways to tackle this, although exploration is definitely not a solved problem yet ;)
In general, I believe the right thing to do here is to measure the uncertainty of your policy or Q-value estimates and use that to construct some sort of exploration bonus. An intuitive example is given in Exploration by Random Network Distillation. They make two randomly initialized neural nets, one of which is never updated. At every transition, they feed transition data through both neural nets and use the difference in output between them as an estimate of uncertainty, and this quantity is added to the reward. Then they update the modifiable neural net towards the other one. This way, on a completely novel transition, the two neural nets will likely have very different outputs so the reward will be augmented a lot. Of course, this will hopefully encourage the agent to explore.


How have robots developed and how sophisticated might they be in the future? We all know how robots are getting more and sophisticated and more interesting

What is the future of robotics in relation to AI, how and how will AI work with robotics in improving to affect our world in a positive way.

The future of robotics

Companies like Google are doing restless research in robotics whereby machine learning algorithms are applied to empower self driving cars well equipped with computer vision, etc.

The future is so exciting in that it will be an era where artificially intelligence machines can rewrite their own source code and constantly improve upon themselves at an exponential rate -- ASI: artificial super intelligence.

Right now, robots like Sophia are using machine learning algorithms to answer or respond to human interactions.


Is Deep Learning the repeated application of Linear Regression? Is Deep Learning the repeated application of Linear Regression?

A neural network can be reduced to a linear regression model only if we use linear activation functions (i.e. $\sigma(x) = x$), and only if we do not use any  neural network specific techniques such as convolution, residuals, etc., as shown below:
$\text{neural network}(x) = \sigma_n(W_{n} \sigma_{n-1}(W_{n-1}\dots\sigma_1(W_1 x + b_1) + \dots + b_{n-1}) + b_n) \\
= W_n (W_{n-1} \dots (W_1 x + b_1) + \dots + b_{n-1}) + b_n \\
= \left( W_n W_{n-1} \dots W_1 \right) x + \left( W_n W_{n-1} \dots W_2 \right) b_1 + \left( W_n W_{n-1} \dots W_3 \right) b_2 + \dots + W_n b_{n-1} + b_n \\
= W_z x + b_z$
where $W_z = \displaystyle \prod_i W_i$ is a weight matrix and $b_z$ is some vector constant.
This follows the linear regression model form $y = Wx + b$, where $W$ is the weight matrix and $b$ is the vector constant. As a result, you can analytically solve for $W_z$ and $b_z$ using linear regression techniques, and no longer need gradient descent.
Note that for this to work well as a linear regression, you need to check for the OLS data assumptions, such as making sure the regressors have no collinearity, the residuals have no heteroskedasticity, there's no auto-regression, and that the errors are roughly normally distributed (more info). Deep neural networks with non-linear activation functions do not require these assumptions since they are universal approximators, although checking for some conditions may help make the task easier to predict (more info*).
*Note - this link talks specifically about time series data with neural network, but the same concept applies to any task in general.
The neural network specific techniques, such as ReLu, convolutions, residuals, etc., is what allows the network to learn non-linear relationships, and therefore make neural networks something more than just repeated applications of linear regression.


How should I define the reward function in the case of Connect Four? I'm using RL to train a Network on the game Connect4. It learns quickly that 4 connected pieces is good. It gets a reward of 1 for this. A zero is rewarded for all other moves.
It takes quite a time until the AI tries to stop the opponent from winning.
Is there a way this could be further reinforced?
I thought about giving a negative reward for the move played before the winning move. Thinking about this, I came to the conclusion that this is a bad idea. There'll be always a looser (except for ties), therefore there always be a last move from the losing player. This one hasn't to be a bad one. Mistakes could have been made much earlier.
Is there a way to improve this awareness of opponents? Or does it just have to train more?
I'm not perfectly sure if the rewards will propagate back in a way that encourages this behavior with my setup.

The classic working reward scheme for two player zero sum games (i.e. if I win, you lose and vice versa) is simply:

+1 for a win
0 for a draw
-1 for a loss

These rewards should be associated with the last move made by the player before the game is resolved.

  I thought about giving a negativ reward for the move played before the winning move.

That is what is normally done.

  Thinking about this I came to the conclusion that this is a bad idea. There'll be always a looser (except for ties), therefor there always be a last move from the loosing player. This one hasn't to be a bad one. Mistakes could have been made much earlier.

It's OK, what the agent is learning is the value of a combination of state and action. It will, eventually, correctly associate the negative reward with a chain of board states that decrease in value and that it will then try to avoid.

This scenario of delayed rewards, not necessarily related to the choice of the current action, is handled correctly by reinforcement learning algorithms. It is still a tough problem though - a more delayed and difficult to predict reward makes for a tougher RL problem. However, even basic learners, such as Q-learning, can eventually solve this.

To get better performance, you might want to look into adding planning algorithms to the RL part.


Are there tools to help clean a large dataset so that it only contains faces? I have a fairly large dataset consisting of different images with and without persons that I want to use for a project.

The problem is that I only want the pictures that contain faces, and it is best if there is only a crop of the face.

I already looked at Facenet and Openface, but I thought that there must be a simpler already trained solution just to sort the dataset so I can get started with my own project.

I don't know of a tool but you could write a simple script to detect faces and crop it. It's quite simple with the Haar cascade in openCV to detect faces and use inbuilt functions to crop your image based on the size of the detected face.

Hope that helps !


On-policy state distribution for episodic tasks on Sutton & Barto, page 199 In Sutton & Barto's "Reinforcement Learning: An Introduction", 2nd edition, page 199, they describe the on-policy distribution for episodic tasks in the following box:

I don't understand how this can be done without taking the length of the episode into account. Suppose a task has 10 states, has probability 1 of starting at the first state, then moves to any state uniformly until the episode terminates. If the episode has 100 time steps, then probability of the first state is proportional to $1 + 100\times 1/10$; if it has $1000$ time steps, it will be proportional to $1 + 1000\times 1/10$. However, the formula given would make it proportional to $1 + 1/10$ in both cases. What am I missing?

Let's first assume that there is only one action so that $\pi(a|s) = 1$ for every state - action pair which simplifies the discussion.
Now let's consider a case with 100 time steps, 10 states and uniform distribution for starting state $s_0$ with $h(s_0) = 1$. The result would be
\begin{align}
\eta(s_0) &= 1 + \sum_{i = 0}^9 \eta(s_i) \cdot p(s_0|s_i) =\\
&= 1 + \sum_{i = 0}^9 10 \cdot \frac{1}{10} = 11
\end{align}
Now let's consider a case with 1000 time steps where other settings are the same as in the first case.
\begin{align}
\eta(s_0) &= 1 + \sum_{i = 0}^{9} \eta(s_i) \cdot p(s_0|s_i) =\\
&= 1 + \sum_{i = 0}^{9} 100 \cdot \frac{1}{10} = 101
\end{align}
In the first case
\begin{equation}
\mu(s_0) = \frac{11}{9\cdot 10 + 11} = 0.1089
\end{equation}
and in the second case you have
\begin{equation}
\mu(s_0) = \frac{101}{9\cdot 100 + 101} = 0.1009
\end{equation}
so it looks like you are correct that $\mu(s)$ depends on the length of the episode, but they didn't really say that it doesn't. Obviously as the length of the episode increases so will the number of times a certain state was visited so you could say that formula implicitly depends on the number of time steps. If $h(s_i)$ is equal for every state then results would be the same in both cases regardless of number of time steps. Also, as the number of possible states grows very large, as it usually is in real problems, the results would be approaching each other as the number of states grows.


What are the most important components in the algorithm of a minimum AI? If someone wants to develop a basic AI with some code modules,Let us say the AI just has to provide an action when stimulated in a certain situation based on its previous understanding of situations.

I can think of at least 3 of such components:

Real-time Understanding/Learning: Using Deep Learning/ConvNets, Supervised/Unsupervised.
Logical Decision-Making: Calculating the results of various decisions when applied on current situation based on previous understanding and choosing the most appropriate one logically.
Action/Reaction: Acting precisely in the new situation according to the decision-made.

Any ideas?

Some good places to start would be cognitive architectures and as mentioned in another answer intelligent agents. The question is broad but you definitely want to look into planning & decision making. You might also want to check out the L5 and L6 layers of Hierarchical Temporal Memory (As in Nupic) as it relates to feedback, behavior and attention.

If I were you I'd aim for more cognitive solutions (I realize that term is a bit ambiguous itself when we talk about machines). There's also new AI initiative going on involving probabilistic programming. See Probabilistic Models of Cognition made by Goodman (Stanford University) and Tenenbaum (MIT) or Anglican made by Wood (University of Oxford) et al.


Training while predicting on dataset I have been trying to figure out whether if I train a model and then while predicting is it possible to train images too just like humans
Somehow converting valid images to the dataset by asking us when an object is shown
Like the Google Photos somehow they ask us if they predicted a face correctly and then reinforces on it

Yes, this method of training a model is commonly known as online learning and specific learning algorithms have been designed for this purpose, such as, Stochastic Gradient Descent(SGD). As opposed to Batch Gradient descent, which computes gradients over the entire training set at each step, the SGD algorithm computes gradients for individual samples and updates the model's parameters.
The online learning strategy is not specific to Reinforcement Learning(RL) methods but is also suitable for the broader classification of Machine Learning algorithms. Suppose, we are given a simple model pretrained on a large Face Recognition dataset, and wish the model to continuously adapt to new faces, we may keep feeding in the new input images with their correct labels (obtained from feedback), as they appear, and apply SGD to incrementally update the model. To go about this using an RL based approach, we may penalize the model with negative rewards for every label misclassified and update it.


How to weight important features Let's say there's a ball with features position, velocity, acceleration.

These three are all concatenated as inputs to my neural network.

However, I have prior knowledge that position is way more predictive than the other features.

How do I weight the position feature much more strongly than the others? Would just applying a large scalar coefficient to it as preprocessing work? Seems unprincipled...

If you are training a neural network, it should learn correct weights to use the most predictive feature without any interference. For a strongly predictive feature matched with weaker ones, most NNs will learn the stronger association very quickly. I'm not sure whether there is an easy way to add this as a prior. You could do things such as inject the most influential feature in one or more hidden layers, but that does seem ugly, and only really helps if the predictive feature has certain types of relationship to output (it would probably help most if there is a strong linear relationship).

Training problems are usually in the opposite direction - how to extract the small amount of information from more noisy and less influential features that may still add up and sometimes contradict the main predictor.

The only thing that springs to mind for NNs that may help in this situation (mostly about fixing the problem of using the less reliable features) is the architecture of residual neural networks. The use of skip connections builds the neural network in such a way that a default "do nothing" identity function is encouraged as layers are added, and this allows for combining weak but complex relationships with stronger simple ones. A residual NN can do this with less compromise than changing network hyper parameters to find best fit on other NNs.

However, residual neural networks are good for managing different degrees of complexity that manifest within a problem, which is not necessarily the same as different degrees of predictive power between features.

  Would just applying a large scalar coefficient to it as preprocessing work?

No, this would usually be counter-productive for training a neural network. You should instead be normalising all inputs - a common and effective technique for neural networks is to ensure that each feature is scaled and offset so it has mean 0, standard deviation 1.


How do I decide the optimal number of layers for a neural network? How do I decide the optimal number of layers for a neural network (feedforward or recurrent)?

There is a technique called Pruning in neural networks, which is used just for this same purpose.

The pruning is done on the number of hidden layers. The process is very similar to the pruning process of decision trees. The pruning process is done as follows:

Train a large, densely connected, network with a standard training
algorithm
Examine the trained network to assess the relative importance of the
weights
Remove the least important weight(s)
retrain the pruned network
Repeat steps 2-4 until satisfied

However, there are several optimized methods for pruning neural nets, and it is also a very active area of research.


Time Series: LSTM or Augmented Vector Space? In time Series prediction, we have a stream of vectors. There are different approaches for accounting for the temporal patterns between these vectors.

There's two that I'm considering. An LSTM or augmenting the feature space. What's the difference between the two? The most obvious to me is that an LSTM is more expressive and can get superior accuracy if modelled properly.

LSTM is a neural network which learns for an input x an output y. In additional to CNNs or MLPs it considers a hiddenstate h (which is influenced by prvious inputs) when your next input x is feed into the network.

Augmenting the feature Space is a technique which you do previous training your LSTM (to augment your data set in order to generatre more data and let the LSTM better generalize to new data). In the field of image recognition you can rotate your images by 40 degree to generate a new one. This process is known as data augmentation. Such methods also appliable to time series.

In summary: first, you start with augmenting your input feature space in order to improve prediction accuracy and then training your LSTM with the augmeneted training data set.


What are some alternatives to the book "Artificial Intelligence: A Modern Approach"? There are two textbooks that I most love and am most afraid of in the world: Introduction to Algorithms by Cormen et al. and Artificial Intelligence: A Modern Approach by Norvig et al. I have started the "AI: A Modern Approach" more than once, but the book is so dense and full of theory that I get discouraged after a couple of weeks and stop.

I am looking for a similar AI book but with an equal emphasis on theory and practice. Some examples of what I am looking for:

The Elements of Statistical Learning by Tibshirani et al. (detailed theory)
An Introduction to Statistical Learning: With Applications in R by
Tibshirani et al. (theory+practical)
Digital Image Processing by Gonzalez et al. (detailed theory)
Digital Image Processing Using MATLAB by Gonzalez et al.
(theory+practical)

The following 2 books helped me understand the basics and guided me through my first AI / CI implementations.

Computational Intelligence: An Introduction by Andries P. Engelbrecht
It includes the most relevant developments in computational intelligence with good discussions on intelligence and artificial intelligence (AI).

Computational Intelligence: Concepts to Implementation by Russell C. Eberhart and Yuhui Shi
It is a comprehensive textbook, supported with lots of practical examples.


How should I define the reward function to solve the Wumpus game with deep Q-learning? I'm writing a DQN agent for the Wumpus game.
Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for falling in pits and dying from the wumpus, -1 each move?
This is naturally cumulative, in that the score changes after each action taken by the agent. Alternatively, is it just a +1 for win, -1 for a loss and 0 in all other situations?

The reward function is up to you when you set the goals for the agent.

If the goal is to score as highly as possible, before ending the game, then use the score. You may want to scale the score down if you are using neural networks, to prevent needing to handle very large error values in early phases of learning.

If the goal is to win the game, and you do not care about the score, then use the win/loss end result. I am not familiar with the game, but if it is possible to win the game - e.g. reach an exit - whilst not collecting all the gold, then the agent may choose to do that if it reduces the chance of losing.

The second option is harder for the agent to assess. You may want the current score to be one of the state variables, as the score is likely to be correlated with win/loss.
Most computer games are designed around giving a numerical score as feedback for human play, with high score tables, players considered "better" if they get more points etc. If you want your agent to compete in the same way, then using the score directly will help achieve that goal.


Training an RNN to answer simple quesitons I would like to train an RNN to follow the sentences:

"Would you like some cheese"? with "Yes, I would like some cheese."

So whenever the template "Would you like some ____?" appears then RNN produces the sequence above. And it should even work on sentences which are new like "Would you like some blumf?"

I have thought of various ways of doing this. Such as, as well as having 26 outputs for letters of the alphabet have about 20 more for "repeat the character that is 14 characters to the left" and so on.

Has this been done before or is there a better way?

I have thought of various ways of doing this. Such as, as well as having 26 outputs for letters of the alphabet have about 20 more for "repeat the character that is 14 characters to the left" and so on

Creating a system of rules like your example above is the very opposite of training an RNN to perform this task.

If you would like to train an RNN to answer simple questions you would not need to come up with ingenious rules, but with enough training data of the form

question -> answer

Then you could use one of many popular sequence-to-sequence NLP tools to try and learn this behaviour, effectively treating the problem as if it were machine translation between languages.

More broadly, yes, question answering has been done before, it is very popular in fact. It is an active subfield of NLP research and many methods have been developed, some of them involving RNN networks.


How is the log-derivative trick of a trajectory derived? I am looking at this formula which breaks down the gradient of $P(\tau |\theta)$ the first part is clear as is the derivative of $\log(x)$, but I do not see how the first formula is rearranged into the second.

The identity $$\nabla_{\theta} P(\tau \mid \theta) = P(\tau \mid \theta) \nabla_{\theta} \log P(\tau \mid \theta)\tag{1}\label{1},$$

which can also be written as

\begin{align}
\nabla_{\theta} \log P(\tau \mid \theta)
&=
\frac{\nabla_{\theta} P(\tau \mid \theta)}{P(\tau \mid \theta)}\\
&=\frac{1}{P(\tau \mid \theta)} \nabla_{\theta} P(\tau \mid \theta)
\end{align}

directly comes from the general rule to derive the logarithm of a function and the chain rule
\begin{align}
\frac{d \log f(x)}{d x}
&= \frac{1}{f(x)} \frac{d f}{dx}.
\end{align}
Note that $\log f(x)$ is a composite function and that's why we apply the chain rule and that the derivative of $\log x = \frac{1}{x}$, as your text says.

People shouldn't call this a trick. There's no trick here. It's just basic calculus.

Why do you need identity \ref{1}? Because that identity tells you that the derivative of the probability of the trajectory given the parameter $\theta$ with respect to $\theta$ is $P(\tau \mid \theta)$ times the gradient of the logarithm of that same probability. How is this useful? Because the logarithm will turn your product into a sum (and the derivative of a sum is the sum of the derivatives of the elements of the sum), Essentially, the identity \ref{1} will help you to compute the gradient is an easier way (at least, conceptually).


Why is my loss (binary cross entropy) converging on ~0.6? (Task: Natural Language Inference) I’m trying to debug my neural network (BERT fine-tuning) trained for natural language inference with binary classification of either entailment or contradiction. I've trained it for 80 epochs and its converging on ~0.68. Why isn't it getting any lower?

Thanks in advance!

Neural Network Architecture:

Training details:

Loss function: Binary cross entropy
Batch size: 8
Optimizer: Adam (learning rate = 0.001)
Framework: Tensorflow 2.0.1
Pooled embeddings used from BERT output.
BERT parameters are not frozen.

Dataset:

10,000 samples
balanced dataset (5k each for entailment and contradiction)
dataset is a subset of data mined from wikipedia.
Claim example: "'History of art includes architecture, dance, sculpture, music, painting, poetry literature, theatre, narrative, film, photography and graphic arts.'"
Evidence example: "The subsequent expansion of the list of principal arts in the 20th century reached to nine : architecture , dance , sculpture , music , painting , poetry -LRB- described broadly as a form of literature with aesthetic purpose or function , which also includes the distinct genres of theatre and narrative -RRB- , film , photography and graphic arts ."

Dataset preprocessing:

Used [SEP] to separate the two sentences instead of using separate embeddings via 2 BERT layers. (Hence, segment ids are computed as such)
BERT's FullTokenizer for tokenization.
Truncated to a maximum sequence length of 64.

See below for a graph of the training history. (Red = train_loss, Blue = val_loss)

It seems to be overfitting and your model is not learning. Try SGD optimizer with a learning rate of 0.001
ADAM optimizer will give you a soon overfitting, and decreasing the learning rate will train your model better. The learning rate is about steps to change weights, in this plot you see that the validation loss is not changing with an optimization goal


What other kind of AIs exist apart from goal-driven? Goal-driven AIs is the only kind of AI I am aware of. However, Marcus Hutter claims the following

  Most, if not all known facets of intelligence can be formulated as goal driven or, more generally, as maximizing some utility function. It is, therefore, sufficient to study goal driven AI.

which doesn't necessarily imply that there are other types of AIs (apart from goal-driven), but (at least, in the way it is phrased) suggests that there are other types of AIs. If there exist other types of AIs, which are they?

Whether all AI are goal-driven depends on how far you're willing to extend the definition of 'goal'.

AIs are often defined through their primary tasks. Face recognition is a process; a face recognition AI is a limited, special-purpose AI. Paperclip collection is a process, the Paperclip Maximizer is a general, goal-driven AI.

There will always be some sort of goal, but it may in certain cases become so nebulous it's really a stretch to continue calling it a goal. For example,  unsupervised learning (as suggested by Aiden Grossman) - the goal you set is "learn", with very little qualifiers. The underlying AI may develop in surprising and unexpected ways; it may even modify own goals.

Similarly, another "aimless" AI type is a simulator of biological intelligence - say, replicating neural connections of an insect in software, making it run and observing the behaviors. The AI itself is not given a specific goal - it's a tool/subject of study, not something directly producing results. Still, the goals are exactly the same as goals of the original insect: feed, multiply, stay safe; unattainable due to hardware limitations but still pursued.

Therefore, whether all AIs are goal-driven is arguable. The only AI that is not actually driven by any goal whatsoever is one that's switched off. But whether you consider all the possible nebulous, abstract or accidental goals that drive the AI to call it actually goal-driven, that's up to debate.


How can I detect datetime patterns in text? I want to explore and experiment the ways in which I could use a neural network to identify patterns in text.

examples:

Prices of XYZ stock went down at 11:00 am today
Retrieve a list of items exchanged on 03/04/2018
Show error logs between 3 - 5 am yesterday.
Reserve a flight for 3rd October.
Do I have any meetings this Friday?
Remind to me wake up early tue, 4th sept

This is for a project so I am not using regular expressions. Papers, projects, ideas are all welcome but I want to approach feature extraction/pattern detection to have a model trained which can Identify patterns that it has already seen.

Approaches

There are two main approaches to detecting any human readable representation of a discrete quantity within text.

Detect well known and stable patterns in the input stream and by adjacency determine the output stream.
Windowing through the text in the input stream and directly detect the quantities.

There are other approaches and there are hybrids of these two or one of these and the other approaches, but these two are the theoretically most straightforward and likely to produce both reliability and accuracy.

Re-entrant Learning

Whether the training involves re-entrant learning techniques, such as reinforcement, is a tangential issue that this answer will not address, but know that whether all training is solely a deployment component or whether adaptation and/or convergence occurs in real time is an architectural decision to be made.

Practical Concerns

Practically, the outputs of each recognition are as follows.

Starting index
Ending index
Integer year or null
Integer day of year or null
Integer hour in military time or null
Minute or null
Second or null
Time zone or null
Probability the recognition unit was correctly identified
Probability the recognition produced accurate results

Also practically, the input must either be from within one particular locale's norms in terms of

Calendar,
Time,
Written language,
Character encoding, and
Collation,

... or ...

The learning must occur using training sets that include the locales that will be encountered during system use

... or ...

Much of the locale specific syntax must be normalized to a general date and time language such as this:

जनवरी --> D_01

Enero --> D_01

Janúar --> D_01

so that Filipino and Icelandic names for the first month of the year enter the artificial network as the same binary pattern.

**Date and Time Specifically*

In the case of 1. above, which is semi-heuristic in nature, and assuming that the locale is entirely en-US.utf-8, the CASE INSENSITIVE patterns for a PCRE library or equivalent to use as a search orientation heuristic include the following.

(^|[^0-9a-z])(19|20|21)[0-9][0-9])([^0-9a-z]|$)
(^|[^0-9a-z])(Mon|Monday|Tue|Tues|Tuesday|Wed|Wednesday|Thu|Thur|Thurs|Thursday|Fri|Friday|Sat|Saturday|Sun)([^0-9a-z]|$)
(^|[^0-9a-z])(Jan|January|Feb|February|Mar|March|Apr|April|May|Jun|June|Jul|July|Aug|August|Sep|Sept|September|Oct|October|Nov|November|Dec|December)([^0-9a-z]|$)
(^|[^0-9a-z])(Today|Yesterday|Tomorrow)([^0-9a-z]|$)
(^|[^0-9])[AP]M|[AP][.]M[.]|Noon|Midnight)([^0-9a-z]|$)
(^|[^a-z])(0?[1-9])(:[0-5][0-9]){1,2}([^a-z]|$)

There should be others for time, hyphenated or slash delimited dates, or time zone.

The positions and normalized encoding of these date and time artifacts are then substituted into the artificial network inputs instead of the original text in the stream, reducing redundancy and improving both the speed of training and the resulting accuracy and reliability of recognition.

In the case of 2. above, the entire burden of recognition is left to the artificial network.  The advantage is less reliance on date and time conventions.  The disadvantage is a much larger burden placed on training data variety and training epochs, meaning a much higher burden on computing resources and the pacience of the stake holder for the project.

Windowing

An overlapping windowing strategy is necessary.  Unlike FFT spectral analysis in real time, the windowing must be rectangular, because the size of the window is the width of the input layer of the artificial network.  Experimenting with the normalization of input such that the encoding of text and data and time components entering the input layer could greatly vary the results in terms of training speed, recognition accuracy, reliability, and adaptability to varying statistical distributions of date and time instances and relationships.


Is there a multi-agent deep reinforcement learning algorithm which is for environments with only discrete action spaces (Not hybrid)? Is there a multi-agent deep reinforcement learning algorithm which is for environments with only discrete action spaces (Not hybrid) and have centralized training?
I have been looking for algorithms, (A2C, MADDPG etc.) but still havent find any algorithm that provides all of properties i mentioned (Multi agent + discrete action space + deep learning + centralized training).
I am wondering if we use an actor network that gets state as input and concatenated discrete actions of agents as output (For example if agent has 3 actions and we have 4 agents output can be [0,0,1, 0,1,0, 0,0,1, 1,0,0]) is that would be bad idea ?

A natural policy to act in an environment with discrete action space would be a softmax.
This paper describes a method that uses the idea of centralized training, and I believe could be used in your implementation.
With regard to your last question, I don't know if i understood, but if you have a system that must perform 3 actions, you could assign each action to a specific agent (assuming we have three different action spaces). Then you would have a cooperation game with 3 agents, where all of them have a common reward function. In theory, this 3 agents represents an individual agent that interacts with the environment.


Why is the hypothesis function $h_{\theta}(x)$ equivalent to $E[y | x; \theta]$ in generalised linear models? Reading through the CS229 lecture notes on generalised linear models, I came across the idea that a linear regression problem can be modelled as a Gaussian distribution, which is a form of the exponential family. The notes state that $h_{\theta}(x)$ is equal to $E[y | x; \theta]$. However, how can $h_{\theta}(x)$ be equal to the expectation of $y$ given input $x$ and $\theta$, since the expectation would require a sort of an averaging to take place?

Given x, our goal is to predict the expected value of $T(y)$ given $x$. In most of our examples, we will have $T(y) = y$, so this means we would like the prediction $h(x)$ output by our learned hypothesis h to satisfy $h(x) = E[y|x]$.
To show that ordinary least squares is a special case of the GLM family of models, consider the setting where the target variable y (also called the response variable in GLM terminology) is continuous, and we model the conditional distribution of y given x as a Gaussian $N(\mu,\sigma^2)$. (Here, $\mu$ may depend $x$.) So, we let the ExponentialFamily($\eta$) distribution above be the Gaussian distribution. As we saw previously, in the formulation of the Gaussian as an exponential family distribution, we had μ = η. So, we have
$$h_{\theta}(x) = E[y|x; \theta] = \mu = \eta = \theta^Tx.$$

EDIT
Upon reading other sources, $y_i \sim N(\mu_i, \sigma^2)$ meaning that each individual output has it's own normal distribution with mean $\mu_i$ and $h_{\theta}(x_i)$ is set as the mean of the normal distribution for $y_i$. In that case, then the hypothesis makes sense to be assigned the expectation.

In generalised Linear models, each output variable $y_i$ is modelled as a distribution from the exponential family, with the hypothesis function $h_{\theta}(x)$ for a given $\theta$ as the expected value of $y_i$ and maximum likelihood estimation is usually the method used to solve GLM's.


How to detect LEGO bricks by using a deep learning approach? In my thesis I dealt with the question how a computer can recognize LEGO bricks. With multiple object detection, I chose a deep learning approach. I also looked at an existing training set of LEGO brick images and tried to optimize it.

My approach

By using Tensorflow's Object Detection API on a dataset of specifically generated images (Created with Blender) I was able to detect 73.3% of multiple LEGO Bricks in one Foto.

One of the main problems I noticed was, that I tried to distinguish three different 2x4 bricks. However, colors are difficult to distinguish, especially in different lighting conditions. A better approach would have been to distinguish a 2x4 from a 2x2 and a 2x6 LEGO brick.

Furthermore, I have noticed that the training set should best consist of "normal" and synthetically generated images. The synthetic images give variations in the lighting conditions, the backgrounds, etc., which the photographed images do not give. However, when using the trained Neural Network, photos and not synthetic images are examined. Therefore, photos should also be included in the training data set.

One last point that would probably lead to even better results is that you train the Neural Network with pictures that show more than one LEGO brick. Because this is exactly what is required by the Neural Network when it is in use.

Are there other ways I could improve upon this?

(Can you see any further potential for improvement for the Neural Network? How would you approach the issue? Do any of my approaches seem poor? How do you solve the problem?)

So I am assuming that you are trying to detect a lego brick from the image. One idea is that you can use transfer learning. Leveraging a pre-trained machine learning model is called transfer learning. The underlying idea behind transfer learning is that one takes a well-trained model from one dataset or domain, and applies it to a new one.  François Chollet has written a very comprehensive guide to transfer learning (https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html)

I admit I took some of this information from Christopher Bonnett's article named Classifying e-commerce products based on images and text.

I also suggest using the Lego brick dataset from Kaggle on this link: https://www.kaggle.com/joosthazelzet/lego-brick-images
It has over 12,700 lego brick images.

If processing power is a problem, you can use Amazon Web Services for cloud computing. It is inexpensive for small scale operations like this.

Of course for the object detection, you can always increase the number of convolution layers. However, if you have too many layers, you should also include residual blocks/residual network. This would allow neural networks even with over a thousand layers to operate effectively. This video should help you understand how residual networks work (https://www.youtube.com/watch?v=ahkBkIGdnWQ)

Finally, make sure not to overfit during your training and if you do follow the residual network idea, you should also include upsampling in your convolution neural network( More in here: https://towardsdatascience.com/up-sampling-with-transposed-convolution-9ae4f2df52d0)

I hope this helped and good luck on your endeavor.


Is this technique image processing or computer vision? If I use my mobile camera on a signboard or announcement board on a road or in a street (like the one attached in photo) where the message is written in Russian and my mobile shows me that message in English, would this be an image processing or computer vision technique?

This is a computer vision task, given that it requires a high-level understanding of the image (that is, it contains text, and the text needs to be translated to another language), which can use an image processing technique (for example, to deblur the possibly blurry original photo).


How can we conclude that an optimization algorithm is better than another one When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 score .etc, and do the same for an old optimization algorithm, or do we need to compute the average performance,i.e.,the average value of accuracy or f1 scores for these runs, to show that it is better than the old optimization algorithm? Because when I read the papers on a new optimization algorithm, I don't know how they calculate the performance and draw the train-loss vs iters curves, because it has random effects, and for different runs we may get different performance and different curves.

See here for a potential way to do it:

http://infinity77.net/global_optimization/#motivation-motivation

http://infinity77.net/global_optimization/#rules-the-rules

You basically test the two (or more) optimization algorithms against known objective functions, with several random (but repeatable) starting points and then analyze the outcome.


Technology for predicting body measurements of a person, with a full body photo of them I am working on making an app that would require the ratio of the height and the largest width of a person, in order to group the individual into a certain body fat percentage category.
In short, I want the body measurements of them by inputting their full body photos.
Can anyone help me out with what technology I should study and research on to implement this?
Thank you.

If the user will be imputting the body height/largest width measures, and you want to have a couple of groups where you final results will be stored at, you should look at the classification algorithms(Logistic Regression, KNN, Random Forests).
If on the other hand, you want your program to detect the user from an image and take the measurements itself, you would need to look into Convolutional Neural Networks, along with person detection/segmentation


What's the best architecture for time series prediction with a long dataset? I have to build a neural network without any architecture limitations which have to predict the next value of a time series.

The dataset is composed of 400.000 values, which are given in hex format. For example

0xbfb22b14
0xbfb22b10
0xbfb22b0c
0xbfb22b18
0xbfb22b14

I think LSTM is suitable for this problem, but I am worried about the length of the input. Would it be a good idea to use CNN?

def structure(step,n_features):
    # define model
    model = Sequential()
    model.add(LSTM(50, activation='relu', return_sequences=True, input_shape=(step, n_features)))
    model.add(Dense(1))
    model.compile(optimizer='adam', loss='mse')
    return model

What about this one ?

"model": {
        "loss": "mse",
        "optimizer": "adam",
    "save_dir": "saved_models",
        "layers": [
            {
                "type": "lstm",
                "neurons": 999,
                "input_timesteps": 998,
                "input_dim": 1,
                "return_seq": true
            },
            {
                "type": "dropout",
                "rate": 0.05
            },
            {
                "type": "lstm",
                "neurons": 100,
                "return_seq": false
            },
            {
                "type": "dropout",
                "rate": 0.05
            },
            {
                "type": "dense",
                "neurons": 1,
                "activation": "linear"
            }

Yes, LSTM are ideal for this. For even stronger representational capacity, make your LSTM's multi-layered.
Using 1-dimensional convolutions in a CNN is a common way to exctract information from time series too, so there's no harm in trying. Typically, you'll test many models out and take the one that has best validation performance.


What are the different approaches used in Machine Learning? There seem to be so many sub-fields, so I'm interested in getting a better understanding of the approaches.
I'm looking for information on a single framework per answer, in order to allow for granularity without the overall answer getting too long. For instance; Deep Learning Neural Networks would be a single answer.

Things in italics should give you enough googleable terms to start a deeper dive :P.
There are 3 main branches of statistical ML.

Supervised Learning
This approach is taken when a problem can be phrased as associating some $X$ with some $Y$. For example, classifying a picture of a cat ($X$) with the label “Cat” ($Y$). Training in supervised learning usually means presenting some $X$, having an agent predict the label, comparing the prediction with the answer to get an error metric, and finally using the error to update the agent to make better predictions on future $X$s. MNIST is a great example of a classification domain.
1.1 Supervised learning also works with regression when we want to learn an associated number instead of a label, such as the risk of heart disease given years of smoking and weight.

Unsupervised Learning
This type of learning is used when we want to draw inferences from a dataset without labeled responses. It can be used to find hidden patterns or grouping in data.

Reinforcement Learning
The $X$s and $Y$s in supervised learning are i.i.d., meaning that they don’t relate to the previous $X$ that a model has previously observed. However, there are many problems where the most recent $X$, $X$ at time $t$ ($X_t$), is very important to figuring out what $Y$ to predict ($Y_{t+1}$) for the next $X$ ($X_{t+1}$). The field of reinforcement learning (RL) builds on this notion by framing observations as data that is generated by an agent interacting with an environment. Specifically, the field of RL poses the problem of finding a behaviour (a policy) to select actions such that an agent using the policy can maximize a special signal called the reward from the environment. The agent starts in a state of the world ($S_t$) at time $t$ and takes some action ($A_t$) which puts it in a new state ($S_{t+1}$) and gives it a reward ($R_{t+1}$).

Neural networks are a technique for finding features (some kinda notable differences) in data. They can be used in all of the above learning strategies but are not necessary for them although their application has had some very powerful results recently. Deep ___ learning is simply the application of a multilayered neural network on one of the above learning methods.


What are the mathematical prerequisites needed to understand research papers on neural networks? I know we have developed some mathematical tools to understand deep neural networks, gradient descent for optimization, and basic calculus. Recently, I encountered arxiv paper that describes higher mathematics for neural networks, such as functional analysis. For example, I remember universal approximation theorem was proved with the Hann-Banach theorem, but I lost the link of that article, so I need to find similar papers or articles to develop my understanding of neural networks mathematically (like with functional analysis, in short, I need to learn more advanced math for research), can you suggest some books or arxiv papers or articles or any other source that describes mathematics for deep neural networks?

Knowing you want to focus on the theory, I think that a good choice is Deep Learning Book from Ian Goodfellow et al., which is publicly available. It has three main parts. On the first one the author presents the math/ statistic tools that will be needed to understand the following parts. On the second part, the author explains the current state of the art in Deep Learning and on the last part more advanced topics are introduced. The author also uses references so as to facilitate extra resources to dive more into the theory.
On the other hand I strongly recommend you Google Scholar, there you have plenty of information given by articles/ papers of the current state of the art techniques in the field. There you can also find papers related on what that you mentioned about the Hahn-Banach Theorem.


Machine learning to predict 8*8 matrix values using three independent matrices Problem Statement

I have 4 main input features.

This is a small snippet of the data for clearer understanding.

Gate name -> for example AND Gate

index_1 -> [0.001169, 0.005416, 0.01391, 0.03037, 0.06381, 0.1307, 0.2645, 0.532]

index_2 -> [7.906e-05, 0.001123, 0.00321, 0.007253, 0.01547, 0.03191, 0.06478, 0.1305]

values -> [[11.0081, 14.0303, 18.8622, 27.3426, 43.8661, 76.7538, 142.591, 274.499],
           [11.3461, 14.3634, 19.1985, 27.6827, 44.2106, 77.0954, 142.926, 274.879],
           [12.258, 15.2816, 20.1095, 28.5856, 45.1057, 77.9778, 143.8, 275.758],
           [13.665, 16.7457, 21.5835, 30.0545, 46.5581, 79.4212, 145.252, 277.192],
           [15.6636, 18.9526, 23.9051, 32.4281, 48.9011, 81.7052, 147.477, 279.371],
           [17.8838, 21.5839, 26.8957, 35.7103, 52.3901, 85.2132, 150.89, 282.714],
           [19.3338, 23.6933, 29.7184, 39.1212, 56.4053, 89.9721, 155.913, 287.637],
           [18.7856, 23.9999, 31.1794, 41.7549, 60.0043, 95.0488, 162.951, 295.005]]

My task is to predict this values matrix, given that I have index_1 and index_2. Originally this values matrix is propagation delay, calculated using a simulator called SPICE.

Where I am facing problem

There is no written relation between Index_1, index_2 or values since simulator calculates this value using it's own models.
I have made a CSV file which contains the data in separate columns.
Another approach that I thought. If I can give index_1, index_2 and any 5*5 sub-matrix to the model, and the model can predict the values of whole 8*8 Matrix. But the problem is again, which machine learning model do I use.

Approaches Tried so Far

I have tried a CNN model for this but it is giving me very low accuracy.
Used one dense fully connected neural network but it is over-fitting the data and not giving me any values for matrix.

I am still stuck at how to predict the matrix values given this data.  What are other strategies can be used?

In principle, you can use a fully connected neural network with reshaping for this kind of problem. The questions you should ask yourself are:

Baselines: What are the simplest algorithms to approach the problem? How good would a human be?
What do I know? Are there any properties of the 8x8 matrix that will always be true? For example, it seems as if the values from left to right strictly increase. Same for top to bottom. This can be used!
Are the outputs independent? (e.g. having the (1,1) entry of the matrix, do I know something about the $(i, j)$ entry of it?

Then, of course, there are more specific things I could imagine. If you were not clear about (2), you might have (wrongly) used softmax/tanh/sigmoid in the last layer. You might simply have too little training data for neural networks. Your neural network implementation might be broken.


Is there anything that ensures that convolutional filters end up different from one another? I found this question very interesting, and this is a follow up on it.
Presumably, we'd want all the filters to converge towards some complementary set, where each filter fills as large a niche as possible (in terms of extracting useful information from the previous layer), without overlapping with another filter.
A quick thought experiment tells me (please correct me if I'm wrong) that if two filters are identical down to maximum precision, then without adding in any other form of stochastic differentiation between them, their weights will be updated in the same way at each step of gradient descent during training. Thus, it would be a very bad idea to initialise all filters in the same way prior to training, as they would all be updated in exactly the same way (see footnote 1).
On the other hand, a quick thought experiment isn't enough to tell me what would happen to two filters that are almost identical, as we continue to train the network. Is there some mechanism causing them to then diverge away from one another, thereby filling their own "complementary niches" in the layer? My intuition tells me that there must be, otherwise using many filters just wouldn't work. But during back-propagation, each filter is downstream, and so they don't have any way of communicating with one another. At the risk of anthropomorphising the network, I might ask "How do the two filters collude with one another to benefit the network as a whole?"

Footnotes:

Why do I think this? Because the expession for the partial derivative of the $k$th filter weights with respect to the cost $\partial W^k/\partial C$ will be identical for all $k$. From the perspective of back-propagation, all paths through the filters look exactly the same.

Yes, your thought experiment is correct, and the concept is known as breaking the symmetry. This is why biases can be initialized to $0$ (bias initialization doesn't matter), but weights should be randomly initialized to different numbers -- to break the symmetry. Otherwise, if not, the network will function as if it has $n-1$ filters (or however many filters that are unique) instead of the full $n$ filters.
As for your main question, if two filters are initialized to very similar values, they may branch out as long as that is what minimizes the training loss. There is no collusion or coordination going on; each filter updates completely independently. You can even freeze all the other filters and only perform gradient descent on one filter at a time. Each filter just follows the direction of their gradient to minimize the training loss.
Consider the backprop equations as defined by this online book:

The gradient of the current layer's weights depends on

The future layers' weights, errors, and activation function's derivatives
The current layer's activation function's derivative, and
The previous layer's outputs.

Each weight in the layer (i.e. each filter in the layer) looks at different parts of these three components (indexed by $j$ and $k$ in equation $BP4$). It is this different perspective that allows them to update their gradients in different directions, even if their initial weights are very similar to each other. Note that it is possible that they end up with the same gradient, but it is very unlikely.


How to obtain a formula for loss, when given an iterative update rule in gradient descent? From the reinforcement learning book section 13.3:

Using pytorch, I need to calculate a loss, and then the gradient is calculated internally.

How to obtain the loss from equations which are stated in the form of an iterative update with respect to the gradient?

In this case:

$\theta \leftarrow \theta + \alpha\gamma^tG\nabla_{\theta}ln\pi(A_t|S_t,\theta)$

What would be the loss?

And in general, what would be the loss if the update rule were

$\theta \leftarrow \theta + \alpha C\nabla_{\theta}g(x|\theta)$

for some general (derivable) function $g$ parameterized by theta?

You can find an implementation of the REINFORCE algorithm (as defined in your question) in PyTorch at the following URL: https://github.com/JamesChuanggg/pytorch-REINFORCE/. First of all, I would like to note that a policy can be represented or implemented as a neural network, where the input is the state (you are currently in) and the output is a "probability distribution over the actions you can take from that state received as input".

In the Python module https://github.com/JamesChuanggg/pytorch-REINFORCE/blob/master/reinforce_discrete.py, the policy is defined as a neural network with 2 linear layers, where the first linear layer is followed by a ReLU activation function, whereas the second is followed by a soft-max. In that same Python module, the author also defines another class called REINFORCE, which creates a Policy object (in the __init__ method) and defines it as property of that class. The class REINFORCE also defines two methods select_action and update_parameters. These two methods are called from the main.py module, where the main loop of the REINFORCE algorithm is implemented. In that same main loop, the author declares lists entropies, log_probs and rewards. Note that these lists are re-initialized at ever episode. A "log_prob" and an "entropy" is returned from the select_action method, whereas a "reward" is returned from the environment after having executed one environment step. The environment is provided by the OpenAI's Gym library. The lists entropies, log_probs and rewards are then used to update the parameters, i.e. they are used by the method update_parameters defined in the class REINFORCE.

Let's see better now what these methods, select_action and update_parameters, actually do.

select_action first calls the forward method of the class Policy, which returns the output of the forward pass of the NN (i.e. the output of the soft-max layer), so it returns the probabilities of selecting each of the available actions (from the state given as input). It then selects the probability associated with the first action (I guess, it picks the probabilities associated with the action with the highest probabilities), denoted by prob (in the source code). Essentially, what I've described so far regarding this select_action method is the computation of $\pi(A_t \mid S_t, \theta)$ (as shown in the pseudocode of your question). Afterwards, in the same method select_action, the author also computes the log of that probability I've just mentioned above (i.e. the one associated with the  action with the highest probability, i.e. the log of prob), denoted by log_prob. In that same method, the entropy (as defined in this answer) is calculated. In reality, the author calculates the entropy using only one distribution (instead of two): more specifically, the entropy is calculated as follows entropy = -(probs*probs.log()).sum(). In fact, the entropy loss function usually requires the ground-truth labels (as explained in the answer I linked you to above), but, in this case, we do not have ground-truth labels (given that we are performing RL and not supervised learning). Nonetheless, I can't really tell you why the entropy is calculated like this, in this case. Finally, the method select_action then return action[0], log_prob, entropy.

First of all, I would like to note that the method update_parameters is called only at the end of each episode (in the main.py module). In that same method, a variable called loss is first initialized to zero. In that method, we then iterate the list of rewards for the current episode. Inside that loop of the update_parameters method, the return, R is calculated. R is also multiplied by $\gamma$. On each time step, the loss is then calculated as follows

loss = loss - (log_probs[i]*(Variable(R).expand_as(log_probs[i])).cuda()).sum() - (0.0001*entropies[i].cuda()).sum()

The loss is calculated by subtracting the previous loss with

(log_probs[i]*(Variable(R).expand_as(log_probs[i])).cuda()).sum() - (0.0001*entropies[i].cuda()).sum()

where log_probs are the log probabilities calculated in the select_action method. log_probs is the part $\log \pi(A_t \mid S_t, \theta)$ of the update rule of your pseudocode. log_probs are then multiplied by the return R. We then sum the result of this multiplication over all elements of the vector. We then subtract this just obtained result by the entropies multiplied by 0.0001. I can't really tell you why the author decided to implement the loss in this way. I would need to think about it a little more.

The following article may also be useful: https://pytorch.org/docs/stable/distributions.html.


What are the advantages of complex-valued neural networks? During my research, I've stumbled upon "complex-valued neural networks", which are neural networks that work with complex-valued inputs (probably weights too). What are the advantages (or simply the applications) of this kind of neural network over real-valued neural networks?

According to this paper, complex-valued ANNs (C-ANNs) can solve problems such as XOR and symmetry detection with a smaller number of layers than real ANNs (for both of these a 2 layer C-ANN suffices, whereas a 3-layer R-ANN is required).

I believe that it is still an open question as to how useful this result is in practice (e.g. whether it actually makes finding the right topology easier), so at present the key practical advantage of C-ANNs is when they are a closer model for the problem domain.

Application areas are then where complex values arise naturally, e.g. in optics, signal processing/FFT or electrical engineering.


Can we implement GAN (Generative adversarial neural networks) for classication problem like Fraud detecion? Problem: Fraud detection
Task : classifying transactions as either fraud/ Non-fraud using GAN

Generative networks was the first popular class of topologies that had compound feedback, corrective signaling at more than one layer.  In popular network designs such as MLPs (multilayer perceptrons), CNNs (convolutional neural networks), and LSTM (long short term memory) networks, the backpropagation is a single layer mechanism.  The mechanism distributes corrective a signal creating a closed loop that ideally converges to an optimal network behavior.

The speed, accuracy, and reliability of convergence of networks in an AI system depend on a number of engineering choices at a lower level of design and development.

Selection of algorithm variants and extensions
Selection of data set
Drawing of the training samples
Data normalization
Initialization
Assignment of learning rate or learning rate policy
Other hyper-parameters

Generative networks are designed to achieve a higher level balance between two networks, each of which have their own backpropagation, and thus have a compound feedback topology.

Considering the original GAN approach in the context of fraud detection, a discriminative network $D$, in a way, detects fraudulence originating from the generative network $G$, and, in a way, $G$ learns authenticity from the corrective signal provided by $D$, even if the authenticity learned is superficial.  However, in the case of credit card fraud detection from a numerical set of data, the fraudulence is not internal to the system to be designed as in the case of GAN.  Without both the $G$ and $D$ networks, the design is not a generative network.

Since the comment stated, "I have to explore GAN for this problem," the key question is this.

  If this system is to benefit from a GAN, what would the GAN generate that could contribute to the solution?

If the answer is nothing, then a GAN is not the correct approach.  A discriminative network without a generative network is not a GAN.  If this question is from an assignment to train a network using a static and already existing data set, a MLP may suffice, depending on the expectations of the teaching staff.  If that is the case, the goal will be to analyze the data set using statistical tools to determine if it is already normalized well enough to lead to excellent convergence accuracy, speed, and reliability.  If normalization is indicated, find ways to present the data to the input layer of the MLP in such a way that each input is relevant and has a well distributed range of values across the data type.

Ensure there is no data loss in this process, and remove redundancy where it can easily be removed.  By redundancy removal is meant, if one of the dimensions of data have only four possible values of between 10 and 1,000 bytes, encode the four possible values in two bits because to do so removes unnecessary burden from the training.

Similarly, the output layer must be chosen to produce an encoded set of possible output values and the decoding must be written to produce usable output.  In this case, it appears that the labels are, Fraudulent and Authentic.  Which means that the last layer would be a single binary threshold cell.

For this simple MLP approach and for many of the approaches mentioned next, there are examples available in Python, Java, C++ and other languages from which implementation can be initiated.  If the design must stand up to a more realistic fraud detection scenario, either for credit cards, junk mail, banking, log-in, or communications those experienced with fraud detection know that the above approach is entirely insufficient except as a starting point for learning.  In such a case, these considerations further complicate the design of the solution.

Fraudulence and authenticity is not necessarily sufficient information for investigators.
    The likelihood of fraudulence and the reason why fraudulence is suspected is also pertinent for end users of production output,
    so that security holes can be blocked and investigation may be directed toward prosecution or some other set of remedies.
Fraudulence is usually a security breach phenomena that involves continuous improvement in the attacks until the attackers
    are located and their work is interrupted by arrest.
    What initially works as fraud detection may not work a week, a day, an hour, or a minute later.
Fraud detection, if it to be useful in a real financial context, must be part of a fraud prevention system,
    and training on a static data set may be meaningless in that larger usage environment.

In AI systems that work in real production environments, it may be worth investigating using an LSTM network in combination with a reinforcement learner.  Such may be over a beginner's level of design and implementation experience, but using either an LSTM network or a basic Q-learning approach instead of both of them together may be a reasonable starting point for dedicated students.  If the answer to the question about what a GAN could generate that would be of use must be yes, the following ideas may be helpful.

A generative network could generate transaction data with a distribution that resembles authentic transactions, and such has been successful in the past.
A generative network could generate transaction data with a distribution that resembles attacks on an account by a fraudulent user. That has been done too.
A generative network could theoretically generate hacker agents, although this may not have yet been done.
A generative network could be used to generate patterns of authentic transactions based on some authenticity measure other than the data set mentioned, and the trained discriminative network could then be employed in some way in a fraud detection system in production, although it is unclear what advantage would be gained.

The reason a GAN was suggested would need to be clarified before this answer could provide a more exact direction for AI system topology and project approach.


A3C fails to solve MountainCar-v0 enviroment (implementation by OpenAi gym) While I've been able to solve MountainCar-v0 using Deep Q learning, no matter what I try I can't solve this enviroment using policy-gradient approaches. As far as I learnt searching the web, this is a really hard enviroment to solve, mainly because the agent is given a reward only when it reaches the goal,which is a rare event. I tried to apply the so called "reward engineering", more or less substituting the reward given by the enviroment with a reward based upon the "energy" of the whole system (kinetic plus potential energy), but despite this no luck.
I ask you:

is correct to assume that MountainCar-v0 is beyond the current state of the art A3C algorithm, so that it requires some human intervention to suggest the agent the policy to follow, for example adopting reward engineering?
could anyone provide any hint about which reward function could be used, provided that reward engineering is actually needed ?

Thanks for your help.

I don't know about your first question, but I got a  basic policy gradient approach with the kinetic energy as reward working on MountainCar-v0.

You can implement it based on this blog and the notebook you find there. It uses an MLP with one hidden layer of size 128 and standard policy gradient learning.

The reward engineering boils down to replacing the reward variable with the kinetic energy $v^2$ (no potential energy and no constant factor, the reward itself is not used). I takes $>1000$ episodes to solve the environment consistently.

I'm afraid the solution is not very satisfactory and I don't have the feeling there is much to learn from it. The solution is originally for the cartpole problem and it stops working for me if I change hyperparameters/optimizer or the specifics of the reward.


Why does the "reward to go" trick in policy gradient methods work? In the policy gradient method, there's a trick to reduce the variance of policy gradient. We use causality, and remove part of the sum over rewards so that only actions happened after the reward are taken into account (See here http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-5.pdf, slide 18).
Why does it work? I understand the intuitive explanation, but what's the rigorous proof of it? Can you point me to some papers?

An important thing we're going to need is what is called the "Expected Grad-Log-Prob Lemma here" (proof included on that page), which says that (for any $t$):
$$\mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) \right] = 0.$$
Taking the analytical expression of the gradient (from, for example, slide 9) as a starting point:
$$\begin{aligned}
\nabla_{\theta} J(\theta) &= \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \right) \left( \sum_{t=1}^T r(s_t, a_t) \right) \right] \\
%
&= \sum_{t=1}^T \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=1}^T r(s_{t'}, a_{t'}) \right] \\
%
&= \sum_{t=1}^T \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=1}^{t-1} r(s_{t'}, a_{t'}) + \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=t}^T r(s_{t'}, a_{t'}) \right] \\
%
&= \sum_{t=1}^T \left( \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=1}^{t-1} r(s_{t'}, a_{t'}) \right] \\
+ \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=t}^T r(s_{t'}, a_{t'}) \right] \right) \\
\end{aligned}$$
At the $t^{th}$ "iteration" of the outer sum, the random variables
$ \sum_{t'=1}^{t-1} r(s_{t'}, a_{t'}) $
and
$ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) $
are independent (we assume, by definition, the action only depends on the most recent state), which means we are allowed to split the expectation:
$$\nabla_{\theta} J(\theta) = \sum_{t=1}^T \left( \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \sum_{t'=1}^{t-1} r(s_{t'}, a_{t'}) \right] \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \right] \\
+ \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=t}^T r(s_{t'}, a_{t'}) \right] \right)$$
The first expectation can now be replaced by $0$ due to the lemma mentioned at the top of the post:
$$
\begin{aligned}
\nabla_{\theta} J(\theta)
%
&= \sum_{t=1}^T \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=t}^T r(s_{t'}, a_{t'}) \right]  \\
%
&= \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \left( \sum_{t'=t}^T r(s_{t'}, a_{t'}) \right). \\
\end{aligned}
$$
The expression on slide 18 of the linked slides is an unbiased, sample-based estimator of this gradient:
$$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (a_{i, t} \mid s_{i, t}) \left( \sum_{t'=t}^T r(s_{i, t'}, a_{i, t'}) \right)$$

For a more formal treatment of the claim that we can pull $\sum_{t'=1}^{t-1} r(s_{t'}, a_{t'})$ out of an expectation due to the Markov property, see this page: https://spinningup.openai.com/en/latest/spinningup/extra_pg_proof1.html


Training an AI to recognize my voice (or any voice) I want to start a project for my artificial intelligence class about speaker recognition. Basically, I want to train my AI to detect if it's me who's speaking or somebody else. I would like some suggestions or libraries to work with.

The human voice is based on the neural muscular control of vocal apparatus made up of many parts.

Diaphragm
Vocal cords
Throat (constrictors and anti-constrictors)
Nasal cavity
Cheek
Jaw
Tongue

These coordinated muscular manipulations produce envelopes (controlling) of audio that can be characterized by periodic and transient wave forms.

Volume
Pitch
Tone (relative volume of harmonics)
Consonant transients

Voices are unique to the learning state of neural activity and anatomic attributes, which is a way of saying that vocal habits and the physical attributes of the voice supports the distinguishing of vocal identity.

Strength of vocal muscles
Connectivity of muscles to bone, tendons, and cartilage
Shape of inner surface of vocal pathways
Neural coordination of those muscles
Neural production of phonetic control to produce linguistic elements
Neural serialization of semantic structures (ideas)

The detection of distinguishing features of voices by the ear is equally complex. In a room full of people talking, the brain can learn to track a single voice.

It is important to note that performing voice recognition to determine the identity of the human source is significantly different than performing voice recognition to produce text. To produce text accurately, the NLP must determine language elements and construct a semantic network that represents the vocal content or a text from that representation to be accurate in the case of like sounding words. Fortunately, the identification of the speaker is easier in some ways than the accurate voice to text. Unfortunately, the identification of the speaker has general limitations discussed below.

The first stage of hearing in the ear is mechanical, involving the length of hairs along the cochlear surface, which is like a radio tuner that discriminates all frequencies within a range simultaneously. The software equivalent is a spectrum derived by applying a root mean square to the result of an FFT (fast Fourier transform) to provide magnitudes.

$$ m_f := \sqrt{t_f^2 + {(it_f)}^2} $$

The phase component of the FFT results ($\, \arctan(t, it) \,)$ can be discarded, since it is not correlated with neural control of voice.

The application of the FFT to speech (as with any changing audio) requires windowing over the audio samples using one of the windowing tapers, such as the Hann window or Blackman window. The input is the audio stream or file contents as a sequence of pressure samples, the audio. The output is a sequence of spectra, each containing the volume of each frequency in the vocal range, from about 30 Hertz to 15 K Hertz.

This series of spectra can be fed into the initial layer of one of the more advanced RNNs (recurrent neural networks), such as the LSTM (long short term memory) networks, its bidirectional version, the B-LSTM, or a GRU (gated recurrent network), which is touted as training equally well with less time or computing resource consumption.

The identity of the speaker is the label. The series of spectra are the features.

Using the PAC (probably approximately correct) learning framework, it may be possible to estimate, in advance of experimentation, the minimum number of words the speaker must speak to produce a particular accuracy and reliability in use of the learned parameters from the network training.

It will take some study to set up the hyper-parameters and design the layers of the network in terms of depth (number of layers) and width sequence (number of cells per layer, which may vary from layer to layer).

The use case limitation of this system is that each speaker must read some text that provides adequate training example sequences of adequate length, so that there are sufficient number of overlapping windows for the FFT to transform into spectra so that the training converges reasonably.

There is no way around the individual user training as there is with recognition of linguistic content, which can be trained across a large set of speakers to recognize content somewhat independent of the speaker. The system can be adjusted and improved to minimize the amount of speech required, but information theory constraints keep that quantity from ever approaching zero.

No network, whether artificial or biological, can learn something from nothing. Claude Shannon and John von Neumann realized decades ago that there is a kind of conservation of information, just as there is a conservation of matter and energy in space below nuclear reaction thresholds. This led to the definition of a bit and the formulation of information as a quantity of bits corresponding to a narrowing of probability that the information provides.

$$ b_i = - \log_2 {\frac {P(x|i)} {P(x)}} $$


Why is Python such a popular language in the AI field? First of all, I'm a beginner studying AI and this is not an opinion-oriented question or one to compare programming languages. I'm not implying that Python is the best language. But the fact is that most of the famous AI frameworks have primary support for Python. They can even be multilanguage supported, for example, TensorFlow that supports Python, C++, or CNTK from Microsoft that support C# and C++, but the most used is Python (I mean more documentation, examples, bigger community, support, etc). Even if you choose C# (developed by Microsoft and my primary programming language) you must have the Python environment set up.

I read in other forums that Python is preferred for AI because the code is simplified and cleaner, good for fast prototyping.

I was watching a movie with AI thematics (Ex_Machina). In some scenes, the main character hacks the interface of the house automation. Guess which language was on the scene? Python.

So what is the big deal with Python?

Why is there a growing association between Python and AI?

Python comes with a huge amount of inbuilt libraries. Many of the libraries are for Artificial Intelligence and Machine Learning. Some of the libraries are TensorFlow (which is a high-level neural network library), scikit-learn (for data mining, data analysis and machine learning), pylearn2 (more flexible than scikit-learn), etc. The list keeps going and never ends. You can find some libraries here.
Python has an easy implementation for OpenCV. What makes Python favorite for everyone is its powerful and easy implementation.
For other languages, students and researchers need to get to know the language before getting into ML or AI with that language. This is not the case with Python. Even a programmer with very basic knowledge can easily handle Python. Apart from that, the time someone spends on writing and debugging code in Python is way less when compared to C, C++ or Java. This is exactly what the students of AI and ML want. They don't want to spend time on debugging the code for syntax errors, they want to spend more time on their algorithms and heuristics related to AI and ML
Not just the libraries but their tutorials, handling of interfaces are easily available online. People build their own libraries and upload them on GitHub or elsewhere to be used by others.
All these features make Python suitable for them.


ML model that is most suited to analyse Google Analytics data Google Analytics allows me to collect data about every web-session. For simplicity, let's assume for each user, we collect the number of pages and time spent on site for each session:

user_id visit_id page_views time_spent result
1       1        10         100        0
1       2        31         510        0
1       3        1          10         1

How would you model this data? What I would like the ML algorithm to
    do:

Extract as much information as possible
Have a flexible number of
inputs (e.g. the number of sessions can go to infinity)

What I can think of:

Aggregate the data per user e.g. average page_views or total page_views and feed it into a general algorithm e.g. random forrest (but I lose information with aggregation)
Use LSTM and feed at most last 3 visits (will also lose information, but would this perform better than aggregation?)

Goal:
To build a predictive model to analyse all user sessions and make a prediction whether the person will convert or not.

I understand that in your example you are interested in modelling the outcome of the 'result' column.

One easy model I would suggest is to model it using the Bernoulli distribution (https://en.wikipedia.org/wiki/Bernoulli_distribution) with the probability of success p.

Then you can model p with something like this

x = a + b * log(page_views) + c * log(time_spent) + e

p = exp(x) / (1+exp(x))

where e is normally distributed,
e ~ N(0, sigma^2)
(or simply centered around zero).

a, b and c are parameters that you can estimate.

I.e. the probability of success (conversion) is modeled as a sigma function of a certain variable that depends on page_views and time_spend. You can also add squares (and higher powers) of page_views and time_spend in the equation of x (i.e. upto a certain threshold page_views can have a positive effect on conversion and then a negative one, then again a positive effect).

Also reading about logistic regression should put you on the right track: https://en.wikipedia.org/wiki/Logistic_regression


What is the impact of changing the crossover and mutation rates? What is the impact of using a:

low crossover rate

high crossover rate

low mutation rate

high mutation rate

The crossover rate, $p_c \in [0, 1]$, is a hyper-parameter that controls the rate at which solutions are subjected to crossover. So, the higher $p_c$, the more crossovers you perform, so the more diversity (in terms of solutions/chromosomes) you may introduce in the population. Typical values of $p_c$ are in the range $[0.5, 1.0]$.  For example, in this simple implementation, I used $p_c = 0.8$. In this case, I manually searched for this value (and it didn't take me much time to make the GA find the solution I was looking for), but in other cases/problems you may need to perform some hyper-parameter optimization (e.g. with grid search).
The mutation rate, $p_m \in [0, 1]$ (also a hyper-parameter) controls the rate at which you mutate single chromosomes or the genes of single chromosomes. For example, if you have a binary chromosome of $n$ bits, you could flip the $i$th bit with probability $p_m$, for all $i$ independently. Mutation can prevent premature convergence, but, if $p_m$ is very high, the genetic algorithm becomes a random search, so the value of $p_m$ is typically not high. This paper (which you probably should read because it contains more information about the mutation and crossover operations, although it is a bit old) states that typical values are in the range $[0.005, 0.05]$, but, in this implementation, I used $p_m = 0.5$.
So, in general, higher values of these rates promote exploration at the expense of exploitation. In this sense, this is similar to the $\epsilon$ in the $\epsilon$-greedy behaviour policy in Q-learning (a famous reinforcement learning algorithm).
However, note that each of these operations may undo the other operation. For instance, let's suppose that we have the binary chromosomes $p_1 = [0, 1, 0]$  and $p_2 = [0, 0, 1]$, which we can combine by replacing (with probability $p_c$) the bit at position $i$ of $p_1$ with the bit of the chromosome $p_2$ at the same position $i$. So, after the crossover, you could get the children $c_1 = [0, 0, 0]$ and $c_2 = [0, 1, 1]$. If you perform a mutation by flipping the bits, then you could end up with the same chromosomes as the original ones, that's also one of the reasons why you may not want to have a very high mutation rate. So, it's not completely true that, the higher the mutation rate, the more you explore, because you can end up with the same solutions as before, so you have not explored anything.
You could also have an adaptive value for these parameters. See e.g. this paper for an example of such an adaptive schedule.


How should I approach the game "Achtung, Die Kurve" ("Curve Fever") using AI? One of my friends built a version of "Achtung, Die Kurve!", or "Curve Fever" in Python. I was starting to study ML and decided to tackle the game from a learning perspective - write a bot that would crush him in the game. Did some research and found Deep Q learning. Decided to go with that and after a whole lot of throwing around different hyperparameters and layers, I decided I need some help on this. I am new to Deep and Machine Learning in general, so I may have missed things. I was kinda discouraged when I saw that Deep Q is SO impractical currently in the field.

how would you guys tackle this problem? I need some guidance/help building it if someone is up to the task.

Start slowly.
Don't jump straight into Deep Learning, arguably the most complex class in Reinforcement Learning techniques. First work with simpler algorithms, like the original Q-Learning. Define what are good inputs and outputs for your game, and start tuning some hyper-parameters (like future rewards discount factor).
From there, go for Deep Learning. Check other implementations (like DQN, Atari n-step Q-Learning and A3C), and adapt their code to yours, rather than starting from scratch.


What is a "surrogate model"? In the following paragraph from the book Automated Machine Learning: Methods, Systems, Challenges (by Frank Hutter et al.)

  In this section we first give a brief introduction to Bayesian optimization, present alternative surrogate models used in it, describe extensions to conditional and constrained configuration spaces, and then discuss several important applications to hyperparameter optimization.

What is an "alternative surrogate model"? What exactly does "alternative" mean?

A surrogate model is a simplified model. It is a mapping $y_S=f_S(x)$ that approximates the original model $y=f(x)$, in a given domain, reasonably well. Source: Engineering Design via Surrogate Modelling: A Practical Guide

In the context of Bayesian optimization, one wants to optimize a function $y=f(x)$ which is expensive (very time consuming) to evaluate, therefore one optimizes the surrogate model $y_S=f_S(x)$ which is cheaper (faster) to evaluate.


Markov Model for a Traffic Intersection I need some help in developing a Markov Model for a crossroads there is no one way road and i am assuming at this time that traffic is only allowed to go straight no turns are allowed. There are 4 roads and 4 signals(agents).

Why not use a single agent to control the intersection with the following rules:

define 8 traffic lights
Each light has two possible values, 0 or 1, with 0 = Red and 1 = Green
Lights are as follows:

0 = North to South
1 = North to East (Left Turn)
2 = East to West
3 = East to South (Left Turn)
4 = South to North
5 = South to West (Left Turn)
6 = West to East
7 = West to North (Left Turn)

map lights into a bitmap of 8 bits, with the least significant bit (left bit) is the North to south

Given the above, we have the following valid moves, which are both State Space  and Action Space :

State/Move 0 - Vertical Moves only   = 10001000
State/Move 1 - Horizontal Moves only = 00100010
State/Move 2 - Originated from North only = 11000000
State/Move 3 - Originated from East only = 00110000
State/Move 4 - Originated from South only = 00001100
State/Move 5 - Originated from West only = 00000011
State/Move 6 - Vertical Diagonal only = 01000100
State/Move 7 - Horizontal Diagonal only = 00010001

i.e. S = {10001000, 10001000, 00100010, 11000000, 00110000, 00001100, 00000011, 01000100, 00010001}

and  A = {10001000, 10001000, 00100010, 11000000, 00110000, 00001100, 00000011, 01000100, 00010001}

Now we have a ∈ A and s ∈ S and at any given s, a will cause the next state s' to have the same value of a

Final component to complete the moel, is the reward signal, which can be simply the negative sum of time spent per car in the intersection

i.e.     rt+1 = -∑ci ∈ Cars time(ci)

a single agent, will be able to optimize the solution to avoid punishment (negative value caused by wait time)

Update

You can increase the state space to double or triple its size to accommodate for time, by appending a time counter component to the bitmap.
in addition to an extra (9th) action of do nothing,

This should give a more subtle traffic controller. e.g. Vertical Traffic Only, at timer = 0 can be represented by 100010000000 or in hex 880, then at time 1 = 881, and so on until time 15 to be 88F.

Update 2

You can find the source code here and here


When does the selection phase exactly end in MCTS? All sources I can find provide a similar explanation to each phase.

In the Selection Phase, we start at the root and choose child nodes until reaching a leaf.  Once the leaf is reached (assuming the game is not terminated), we enter the Expansion Phase.

In the Expansion Phase, we expand any number of child nodes and select one of the expanded nodes. Then, we enter the Play-Out Phase.

Here is my confusion. If we choose to only expand a single node, the nodes that were not expanded will never be considered in future selections as we only select child nodes until a leaf is reached during the Selection Phase. Is this correct? If not, what am I misunderstanding about the Selection Phase?

If we choose to only expand a single node, the nodes that were not expanded will never be considered in future selections as we only select child nodes until a leaf is reached during the Selection Phase. Is this correct?

No this is not correct.

  If not, what am I misunderstanding about the Selection Phase?

The selection phase does not end only when you reach a node that has no expanded nodes. It ends when you reach a node that has any unexpanded nodes. At which point you typically pick one or more nodes you have not yet expanded at that point in the tree, expand them and collect one or more rollout results for them. Variations are possible such as choosing to whether to expand or continue selecting stochastically, or expanding all child nodes at the same time using value estimates to initialise them - the latter is what AlphaZero does.


Find the expected reward in an expectimax-based dice rolling game? I have this question that I'm kinda stuck on.
It's a game scenario in which we set up an expectimax tree. In the game, you have 3 dice with sides 1-4 that you roll at the beginning. Then, depending on the roll, the player can choose one of the dice to reroll or not reroll anything. Points are assigned like so:

10 points if there's 2 of a kind
15 if there's 3 of a kind
7 if there's a series like 1-2-3 or 2-3-4
Otherwise, or if the sum is higher than the rewards from above, the score = sum of the rolls

For additional context, this is an example expectimax tree I came up with, for the case that the player rolled a 1,2,4 and is considering rerolling or not:

Now lets introduce a new agent -- a robot that's supposed to help the human player. we assume

the human player choses any action with uniform probability regardless of the initial roll
there's a robot that, given a configuration of dice and the human's desired action, actually implements the action with probability 1-p and overrides it with a "no reroll" order with probability p>0. It has no effect if the human's decision is already to not reroll.

For that scenario, I came up with this expectimax tree:

Now for the part I'm actually stuck on -- lets define A, B, C, and D as the expected reward of performing actions "reroll die 1", "reroll die 2", "reroll die 3", and "no reroll." How do we find $R_H$, the expected reward for the human acting without the robot's help, and $R_{AH}$ the expected reward for if the robot helps? We can't use p in the expression, we only have access to A,B,C,D and we're supposed to write it in the form $X + Y_p$
*EDIT: I asked again and the question was worded weirdly. They said we should definitely use p. What was meant by not using p is X and Y themselves can't contain p. But Y will be multiplied by p in the final simplified form.
For $R_{H}$ I think the answer should be $\frac{(A + B + C + D}{4}$ because of uniform distribution over A-D.
I'm supposing that $R_{AH}$ would be $\frac{(A + B + C)(1-p) + D + Dp}{4}$? Because the robot doesn't override with probability $1-p$, but he can only override A-C and does with probability $p$.
I think something feels slightly wrong about my answer but I'm not sure what.

If I understand,

With probability $p$ the robot select no re-roll (action $D$).
With probability $1-p$ the human uniformly select an action between $A,B,C,D$.

Thus the expected reward for if the robot helps is
$$R_{AH}= p\cdot D+(1-p)\frac{A+B+C+D}{4} = \frac{(A+B+C)(1-p)+D+3pD}{4}$$


How to handle a changing in the Reinforcement Learning environment where there is increasing or decreasing in number of agents? I'm working in A2C and I have an environment where there is increasing or decreasing in the number of agents. The action space in the environment will not change but the state will change when new agents join or leave the game.
I have tried
encoder-decoder model with attention but the problem is that the state and the model will change when the number of agents is changing.
I also tried this way where they use LSTM to get the Q value for the agent but I got this message
Cannot interpret feed_dict key as Tensor: Tensor Tensor("state:0", shape=(137,), dtype=float32) is not an element of this graph.

or error like this because of changing of state size
ValueError: Cannot feed value of shape (245,) for Tensor 'state:0', which has shape '(161,)'

(1) Are there any reference papers that deal with such a problem?
(2) What is the best way to deal with the new agents that join or leave the game?
(3) How to deal with the changing of state space?

I depends on your overall model architecture (and problem specification). As I understand it, you take the observations of all agents together and feed it into one model, a central controller, which then predicts the action per available agent.
I believe that this varying number of applicable observations (depending on the number of currently present agents) is what you mean by changing state spaces?
One option that works in these kinds of settings is to use padding. So, a fixed number of individual observations gets predetermined before training, being set to the average number of agents expected to be present in the environment at any timestep.
Then, during training, local, individual observations of all agents (for the current time step) get stacked together, resulting in an overall observation being fed into central controller.
Then, the controller predicts (concurrently/in a single pass) one action (or multiple, depends on what you want it to predict per agent...) per agent.
In the case that the number of present agents currently (=in a given time step) is smaller than the predetermined number of expected agents, then padding is used to fill the empty observation slots by zero-pseudo-observations. The quality of the predicted actions is consequently only assessed for the present agents, neglecting predictions for absent agents, indicated by their empty observation slots.
If the number of controlled agents happens to exceed the expected maximal number of agents present in the simulation, excess agents are selected at random and randomly controlled for the next time step. Then, this (randomized control of a random selection of some agents) continues until no excess agents are present in the simulation anymore. Alternatively, if available, other reasonable controllers might be used to temporarily steer excess agents.
I have seen researchers doing this in the context of traffic control, where a variable number of cars was controlled by a central controller.
See this paper (Section 3.2) for an example of the aforementioned technique.
Alternatively, there exist approaches where each agent gets a periodically updated local copy of some central controller and then contributes to jointly updating the central controller. This paper shortly summarizes a lot of different options and possibilities to choose from in terms of different RL algorithms, where different options might apply depending on the concrete task at hand. Also, the paper discusses different options with respect to algorithms implemented in the python Reinforcement Learning (RL) library RLlib, which might already provide a suitable implementation for a class of algorithm you want to work with.
Unfortunately, a lot of papers don't reveal explicitly whether their (partly) distributed RL training procedures are more suitable, i.e. converge stably to a good solution, for single-agent environments or equally suitable for multi-agent environments as well.
When it comes to introducing new agents to the simulation, this depends entirely on your concrete problem at hand. In the context of vehicle control, you could position a vehicle at the start of the simulated road, steer it (safely) a few time steps straight ahead (to initialize its observation space) and then hand over control to the RL controller. For other types of agents (e.g. soccer playing robots etc.) that might look entirely different. In that case, you might have them sitting or standing next to the field and then have them enter the play ground or so. So, that really depends on your task at hand.