How does one even begin to mathematically model an AI algorithm? How does one even begin to mathematically model an AI algorithm, like alpha-beta pruning or even its thousands of variations, to determine which variation is best?

Begin by learning the mathematical treatments at their foundations.

Game theory pioneered John von Neumann and Oskar Morgenstern
Information theory pioneered by Claude Shannon (Bell Labs)
Incompleteness pioneered by Kurt Gödel (which led to Alonso Church's lambda calculus and Turing's completeness which led to a general criteria that defines what a programming language must do to be called general purpose.

You could take a course in Finite Math and Discrete Math, but there is no better way to understand than to go to the authors of the original ideas, which is why MIT and Cal Tech require it of freshmen.  We don't need to be genius to understand genius.  We just need to take the time to read.  That's the genius of them.

It is good to evaluate the pruning of decision trees mathematically, and modeling the general and specific cases is the right approach.  Congratulations for seeing that.  The code for pruning is far more developed than the mathematics for modeling it.  The literature usually shows speed results just before the conclusions, which is more like sitting in the audience with pop corn and a wager ticket than understanding the anatomy and proper care of a horse.

Once you understand what occurred a century ago that led to all computer science and see what mathematical conventions originated and in what context then you can read work like

Appl. Math. L&t. Vol. 4, No. 6, pp. 77-80, 1991 Printed in Great Britain. 08939659191 Pergamon Press plc Turing Computability with Neural Nets T. SIEGELMANN AND EDUARDO D. SONTAG Department of Computer Science, Rutgers University Department of Mathematics

... and ...

A Survey of Decision Tree Classifier Methodology S. Rasoul Safavian David Landgrebe TR-EE 90-54 September, 1990 School of Electrical Engineering Purdue University West Lafayette, Indiana 47907 (NASA-CR-188208).

WARNING: Don't be surprised if you are disappointed by most recent work after reading von Neuman, Shannon, and Gödel.


How to calculate the false positives and negatives? I have a huge amount of data and I want to calculate my false positive and false negative. Is there a software that can help me determine it?

Yes, you can use sklearn's confusion_matrix. To explicitly extract the false positives and negatives, you can do

from sklearn.metrics import confusion_matrix

y_true = [0, 1, 0, 1]
y_pred = [1, 1, 1, 0]
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()


Understanding the perceptron algorithm in the book "A Course in Machine Learning" The following text is from Hal Daumé III's "A Course in Machine Learning" online text book (Page-41).

I understand that $D$ is the size of the input vector $x$.

What is $y$? Why is it introduced in the algorithm? How/where/when is the initial value of $y$ given?
What is the rationale of testing $ya \leq 0$ for updating weights?

Y is the desired output of the perceptron (often referred to as target) , for the given set of input vectors.

Rationale behind Y.a<=0 :

Prerequisite knowledge :

A=A-B : Moves vector A away from direction of vector B
A=A+B : Moves A in the direction of B
A (.) B >0 ; A vector is directed acutely (<90 deg.) towards B vector
A (.) B <0 ; A vector is directed away from (>90 deg.) from B vector [(.) denotes dot (scalar) product and bold letters indicate vectors]
W is augmented vector (includes threshold as another weight along with normal input weights)
X is augmented input (includes -1 as extra input (corresponding to threshold) along with other normal inputs
a(activation) = W (.) X
a >=0 ; Perceptron output 1
a<0 ; Perceptron output -1 (Not zero as implicit in the given algorithm I think)

Now Rationale :

(Y.a<0)

This means
Either of the following :

Y=-1 and a>0 ; in this case the target output is -1 but as a>0 so the Perceptron outputs 1. So we must move the weight vector away from this set of input vector, so that angle between them increases and the dot product (a) becomes < 0 so that we can get the target output.
Hence : W=W-X

Or ,W=W + (-1)*X
Or, W=W+YX

Y=1 and a<0

This means the target output is 1 but as the activation is <0 so Perceptron is outputting -1. So we must move the weight vector close to this set of input vector so that the activation can become >0 (angle decreases) and the Perceptron can output the desired output.
So :
W = W+X
Or W=W + YX

Again , Y.a=0 is a boundary case.
By now I think you can understand the rationale behind Y.a=0. If any doubt , comment to this answer , I will explain it.

Sorry for so much long answer though. :) :)


How to fairly conduct a model performance with 5-fold cross validation after augmentation? I have, say, a (balanced) data-set with 2k images for binary classification. What I have done is that

randomly divided the data-set into 5 folds;
copy-pasted all 5-fold data-set to have 5 exact copies of data-set (folder_1 to folder_5, all absolutely same data-set)
first fold in folder_1 is saved as test folder and remaining (fold_2, fold_3, fold_4, fold_5) are combined as one train folder
second fold in folder_2 is saved as test folder and remaining (namely, fold_1, fold_3, fold_4, fold_5) are combined as one train folder
third fold in folder_3 is saved as test folder and remaining (namely, fold_1, fold_2, fold_4, fold_5) are combined as one train folder.
similar process has been done on folder_4 and foder_5.

I hope, by now, you got the idea of how I distributed the data-set.

The reason I did so is as follows:

I have augmented the training data (train folder) in each of the folders and used test folders respectively to evaluate (ROC-AUC score). Now I kind of have 5 ROC-AUC scores which I evaluated using test folders. If I get the average value out of those 5 scores.

(Assuming the above cross-validation process is done right) If I were to perform some manual hyperparameter optimizations (like an optimizer, learning rate, batch size, dropout, activation) and perform the above cross-validation with data augmentation and find the best so-called "mean ROC-AUC", does it mean I successfully conducted hyper-parameter optimization?

FYI: I have no problem with computing power OR/AND time at all to loop through the hyper-parameters for this type of cross-validation with data augmentation

If you used your five $X_{test}$ sets multiple times (to measure the average AUC) to decide on the best set of hyperparameters (i.e. optimizer, learning rate, batch size, dropout, activation) then yes, you successfully conducted hyper-parameter optimization. However, the AUC you received for the best set of hyperparameters found (by manual tuning) is not representative of the real performance of your model.

This is because the fact of using a test set to tune the parameters of your model degenerates it back to a "training" set, because the data is not being used to measure the performance of the model but to improve it instead (although on a different level of abstraction, i.e. not to directly influence the parameters of the model, such as neural network weights), making the resulting AUC an overly optimistic biased estimator of the real AUC (that it could have resulted in for an unseen test dataset).

That's why, if you care both about hyperparameter optimisation and being able to measure the "real" performance of your model, you need to split your dataset into three "buckets": training set $X_{train}$, validation st $X_{val}$ and test set $X_{test}$. $X_{test}$ should only be used once (after you've trained and tuned the model), and assuming it has enough samples and the samples are representative of the unseen data, you should get a good estimate of your model performance. $X_{val}$, on the other hand, is your validation set which you can reuse as many times as you want to find the optimal set of hyperparameters that result in the highest performance (i.e. AUC).


Continuous control with DDPG: How to eliminate steady state error? Currently I'm working on a continuous control problem using DDPG as my RL algorithm. All in all, things are working out quite well, but the algorithm does not show any tendencies to eliminate the steady state control deviation towards the far end of the episode.

In the graphs you can see what happens:

In the first graph we see the setpoint in yellow and the controlled continuous parameter in purple. In the beginning, the algorithm brings the controlled parameter close to the setpoint fast, but then it ceases its further efforts and does not try to eliminate the remaining steady state error. This control deviation even increases over time.

In the second graph, the actual reward is depicted in yellow. (Just ignore the other colors.) I use the normalized control deviation to calculate the reward: $r = \frac{\frac{|dev|}{k}}{1+\frac{|dev|}{k}}$.

This gives me a reward that lies within the interval $]0, 1]$ and has a value of $0.5$ when the deviation $dev$ equals the parameter $k$. (That is the parameter $k$ kind of indicates when half of the work is done)

This reward function is relatively steep for the last fraction of the deviation from $k$ to $0$. So it would definitely be worth the effort for the agent to eliminate the residual deviation.

However, it looks like the agent is happy with the existing state and the control deviation never gets eliminated. Eventhough the reward is stuck at ~0.85 instead of the maximum achievable 1.

Any ideas how to push the agent into some more effort to eliminate the steady state error?
(A PID controller would exactly do this by using its I-term. How can I translate this to the RL-algo?)

The state presented to the algo consists of the current deviation and the speed of change (derivatve) of the controlled value. The deviation is not included in the calculation of the reward function, but in the end we wat a flat line with no steady state deviation of course.

Any ideas welcome!

Regards,
Felix

Could you figure out a workaround for this problem of the steady-state error using DDPG?

I'm currently facing the same problem. My application is Satellite attitude control, and no matter which cost function variation I use, the resulting controlled system maintains a constant steady-state error.

It seems to be a limitation of the algorithm. But I can't understand why this happens. Hence, I've decided to use PPO for now. But would like to know if you could solve this problem using DDPG.

All the best,

Wilson


Is it a good idea to pay for an deep learning course? I'm an artificial intelligence enthusiastic and I want to learn about it.

I want to ask you what do you think about the Udacity nanodegree Deep Learning Nanodegree Foundation.

I don't know if it is a good idea to pay for that course or maybe, there are better free resources.

I want to understand what artificial intelligence is, and also learn about machine learning, deep learning, and convolutional networks. I'm interested in image and speech recognition and also in artificial life.

It doesn't seem expensive at $399* (although the asterix needs to be taken into consideration.)

If you're interested in this subject, this may be a decent course, however it is certainly not an accredited institution, thus any "degree" you get from this course will be meaningless in an academic sense.  On the other hand, a certificate shows that you've committed resources toward training.

My high level take on this is that the "Foundation" is looking to capitalize on the recent publicity for Deep Learning, per the AlphaGo milestones.

One thing I can tell you for certain is that this field requires advanced mathematics from a research perspective, and people who work in that area spend years training to gain the requisite skills.

The requirements for this class seem to be restricted to "Python knowledge" with no mention of mathematics, which raises some alarm bells, although it's worth mentioning that once solutions/applications arise, they can be applied to varying degrees without having to understand the mathematics. [See mindcrime's comment]


Does this prove AI Safety is undecidable? Does this prove AI Safety is undecidable?

Proof:

Output meaning output to computer program.

[A1] Assume we have a program that decides which outputs are “safe”.

[A2] Assume we have an example of an unsafe output: “unsafe_output”

[A3] Assume we have an example of safe output: “safe_output”.

[A4] Define a program to be safe if it always produces safe output.

[A5] Assume we have a second program (safety_program) that decides which programs are safe.

[A6] Write the following program:

def h()
   h_is_safe := safety_program(h)
   if (h_is_safe):
      print unsafe_output
   else:
      print safe_output

Clearly h halts.

If the safety_program said h was safe, then h prints out unsafe_output.

If the safety_program said h was not safe, then h prints out safe_output.

Therefore safety_program doesn’t decide h correctly.

This is a contradiction. Therefore we made a wrong assumption: Either safe output cannot be decided, or safe programs cannot be decided.

Therefore, in general, the safety of computer programs, including Artificial Intelligence, is undecidable.

Therefore AI Safety is undecidable.

In my opinion, there are several flaws in your proof and reasonings.

First, note that, in the case of Turing's proof, h will actually loop forever (i.e. not halt) when the oracle says that h halts. In this case, there's an actual contradiction, because h will do the opposite of what the oracle says.

So, to follow Turing's proof, you would need to make h behave unsafely if the oracle says h is safe. But how should we define a safe or unsafe program? There are many unsafe behaviors. For example, in a certain context, an insult could be unsafe, in other contexts, a certain limb movement could be unsafe, and so on. So, an agent is unsafe or behaves unsafely usually with respect to another agent (or itself) or environment. You probably need to keep this in mind if you want to prove anything about the safety of AI agents.

In your second assumption, you are implicitly saying that any machine that produces the output unsafe_output is unsafe, but, of course, this definition is not a realistic definition of an unsafe program.

To help you define safety in a more reasonable and natural way, I think it may be useful to reason first in terms of artificial agents, which are higher-level concepts than Turing machines. Then you could find a way of mapping agents to TMs and attempt to prove your conjectures by using the tools of the theory of computation.


What approach should I take to model forecasting problem in machine learning? I have a dataset which contains 4000k rows and 6 columns. The goal is to predict travel time demand of a taxi. I have read many articles regarding how to approach the problem. So, every writer tell his own way. The thing which I have concluded from all my readings is that I have to use multiple algorithms and check the accuracy of each one. Then I can ensemble them by averaging or any other approach.

Which algorithms will be best for my problem accuracy-wise? Some links to code will be helpful for me.

I currently only have training set of data. After I work on it, it will be evaluated on any testing set by my professor. So, what should I do now? Either split data I have into my own testing and training set or separately generate dummy data as a testing set?

In general, this type of problem is called a regression problem since the target variable (i.e. travel time) can take any value in a continuous domain. In theory, you can use any regression algorithms (a subset supervised learning techniques) to solve this problem. Some of the most popular ones are linear regression, K-nearest neighbor (regressor), and neural networks.

As you observed already, different algorithms result in (sometimes significantly) different results. Also, the parameter configurations (e.g., number of hidden layers in Neural Networks) can make a big difference. Sometimes, ensembling different models can be helpful, but in general, you should try to avoid overfitting (when your model is more complex than your data such that it memorizes the training set instead of learning it!). That may result in a very good performance on your training set but perform very poorly on your professor's testing set.

What I would do is:

exploring the dataset to see what are the contributing factor in travel time (any correlation between the columns).
cleaning and preprocessing my dataset (duplicates, null values, outliers)
reshaping my dataset if needed (normalizing some columns, merging or splitting columns)
dividing my dataset to training and evaluation subsets (so I train on one part and test on the other part to avoid overfitting)
choosing a simple baseline, applying and measuring the accuracy metrics.
trying to fine tune parameters of my baseline or trying other more advanced techniques.
comparing the results and improving any part of the pipeline when necessary (more/less cleaning, parameter tuning, ensembling).


ReLu, Sum and Convolution Layers to Count Pixels of Certain Color Below is an excerpt in an instructor's manual on ML that is explaining deep neural networks, using cat recognition (what else!) from images as example. On how DL performs this feat, the excerpt said that,

  Assume that the first layer returns the number of pixels that are
  brown/black/blue/red, and the second layer finds the most common
  color, and the third layer returns “cat” if previous layer had
  supplied “brown”. [..] Mathematically, this model would be, for the
  first layer, [ sum(r = 255, g=255, b=255), ..., ..., sum(r=255, g=0,
  b=0)] -- this is just a set of appropriately positioned relu functions
  (okay, for r=234, we’d need two relu functions, so two layers, but you
  get the idea). The second layer would be a softmax layer. The third
  layer is simply an identity!

Now I worked with deep nets, but I am not sure how I can structure a DL to do this. ReLu is simply a max(0,x), so how would I filter out pixel vals for example 128,128,128 and sum them up? Wouldn't the convolution layer play a role here too? What would the layout of a simple deep net be that does what is described above?

Thanks,

I can judge from the excerpt only, but I don't see any mention of a convolutional layer there. This doesn't mean that convolution isn't suitable for this task (in fact, it's the best method for image classification so far), it's just seems the instructor proposes this particular network architecture, without convolutions.

Since this architecture is more illustrative and by no means a standard, I can't be sure I fully understand it. But it seems like the first layer outputs a vector of size 4, which contains the counts of four selected colors. The ReLu operation here is a fancy way of filtering extra colors, based on channel value (note that they are close to either 0 or 255 in all four colors). If you select different four colors, this filtering will have to be done differently too, so it's basically hardwired. The second layer is more straightforward.

In general, you shouldn't consider this a "true" deep network, but merely a puzzle to get familiar with ReLu operation.


In AlphaZero, which policy is saved in the dataset, and how is the move chosen? I've been doing some research on the principles behind AlphaZero.
Especially this cheat sheet (1) and this implementation (2) (in Connect 4) were very useful.
Yet, I still have two important questions:

How is the policy network updated? In (2), board positions are saved in a dataset of tuples (state, policy, value). The value is derived from the result of the self-played game. However, I'm not sure which policy is saved: the number of times that each move has been played, the prior probabilities for each move (I guess not), or something else?

The cheat sheet says that (for competitive play) the move is chosen with the greatest N (=most visited). Wouldn't it be more logical to choose the move with the highest probability calculated by the policy head?

However, I’m not sure which policy is saved

The policy from the Monte Carlo tree search is stored, as we can get the policy estimate from the network later by passing the given state through the network, which is used to calculate the cross entropy loss to update the network's policy (summed with Mean squared error loss between value head's prediction and the actual value/reward).

Wouldn’t it be more logical to choose the move with the highest probability calculated by the policy head?

Depends on the number of searches you've performed, after thousands of simulations the MCTS would give better results, as it approximates the minimax tree.


Could AI understand what the number one is and detect it in our real world same as AI can recognize cars and other things? I have a solid understanding what the numbers are. If I want, I can see numbers in everything. Could an AI have the same ability for any incoming information to tag them by numbers same as I have?

A understanding for this level of abstraction is technically possible.
It is not hard to create an AI able to count.
Unfortunately, this does not imply that the AI knows what 1 is.
It knows: This is 1 piece of cake and this is 1 sheet of paper.
But the idea of the number is not grasped yet.
Also, animals can count, but they do not understand why it works.

I would say that not even every human knows what 1 is:
The number, who is the neutral element of the multiplication.
The natural successor of 0.

I can say with certainty, that there is no AI, who reached this level of abstraction.


Is there actually a lack of fundamental theory on deep learning? I heard several times that one of the fundamental/open problems of deep learning is the lack of "general theory" on it because actually we don't know why deep learning works so well. Even the Wikipedia page on deep learning has similar comments. Are such statements credible and representative of the state of the field?

There is a paper called Why does Deep Learning work so well?.

"However, it is still not fully understood why deep learning works so well. In contrast to GOFAI (“good old-fashioned AI”) algorithms that are hand-crafted and fully understood analytically, many algorithms using artificial neural networks are understood only at a heuristic level, where we empirically know that certain training protocols employing large data sets will result in excellent performance. This is reminiscent of the situation with human brains: we know that if we train a child according to a certain curriculum, she will learn certain skills — but we lack a deep understanding of how her brain accomplishes this."


What AI service can define personality portfolio based on text? Imagine this theoretical situation:

  A group of people are asked to provide a solution for an imaginary
  problem via email. Then an AI service runs through their written
  solution, analyzing the kind of words and sentences they use,
  essentially exploring how they faced the problem. Then the service
  finds a personality portfolio of each person.

Is there such an AI service which can find such personality portfolios by analyzing the text they use describing their solution to a problem? If there is not a current service, how might it be approached?

Regards,
Koppany

My first recommendation would be before you create an AI or ML based solution. Kindly consider using a business Q&A Software such as Questions for Confluence by Atlassian among others. An enterprise multiple choice solution could be a simple and elegant fit for this problem.

However, if one was to design a solution in accordance to your specification the first step to resolve the problem would definitely be one of applying NLP. In my answer I utilize the open source procedure laid out by EASE (Enhanced AI Scoring Engine) by Edx and a combined initiative of MIT and Harvard towards improving automated essay scoring (Parachuri V 2013).

Computers cannot directly understand text the way people can. People naturally break sentences into units of meaning, however when using a computer we will first have to instruct the computer how to do it. This process is called tokenization. Tokenization is the task of chopping a phrase into pieces called tokens at the same time discarding some characters i.e. punctuation.

After we tokenize the essay answers, we then convert the tokens into a matrix (bag of words model also known as vector space model). This model is a very convenient way to represent text in matrix form. Here we disregard grammar and even the word order but the frequency of words is retained.

Now we can proceed to the machine learning part!

Here I will recommend that we either use regression or classification. The difference between these two algorithms is that regression assumes you are using a continuous scale while classification is discrete.

In statistical modeling, regression is a set of statistical processes for estimating the relationships in variables.

  A simple linear equation is $y=m*x+b$ , where y is the target value(score), m is a coefficient, and b is a constant.
  In linear regression, we would do something like $y=m{1}*x{1}+m{2}*x{2}+\dots+m{n}*x{n}+b$.

Classification (also sometimes called a decision tree) on the other hand is a data mining method for assigning categories to a collection of data which is useful since we want to classify essays on the basis of their scores.

The next step will be to measure the error in the scores that are generated by the algorithm on the previous step.

In order to measure accuracy, lets use cross-validation. Here we split the training dataset randomly into n parts each segment is a 'fold'. This is called n-fold cross validation. We then iterate from 1 to n predicting the scores of the parts [n] from all the data in parts [!n].

Hope this helps.


Why do we need a model of the environment in Dyna? In chapter 8 of "Reinforcement Learning: An Introduction" by Sutton and Barto, it is stated that Dyna needs a model to simulate the environment.

But why do we need a model? Why can't we just use the real environment itself? Wouldn't it be more helpful to use real environment instead of fake one?

Unlike algorithms presented in other chapters of Sutton and Barto, Dyna is a planning algorithm. That means that it makes decisions online, in a real environment, that attempt to be as optimal as possible given some constraints such as current knowledge and time available to compute between time steps. This differs from learning-only online algorithms which typically take a small step towards optimality on each piece of new experience as it happens.

A planning algorithm can only do its job well if it is allowed to "look ahead" at the consequences of its behaviour whilst learning online. In fact, this is the definition of planning - to choose an action based on reasoning about consequences of that action.

For an algorithm to look ahead before taking an action, it needs a model of how the environment will respond to that action. That model does not need to be coded up directly - e.g. you don't necessarily need to write a physics engine to predict the real world (although a basic one might be a good prior or pre-training step). Instead it can be a learned model, and typically in e.g. Dyna-Q, that is what you use.

There is a strong relation between Dyna-Q, and regular Q-learning with experience replay. In the most basic forms, they are essentially the same algorithm with a different framing. However, you can take the planning ideas further e.g. focus improvements around the currently experienced state and paths to a goal state in Dyna-Q, perhaps making it closer to MCTS conceptually.

  Wouldn't it be more helpful to use real env instead of fake one?

Most real environments do not let you take actions, see the consequences and then rewind in order to re-try. Essentially that is what planning algorithms are making up for - they try to predict consequences. This is important when mistakes made during training have real consequences, for example for a physical robot navigating an environment where there might be a possibility of a fall or collision that damaged something. Whilst online learning algorithms such as SARSA will also help with this in different ways (in SARSA by changing policy to allow for exploratory moves), typically Q-learning will be weaker than Dyna-Q when it comes to learning quickly from mistakes. With the usual caveat: Much still depends on the specific problem and choices of hyperparameters.


Which algorithm is used in the robot Sophia to understand and answers the questions? Which algorithm is used in the robot Sophia to understand and answer the questions?

Sophia ,first , has all the questions and corresponding answers preprogrammed. It is a system which is a hybrid of Bayes Text classification and decision trees.

It may consist of a speak recognizer which converts the question into a string.
This string then travels into a algorithm which gets a suitable answer for it.
The answer is then spoken by the speech synthesizer along with some face actions.
It might also have a sentiment analyser which identifies sentiments in a answer to show face expressions accordingly.
Along with this, it identifies the emotions on the face of the person sitting in front of it, so as to provide a greater expression.

Hence, Sophia understands or thinks nothing like the human brain. It is just a system which gives predefined answers corresponding to the question asked.


Does a neural network exist that can learn every possible training data? The universal approximation theorem states, that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $R^n$.
Michael Nielsen states

No matter what the function, there is guaranteed to be a neural network so that for every possible input, $x$, the value $f(x)$ (or some close approximation) is output from the network.

So, for continuous functions, this seems plausible. Interestingly, in the same article, Nielsen mentioned "any function".
Later, he writes

However, even if the function we'd really like to compute is discontinuous, it's often the case that a continuous approximation is good enough.

The last statement leaves open a gap, to ask how well an approximation practically can be.
Let's ignore contradictory input/output training pairs like $f(0)=0$ and $f(0)=1$, which actually don't event represent a function anyway.
Furthermore, assume that the training data is generated randomly, which would practically result in a discontinuous function.
How does a neural network learn such data? Will a learning algorithm always be able to find a neural network that approximates the function represented by the input-output pairs?

The branch of AI research that answers questions like this is called computational learning theory.

For the specific question you have asked, the universal approximation theorem does indeed prove that any function can be modeled by a sufficiently wide neural network. The definition of a function includes the requirement that each input be mapped to exactly one output, so contradictory labels in training data are excluded explicitly.

Here is a rough sketch that can provide an intuition behind why this is true. This is not a proper proof, but it gives you an idea of the power of "finite number of neurons" in a hidden layer:

A single neuron can essentially learn to draw a straight line across the space of input features. It outputs a value arbitrarily close to 1 for things on one side of the line, and arbitrarily close to 0 for things on the other side of the line.
For any given datapoint, it is possible to enclose the hyper-volume containing that datapoint, and no others by drawing a series of lines in the input space, and defining one side of each line as "inside" the hyper-volume, and the other side as "outside". In 2-d space, this corresponds to drawing the 4 sides of a square around a point.
An output neuron learns to draw lines across the outputs of the hidden layer neurons. It can therefore decide to output 1 only when, say, 4 other neurons are all simultaneously active.

It should seem natural then that a sufficiently wide neural network can memorize all of an input pattern. Since memorization is sufficient for "learning" to fit a pattern this should give you an intuition for the ability of neural networks to fit things.


Choosing Instance Semantic Detection A fixed video camera records people moving through its field of view.

The goal is to detect and track the head, in real-time as it moves through the video.  The norm is there are many heads, which often are sometimes partially obscured.  This example video boxes heads and provides a head count.

There seems to be many different models.  Examples include:

Adaboost-haar Head detection
MASK R-CNN
LBP Cascade

Given the context of the video, what is the thought process that you would use to choose a model?

There may be a conceptual disconnect between the term Semantic Detection and the task of Head Tracking, since sequential recognition of an object in a set of visual samples representing continuous movement isn't technically a semantics problem.  Although a mapping strategy that works with semantic processing may somehow apply to, with appropriate modifications, the mapping of obstructions and tracked objects and their relationship in space-time.

Tracking a head is primarily a surface fitting convergence problem, where the convergence goal and the process that achieved it must be sustainable, especially if the system is required to take actions during the tracking operation.  Distinguishing heads from one another requires a high degree of reliability and accuracy in this convergent continuity.  That is the primary challenge.

Drawing boxes and displaying counts are trivial operations once the head tracking works.

These are some models listed in the question or improvements over them.

Fast LBP Face Detection on low-power SIMD architectures, Olexa Bilaniuk, Ehsan Fazl-Ersi, Robert Laganiere, Christina Xu, Daniel Laroche, Craig Moulder
*A Real-Time AdaBoost Cascade Face Tracker Based on Likelihood Map and Optical Flow, Ranftl, Alonso-Fernandez, Karlsson, Bigun, IET Biometrics, 2017, Vol. 6, no 6, p. 468-477
A Real-Time AdaBoost Cascade Face Tracker Based on Likelihood Map and Optical Flow, IET Biometrics, 6(6): 468-477
A New Approach to Urban Pedestrian Detection for Automatic Braking, Alberto Broggi et. al. IEEE Transactions on Intelligent Transportation Systems, Vol. 10, No. 4, 2009
Mask R-CNN, Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick, Facebook AI Research (FAIR)

One seasoned expert spoke to a particular detail in the final question.

  The thought process experts use to choose a model has no equivalent point value in StackExchange reputation.  As experts retire, some may be willing to sell the professional heuristics they acquired over decades of research and development work if they haven't done so already to a book publisher.

Nonetheless, the above links are a good furtherance of the thought process begun in this question along a direction of continued effective thought.  The following steps provide a road map for how to develop approaches in general.

Read the academic materials until you understand.
Determine your selection criteria.
Test the options against that criteria.
Pick the winner.
If something blocks progress with that one, pick the runner up.


How to input dataset with multi-value properties? I'm trying to learn to use AI, and so I've followed some basic tutorials like training an MLP to predict the price of a car given properties like its age and manufacturer. Now I want to see if I can do it myself, so I thought it'd be fun to predict what score I would give a movie given some data scraped off IMDB.
I immediately got stuck, because how do you deal with the cast? A single property with multiple values, where a particular actor may impact the final score (or a combination of actors - that's for the neurons to suss out).
I haven't found a way to do this when googling, but it may just be that I'm unfamiliar with the terminology. Or have I accidentally chosen a really difficult problem?
Note that I'm completely new to all of this, so if you have suggestions, please try to put it as simply as possible.

You could use scikit-learn's MultiLabelBinarizer. It's essentially the multi-label equivalent of one-hot encoding. For each movie, create a vector of zeros, where each zero is associated with a particular actor. If an actor is in that movie, change their zero to a one. In the context of a neural network, think of it as each actor having their own input neuron, which will fire only if they are in that movie.
The caveat is that to represent all actors, you'd need a rediculously long vector. In such cases, it's often sufficient to only look at the most common, say, 100 and ignore the rest. It intuitively makes sense that having a big actor who's done a lot of movies says more about the quality of a movie than whoever's playing Unimportant Bystander #3. This is actually how natural language processing represents words in the English language - take the top n and ignore the rest.


How do we express $q_\pi(s,a)$ as a function of $p(s',r|s,a)$ and $v_\pi(s)$? The task (exercise 3.13 in the RL book by Sutton and Barto) is to express $q_\pi(s,a)$ as a function of $p(s',r|s,a)$ and $v_\pi(s)$.

$q_\pi(s,a)$ is the action-value function, that states how good it is to be at some state $s$ in the Markov Decision Process (MDP), if at that state, we choose an action $a$, and after that action, the policy $\pi(s,a)$ determines future actions.

Say that we are at some state $s$, and we choose an action $a$. The probability of landing at some other state $s'$ is determined by $p(s',r|s,a)$. Each new state $s'$ then has a state-value function that determines how good is it to be at $s'$ if all future actions are given by the policy $\pi(s',a)$, therefore:

$$q_\pi(s,a) = \sum_{s' \in S} p(s',r|s,a) v_\pi(s')$$

Is this correct?

Not quite. You are missing the reward at time step $t+1$.

The definition you are looking for is (leaving out the $\pi$ subscripts for ease of notation)

$$q(s,a) = \mathbb{E}[R_{t+1} + \gamma v(s') | S_t=s,A_t=a] = \sum_{r,s'}(r + v(s'))p(s',r|s,a)\;.$$

Because $q(s,a)$ relates to expected returns at time $t$, and returns are defined as $G_t = \sum_{b = 0}^\infty \gamma ^b R_{t+b+1}$, thus $R_{t+1}$ is also a random variable at time $t$ that we need to take expectation with respect to, not just the state that we transition into.


How to decide whether a problem needs to be solved algorithmically or with machine learning techniques? There are problems (e.g. this one or this other one) that could potentially be solved easily using traditional algorithmic techniques. I think that training a neural network (or any other machine learning model) for such sorts of problems will be more time consuming, resource-intensive, and pointless.

If I want to solve a problem, how to decide whether it is better to solve algorithmically or by using NN/ML techniques? What are the pros and cons? How can this be done in a systematic way? And if I have to answer someone why I chose a particular domain, how should I answer?

Example problems are appreciated.

There are two different problems described in the linked question and your question: optimization and learning.

Optimization

If you are asking about optimization (the second linked question: Search minimum value with learning machine algorithm) you can have 3 different approaches:

analytical approach
numerial methods
metaheuristics

As you suggest, it is usually better to try them from the first to the last one. It is common that the first approach is unfeasible for optimizing for target function, but very often you can use either mathematical optimization for some specific classes of problems (e.g. linear/quadratic programming) or iterative methods (e.g. conjugate gradient method). Only after considering this approaches it makes sense for the third class of approaches, genetic algorithms being a notable example, which is often classified as an AI approach.

Learning

If you are asking about learning, then the first linked question (Ideas on how to make a neural net learn how to split sequence into sub sequences) seems to be intended as an example. However it doesn't make clear what the problem is, as the target function seems to be obvious, so no learning is needed.

In this case it also makes sense to first try to pin down the problem mathematically and resort to machine learning if it is impossible and if you have the data (input/output examples).


Number Series Continuation? I am new to AI.
I have a series of numbers ranging from x to y and I have a lot of data to train with
What I am trying to do is, let's say from 0 to 1, I train it with data calculated over time and predict what may happen next, training it with my data and then feeding it the last few days and continue the pattern.
I have been thinking about using char-rnn, but from what i understand the data exported is arbitrary and not a continuation of a series. I oftentimes see videos on youtube "AI continues this song" so I'm wondering which I can use and where I can get started to do this myself.
Thank you and have a nice day ☺

I don't actually understand your question, but if your data is completely arbitrary, there is nothing to predict, it have no patterns to recognize or something like that.
But if you say that you are working with time-series data and it have some patterns, then you could start by trying to implement just the forward propagation of a simple RNN. I am really new as well at AI and the first RNN I code was a Elman RNN, and they were very simple equations to implement. I recommend you trying that, and then implement the backpropagation to that RNN.
Something that really helped me starting was searching simple github scrips for RNNs (around 100 lines of code) so that you can see their architecture.
About the example you give, the one about songs, they are actually really predictable, because you have the rime, rithm and tempo, and also they use to repeat the same secuence of notes every stanza.
Hope it helps :)


How to express a fully connected neural network succintly using linear algebra? I'm currently reading the paper Federated Learning with Matched Averaging (2020), where the authors claim:

A basic fully connected (FC) NN can be formulated as: $\hat{y} = \sigma(xW_1)W_2$ [...]
Expanding the preceding expression
$\hat{y} = \sum_{i=1}^{L} W_{2, i \cdot } \sigma(\langle x, W_{1,\cdot i} \rangle))$, where $ i\cdot$ and $\cdot i$ denote the ith row and column correspondingly and $L$ is the number of hidden units.

I'm having a hard time wrapping my head around how it can be boiled down to this. Is this rigorous? Specifically, what is meant by the ith row and column? Is this formula for only one layer or does it work with multiple layers?
Any clarification would be helpful.

The equation $$\hat{y} = \sigma(xW_\color{green}{1})W_\color{blue}{2} \tag{1}\label{1}$$ is the equation of the forward pass of a single-hidden layer fully connected and feedforward neural network, i.e. a neural network with 3 layers, 1 input layer, 1 hidden layer, and 1 output layer, where

the input layer is connected to the hidden layer (all scalar inputs are connected to every neuron in the hidden layer)
the hidden layer is connected to the output layer (all neurons in the hidden layer are connected to all neurons in the output layer

As an example, suppose that we have $N$ real-valued features, there are $L$ hidden units (or neurons) and $M$ output units, then the elements (feature vector and parameters) of equation \ref{1} would have the following shape

$x \in \mathbb{R}^{1 \times N}$
$W_\color{green}{1} \in \mathbb{R}^{N \times L}$
$W_\color{blue}{2} \in \mathbb{R}^{L \times M}$

$\sigma$ is an activation function that is applicable to all elements of the matrix separately (i.e. component-wise). So, $\hat{y} \in \mathbb{R}^{1 \times M}$.
The equation
$$
\hat{y} = \sum_{\color{red}{i} = 1}^{L} W_{\color{blue}{2}, \color{red}{i} \cdot} \sigma(\langle x, W_{\color{green}{1}, \cdot \color{red}{i}} \rangle )\label{2}\tag{2}
$$
is another way of writing equation \ref{1}.
Before going to the explanation, let's try to understand equation \ref{2} and its components.

$W_{\color{green}{1}, \cdot \color{red}{i}} \in \mathbb{R}^N$ is the $\color{red}{i}$th column of the matrix that connects the inputs to the hidden neurons, so it is a vector of $N$ elements (note that we sum over the number of hidden neurons, $L$).

Similarly, $ W_{\color{blue}{2}, \color{red}{i} \cdot} \in \mathbb{R}^M$ is also a vector, but, in this case, it is a row of the matrix $ W_{\color{blue}{2}}$ (rather than a column: why? because we use $\color{red}{i} \cdot$ instead of $\cdot \color{red}{i}$, which refers to the column).

So, $\langle x, W_{\color{green}{1}, \cdot \color{red}{i}} \rangle$ is the dot (or scalar) product between the feature vector $x$ and the $\color{red}{i}$th column of the matrix that connects the inputs to the hidden neurons, so it's a number (or scalar). Note that both $x$ and $W_{\color{green}{1}, \cdot \color{red}{i}}$ have $N$ elements, so the dot product is well-defined in this case.

$ W_{\color{blue}{2}, \color{red}{i}} \sigma(\langle x, W_{\color{green}{1}, \color{red}{i}} \rangle))$ is the product between a vector of shape $M$ and a number $\sigma(\langle x, W_{\color{green}{1}, \cdot \color{red}{i}} \rangle )$. This is also well-defined. You can multiply a real-number with a vector, it's like multiplying the real-number with each element of the vector.

$\sum_{\color{red}{i} = 1}^{L} W_{\color{blue}{2}, \color{red}{i} \cdot} \sigma(\langle x, W_{\color{green}{1}, \cdot \color{red}{i}} \rangle )$ is thus the sum of $L$ vectors of size $M$, which makes $\hat{y}$ also have size $M$, as in equation \ref{1}.

Now, the question is: is equation \ref{2} really equivalent to equation \ref{1}? This is still not easy to see because $xW_\color{green}{1}$ is a vector of shape $L$, but, in equation \ref{2}, we do not have any vector of shape $L$, but we have vectors of shape $N$ and $M$ (and the vectors of shape $M$ are summed $L$ times). First, note that $\sigma(xW_\color{green}{1}) = h\in \mathbb{R}^L$, so $hW_\color{blue}{2}$ are $M$ dot products collected in a vector (i.e. $\hat{y} \in \mathbb{R}^M$), where the $j$th element of $\hat{y}$ was computed as a summation of $L$ elements (a dot product of two vectors is the element-wise multiplication of the elements of the vectors followed by the summation of these multiplications). Ha, still not clear!
The easiest way (for me) to see that they are equivalent is to think that $\sigma(xW_\color{green}{1})$ is a vector of $L$ elements $\sigma(xW_\color{green}{1}) = \ell = [l_1, l_2, \dots, l_L]$. Then you know that to multiply this vector with $\ell$ (from the left), you actually perform a dot product between $\ell$ and each column $W_\color{blue}{2}$. A dot product is essentially a sum, and that's why we sum in equation \ref{2}. So, essentially, in equation \ref{2}, we first multiple $l_1$ with the first row of $W_\color{blue}{2}$ (i.e. by all elements of the first row). Then we multiply $l_2$ by the second row of $W_\color{blue}{2}$. We do this for all $L$ rows, then we sum the rows (to conclude the dot product). So, you can think of equation 2 as first perform all multiplications, then summing, rather than dot product-by-dot product.
So, in my head, I have the following picture. To simplify the notation, let $A$ denote $W_\color{blue}{2}$, so $A_{ij}$ is the element at the $i$th row and $j$th column of matrix $W_\color{blue}{2}$. So, we have the following initial matrix
$$
A =
\begin{bmatrix}
A_{11} & A_{12} & A_{13} & \dots & A_{1M} \\
A_{21} & A_{22} & A_{23} & \dots & A_{2M} \\
\vdots & \vdots & \vdots & \dots & \vdots \\
A_{L1} & A_{L2} & A_{L3} & \dots & A_{LM} \\
\end{bmatrix}
=
\begin{bmatrix}
W_{\color{blue}{2}, \color{red}{1} \cdot}  \\
W_{\color{blue}{2}, \color{red}{2}  \cdot} \\
\vdots  \\
W_\color{blue}{2, \color{red}{L}  \cdot} \\
\end{bmatrix}
$$
Then, in the first iteration of equation \ref{2}, we do the following
$$
\begin{bmatrix}
l_1 A_{11} & l_1 A_{12} & l_1 A_{13} & \dots & l_1 A_{1M} \\
A_{21} & A_{22} & A_{23} & \dots & A_{2M} \\
\vdots & \vdots & \vdots & \dots & \vdots \\
A_{L1} & A_{L2} & A_{L3} & \dots & A_{LM} \\
\end{bmatrix}
$$
In the second, we do the following
$$
\begin{bmatrix}
l_1 A_{11} & l_1 A_{12} & l_1 A_{13} & \dots & l_1 A_{1M} \\
l_2 A_{21} & l_2 A_{22} & l_2 A_{23} & \dots & l_2 A_{2M} \\
\vdots & \vdots & \vdots & \dots & \vdots \\
A_{L1} & A_{L2} & A_{L3} & \dots & A_{LM} \\
\end{bmatrix}
$$
Until we have
$$
\begin{bmatrix}
l_1 A_{11} & l_1 A_{12} & l_1 A_{13} & \dots & l_1 A_{1M} \\
l_2 A_{21} & l_2 A_{22} & l_2 A_{23} & \dots & l_2 A_{2M} \\
\vdots & \vdots & \vdots & \dots & \vdots \\
l_L A_{L1} & l_L A_{L2} & l_L A_{L3} & \dots & l_L A_{LM} \\
\end{bmatrix}
$$
Then we do a reduce sum across the rows to end the dot product (i.e. for each column we sum the elements in the rows). This is exactly equivalent to first performing the dot product between $\ell$ and the first column of $W_\color{blue}{2}$, then the second column, and so on.


When exactly am I overfitting -- contradicting metrics I am training an object detection machine learning pipeline. Among the many metrics provided out of the box by tensorflow object detection API, I look at total_loss and DetectionBoxes_Precision/mAP@.75IOU:

Here the x-axis is the number of steps i.e. model experience. The orange line is for the training data and the blue line is for the validation data.
From the loss graph I would conclude, that at approx 2k steps overfitting starts, so using the model at approx 2k steps would be the best choice. But looking at the precision graph, training e.g. until 24k steps would be a much better model.
Which one is the best model?
Here, loss and the precision metric where picked just for illustrating the dilemma, there are many more metrics available, leading to multiple conclusions about when overfitting actually starts.

Generally, a model is considered to be overfitted when there's a huge gap between training and test/validation performances.
So during the training, you monitor the loss on validation data, and training data, and stop training if validation loss stagnates/increases given the training loss keeps decreasing.
In your scenario, I'm not sure what the total loss metric corresponds to. As I said, you've to measure loss on a held-out data other than training data to detect, and prevent overfitting.


Alphazero policy head loss not decreasing I am now working on training an alphazero player for a board game. The implementation of board game is mine, MCTS for alphazero was taken elsewhere. Due to complexity of the game, it takes a much longer time to self-play than to train.

As you know, alphazero has 2 heads: value and policy. In my loss logging I see that with time, the value loss is decreasing pretty significantly. However, the policy loss only demonstrates fluctuation around its initial values.

Maybe someone here has run into similar problems? I would like to know if its my implementation problem (but then the value loss is decreasing) or just a matter of not enough data.

Also, perhaps importantly, the game has ~17k theoretically possible moves, but only 80 at max are legal at any single state (think chess - a lot of possibles but very few are actually legal at any given time). Also, if MCTS has 20 simulations, then the improved probabilities vector (against which we train our policy loss) will have at most 20 non-zero entries. My idea was that it might be hard for the network to learn such sparse vectors.

Thank you for any ideas!

The loss of the policy head here is really quite different from losses in, for instance, more "conventional" Supervised Learning approaches (where we typically expect/hope to see a relatively steady decrease in loss function).

In this AlphaZero setup, the target that we're updating the policy head towards is itself changing during the training process. When we improve our policy, we expect the MCTS "expert" to also be improved, which may lead to a different distribution of MCTS visit counts, which in turn may lead to a different update target for the policy head from previous update targets. So it's perfectly fine if our "loss" increases sometimes, we may still actually be performing better. The loss is useful for the computation of our gradient, but otherwise it doesn't have much use -- it certainly cannot be used as an accurate indicator of performance / learning progress.

  but only 80 at max are legal at any single state (think chess - a lot of possibles but very few are actually legal at any given time). Also, if MCTS has 20 simulations, then the improved probabilities vector (against which we train our policy loss) will have at most 20 non-zero entries.

This can be a problem yes. The fact that the majority of moves are not legal at any point in time is not a problem, but if you only have 20 MCTS simulations for a branching factor of 80... that's certainly a problem. The easiest fix would be to simply keep MCTS running for longer, but obviously that's going to take more computation time. If you cannot afford to do this for every turn of self-play, you could try:

using only a low MCTS iteration count for some moves, not adding these distributions to the training data for the policy head
using a larger MCTS iteration count for some other moves, and only using the distributions of these moves as training data for the policy head

This idea is described in more detail in Subsection 6.1 of Accelerating Self-Play Learning in Go.


Where or for what could genetic algorithms be used in the context of project management? Where or for what could genetic algorithms (GA) be used in the context of project management (PM)? I thought about task dispatching, but I'm looking for other potential uses of GAs in the context of PM.

GA is well suited to optimize a non-linear fitness function with a lot of variables. Each vector that is a possible solution is evaluated with the fitness function that we want to optimize.
So, GA is well suited to optimize schedules for a lot of people, optimizing resources, etc.
Your mission is to define all variables that you can move-tune and that affect your objectives, and assign an estimated weight on every variable.
For example, create a total project cost as the fitness function, to minimize. One variable will be the number of programmers, multiplied by cost and total months. If you don't want delays over 3 months, add a correction factor: +10000*max(0, months-3), penalizing the fitness function's results where months > 3, etc.
When you apply your GA algorithm to the fitness function, you will find some minimum for your function, if any exists, because the solution may not exist if you added too many constraints.


Why do we use $X_{I_t,t}$ and $v_{I_t}$ to denote the reward received and the at time step $t$ and the distribution of the chosen arm $I_t$? I'm doing some introductory research on classical (stochastic) MABs. However, I'm a little confused about the common notation (e.g. in the popular paper of Auer (2002) or Bubeck and Cesa-Bianchi (2012)).
As in the latter study, let us consider an MAB with a finite number of arms $i\in\{1,...,K\}$, where an agent choses at every timestep $t=1,...,n$ an arm $I_t$ which generates a reward $X_{I_t,t}$ according to a distribition $v_{I_t}$.
In my understanding, each arm has an inherent distribution, which is unknown to the agent. Therefore, I'm wondering why the notation $v_{I_t}$ is used instead of simply using $v_{i}$? Isn't the distribution independent of the time the arm $i$ was chosen?
Furthermore, I ask myself: Why not simply use $X_i$ instead of $X_{I_t,t}$ (in terms of rewards). Is it because the chosen arm at step $t$ (namely $I_t$) is a random variable and $X$ depends on it? If I am right, why is $t$ used twice in the index (namely $I_t,t$)?
Shouldn't $X_{I_t}$ be sufficient, since $X_{I_t,m}$ and $X_{I_t,n}$ are drawn from the same distribution?

Isn't the distribution independent of the time the arm $i$ was chosen?

Yes, but you don't know which arm was chosen at time $t$, that is what $I_t$ represents. $v_i$ would represent the $i$th arms distribution, whereas you want the distribution of the arm that was chosen at time $t$, which is $v_{I_t}$.
$X_{I_t,t}$ is used to represent the arm you chose at time $t$ and the time you chose it. Imagine if we chose the following arms at respective time steps {1,2,5,2,4}, then the reward you get at time 3 (assuming in my example time started at 1) would be $X_{5,3}$. You need this notation because every time you pull arm $i$ you get a different reward, because the reward is a random variable (unless the assumption was a deterministic reward but that would not be interesting).
It has been a while since I read the papers but I assume that the distributions of the arms are stationary, however this notation is more general and would allow for non-stationary distributions.


How to handle images that don’t pertain to image classifier at all? I am trying to create a CNN model that classifies if a person is wearing a seatbelt or not to verify they drive safely. I know to get images of people wearing seatbelts and people not wearing seatbelts, but I have a problem.
What if the person doesn’t submit a picture of them in a car at all? How do I construct the rest of the dataset to determine if that picture is an actual picture of a person wearing a seatbelt?
Do I insert completely random pictures in a different category? Do I classify images that don’t have a high confidence score as "wrong" images? Or leave it?

Yes, a category "no person" or "random image" would make sense. Binary classification is only helpful if you know that your input always belongs to one or the other category, for example by pre-filtering the inputs.


ValueError: Error when checking target: expected dense_3 to have shape (1,) but got array with shape (2,) I am trying to build a CNN model on Keras. The data has a dimension of 921 rows × 10000 columns.

Here is the code:

import numpy as np
import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Activation
from keras.optimizers import SGD

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(merge.iloc[:,166:10166], merge[['Result_cat','Result_cat1']].values, test_size=0.2)

model = Sequential()

model.add(Dense(50, input_shape=(10000,)))
model.add(Activation("relu"))
model.add(Dense(50, activation='softmax'))
model.add(Dense(1, activation='linear'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.SGD(lr=0.01),
              metrics=['accuracy'])

model.fit(X_train, y_train, batch_size=100, epochs=10)
score = model.evaluate(X_test, y_test, batch_size=32)

Then I encountered error

  ValueError: Error when checking target: expected dense_3 to have shape (1,) but got array with shape (2,)

I am new to Keras and CNN. Can someone please explain to me what this means and how I can fix this? Thanks.

Problem is in the output layer and you are using categorical_crossentropy for a loss function. Quoting Keras documentation:

  Note: when using the categorical_crossentropy loss, your targets
  should be in categorical format (e.g. if you have 10 classes, the
  target for each sample should be a 10-dimensional vector that is
  all-zeros except for a 1 at the index corresponding to the class of
  the sample). In order to convert integer targets into categorical
  targets, you can use the Keras utility to_categorical:

I would check if the target values are defined consistently. They seem to be defined for sparse_categorical_crossentropy, which requires a dense layer where number of neurons = number of classes of the problem. If they are defined as integers - as required for sparse_categorical_crossentropy, you can convert them to categorical formatting using to_categorical function, also defined in the documentation page linked above; or you can keep them as integers, and change the number of neurons in the final layer to 2, and loss function to sparse_categorical_crossentropy.


What are the main algorithms used in computer vision? Nowadays, CV has really achieved great performance in many different areas. However, it is not clear what a CV algorithm is.
What are some examples of CV algorithms that are commonly used nowadays and have achieved state-of-the-art performance?

There are many computer vision (CV) algorithms and models that are used for different purposes. So, of course, I cannot list all of them, but I can enumerate some of them based on my experience and knowledge. Of course, this answer will only give you a flavor of the type of algorithm or model that you will find while solving CV tasks.
For example, there are algorithms that are used to extract keypoints and descriptors (which are often collectively called features, although the descriptor is the actual feature vector and the keypoint is the actual feature, and in deep learning this distinction between keypoints and descriptors does not even exist, AFAIK) from images, i.e. feature extraction algorithms, such as SIFT, BRISK, FREAK, SURF or ORB. There are also edge and corner detectors. For example, the Harris corner detector is a very famous corner detector.
Nowadays, convolutional neural networks (CNNs) have basically supplanted all these algorithms in many cases, especially when enough data is available. Rather than extracting the typical features from an image (such as corners), CNNs extract features that are most useful to solve the task that you want to solve by taking into account the information in the training data (which probably includes corners too!). Hence CNNs are often called data-driven feature extractors. There are different types of CNNs. For example, CNNs that were designed for semantic segmentation (which is a CV task/problem), such as the u-net, or CNNs that were designed for instance segmentation, such as mask R-CNN.
There are also algorithms that can be used to normalize features, such as the bag-of-features algorithm, which can be used to create fixed-size feature vectors. This can be particularly useful for tasks like content-based image retrieval.
There are many other algorithms that could be considered CV algorithms or are used to solve CV tasks. For example, RanSaC, which is a very general algorithm to fit models to data in the presence of outliers, can be used to fit homographies (matrices that are generally used to transform planes to other planes) that transform pixels of one image to another coordinate system of another image. This can be useful for the purpose of template matching (which is another CV task), where you want to find a template image in another target image. This is very similar to object detection.
There are also many image processing algorithms and techniques that are heavily used in computer vision. For example, all the filters (such as Gaussian, median, bilateral, non-local means, etc.) that can be used to smooth, blur or de-noise images. Nowadays, some deep learning techniques have also replaced some of these filters and image processing techniques, such as de-noising auto-encoders.
All these algorithms and models have something in common: they are used to process images and/or get low- or high-level information from images. Most of them are typically used to extract features (i.e. regions of the images that are relevant in some way) from images, so that they can later be used to train a classifier or regressor to perform some kind of task (e.g. find and distinguish the objects, such people, cars, dogs, etc. in an image). The classifiers/regressors are typically machine learning (ML) models, such as SVMs or fully-connected neural networks, but there's a high degree of overlap between CV and ML because some ML tools are used to solve CV tasks (e.g. image classification).


How can artificial intelligence predict the next possible moves of the player? When you play video games, sometimes there is an AI that attempts to predict what are you going to do.

For example, in the Candy Crush game, if you finish the level and you still have moves remaining, you get to see fishes or other powers destroying the other candies, but, instead of watching 10 minutes of your combos without moving at all after accomplishing a level, like this Longest video game combo ever probably, it tells an alert that says tap to skip, so basically the AI is predicting all the possible combos that will keep proceeding automatically and calculating every automatic move.

How can artificial intelligence predict such a thing?

Artificial Intelligence can predict such a thing because before they release the game, the train the BOT or AI to play the game million times so they have a model or BOT that can predict every next move or combo that they can do or predict all moves that can finish the game.
for example snake game. what they do to predict moves is train the model or bot to play the game when the snake perform some action . the snake got reward which can be positive or negative reward. the goal of the snake is to learn what action that maximize the reward, given every possible state. States are the observations that the agent receives at each iteration from the environment

this is the link that can give you the detail : https://towardsdatascience.com/how-to-teach-an-ai-to-play-games-deep-reinforcement-learning-28f9b920440a


How should one standardize input when transfer learning Assume one is using transfer learning via a model which was trained on imagenet.

Assume that the pre-processing which was used to achieve the pretrained model contained z-score standardization using some mean and std which was calculated on the training data.
Should one apply the same transformation on their new data?
Should they apply z-score standardization using a mean and std of their own training data?
Assume that the pre-processing now did not contain any standardization.
Should one apply no standardization on their new data as well?
Or should one apply z-score standardization, using the mean and std of their new data, and expect better results?

For example i've seen that the Inception V3 model which was trained by Keras did not use any standardization, and i'm wondering if using z-score standardization on my new data could yield better results.

You should use the same transformation (i.e same mean and std in case of z-score normalization) on the new data while using any pretrained model.
If the pretrained model was trained without any normalization then it does not matter if you use normalization or not as long as range of input data is fixed (0-255 for the 8-bit input images). You can use pretrained models with batch normalization which takes care of internal covariance shift and thus there is no need to normalize data.


How to find proper parameter settings for a given optimization algorithm? Is there any methodology to find proper parameter settings for a given meta-heuristic algorithm, e.g. the firefly algorithm or the cuckoo search? Is this an open issue in optimization? Is extensive experimentation, measurements and intuition the only way to figure out which are the best settings?

How to find the best configuration for an algorithm is an open research question in AI. The topic in general is known as `hyper-parameter optimization' and there are a range of possible methods:

One of the most popular is IRace, but other possibilities include:

Spearmint: uses wrappers in Matlab or Python. It uses MongoDb,
and Bayesian optimisation algorithms.
SMAC requires a python wrapper for the algorithm to be optimized
and has a command line interface.
Hyperopt: a Python library which uses Random Search and Tree of
Parzen Estimators.

This paper argues that Spearmint performs the best, compared with SMAC and Hyperopt, but with significantly longer running times in some cases.


Which neural network has capabilities of sorting input? I believe normally you can use genetic programming for sorting, however I'd like to check whether it's possible using ANN.

Given the unsorted text data from input, which neural network is suitable for doing sorting tasks?

Even a simple multilayer perceptron can sort input data to some extent, as you can see here and here.

However, neural networks for sequential data seem more appropriate, as they can handle sequences of variable lengths. It has been done with an LSTM (Long Short-Term Memory), LSTM+HAM (Hierarchical Attentive Memory) and an NTM (Neural Turing Machine).


What is the solution to the exercise 3.11 in the RL book by Sutton and Barto I'm trying to solve the exercise 3.11 from the second-edition from the Sutton and Barto's book

  Exercise 3.11 If the current state is $S_t$ , and actions are selected
  according to a stochastic policy $\pi$, then what is the expectation of $R_{t+1}$
  in terms of $\pi$ and the four-argument function $p$ (3.2)?

Here's my attempt.

For each state $s$, the expected immediate reward when taking action $a$ is given in terms of $p$ by eq 3.5 in the book:

$r(s,a) = \sum_{r \in R} r \, \sum_{s'\in S} p(s',r| s,a) = E[R_t | S_{t-1} = s, A_{t-1} = a]$ (1)

The policy $\pi(a | s)$ on the other hand gives the probability of taking action $a$ given state $s$.

Is it possible to express the expectation of the immediate reward over all actions $A$ from the state $s$ using (1) as

$$E[R_t | S_{t-1} = s, A] = \sum_{a \in A} \pi(a|s) r(a,s) \ \ \ \ \ \ \ \  (2) ?$$

If this is valid, is this also valid in the next time step

$$E[R_{t+1} | S_{t} = s, A] = \sum_{a \in A} \pi(a|s) r(a,s) \ \ \ \ \ \ \ \ (3) ?$$

If (2) and (3) are OK, then

$$E[R_{t+1} | S_{t} = s, A] = \sum_{a \in A} \pi(a|s) \sum_{r \in R} r \, \sum_{s'\in S} p(s',r| s,a)$$

First note that $\mathbb{E}[R_{t+1} |S_t=s] = \sum_{s',r}rm(s',r|s)$ where $m(\cdot)$ is the mass function for the joint distribution of $S_{t+1},R_{t+1}$.

If you are currently in state $S_t$ and we condition on taking action $a$ then the expected reward at time $t+1$ is given as follows:

\begin{align} \mathbb{E}[R_{t+1} | S_t = s, A_t=a] & = \sum_{s',r}rp(s'r|s,a)\;.
\end{align}

However, action $A_t$ is taken according to some stochastic policy $\pi$ so we need to marginalise this out of our expectation by using the tower law - i.e. we take

$$\mathbb{E}_{A_t \sim \pi}[\mathbb{E}[R_{t+1} | S_t = s, A_t=a]|S_t = s] = \sum_a \pi(a|s)\sum_{s',r}rp(s'r|s,a) = \mathbb{E}[R_{t+1} | S_t = s]\;.$$

To see why this holds we can re-write using some arbitrary mass functions $f(\cdot),h(\cdot),g(\cdot),m(\cdot)$ as

$$\pi(a|s)p(s'r|s,a) = \frac{f(a,s)}{g(s)} \times \frac{h(s',r,a,s)}{f(a,s)} = m(s',r,a|s)\;,$$
so when we would end up with (after re-arranging the summations)

$$\sum_{s',r}r \sum_{a}m(s',r,a|s) = \sum_{s',r}r m(s',r|s) = \mathbb{E}[R_{t+1}|S_t = s]\;;$$
as required.

NB: what you have done is mostly correct except be careful when going from (2) to (3). They are exactly the same equation except for the time stamps, which means you would have to change the time stamps in your $r(s,a)$. Note that when you are at time step $t$ you take your action $A_t$ from your current $S_t$ to transition into state $S_{t+1}$ and then receive reward $R_{t+1}$ (and the next state).


Understand the DDPG algorithm in Keras I'm trying to understand the DDPG algorithm using Keras
I found the site and started analyzing the code, I can't understand 2 things.
The algorithm used to write the code presented on the page

In the algorithm image, updating the critic's network does not require gradient
But the gradient is implemented in the code, why?
with tf.GradientTape() as tape:
    target_actions = target_actor(next_state_batch)
    y = reward_batch + gamma * target_critic([next_state_batch, target_actions])
    critic_value = critic_model([state_batch, action_batch])
    critic_loss = tf.math.reduce_mean(tf.math.square(y - critic_value))

critic_grad = tape.gradient(critic_loss, critic_model.trainable_variables)
critic_optimizer.apply_gradients(zip(critic_grad, critic_model.trainable_variables))

The second question is why in the photo of the algorithm when calculating the actor's policy gradient are 2 gradients multiplied by themselves and in the code only one gradient is calculated for the critic's network and it's not multiplied by the second gradient?
with tf.GradientTape() as tape:
    actions = actor_model(state_batch)
    critic_value = critic_model([state_batch, actions])
    # Used `-value` as we want to maximize the value given
    # by the critic for our actions
    actor_loss = -tf.math.reduce_mean(critic_value)

actor_grad = tape.gradient(actor_loss, actor_model.trainable_variables)
actor_optimizer.apply_gradients(zip(actor_grad, actor_model.trainable_variables))

The answer to your first question is because the line 'update the critic by minimising the loss $L = \frac{1}{N} \sum_i \left( y_i - Q(s_i, a_i |\theta^Q)\right)^2$ is implying that you will do this by using a gradient, i.e. you calculate the gradient of the loss wrt the parameters and perform a gradient descent step.
For the second question, I am not 100% sure because I don't use tensorflow but I would assume that in the automatic differentiation used it will do the multiplication automatically behind the scenes. In PyTorch I know that it automatically calculates the chain rule when you take the derivative, so it is likely to be a reason similar to this.


Extract product information from email receipt HTML I am trying to extract product information from email receipts HTML.  Most services I have found focus on OCR from paper receipts or PDFs.  I would imagine that extraction of product information would be easier from structured HTML.  What type of AI approach would be used to support this?

It depends on the data. If it is structured like form data, then you might not need AI at all — simple regular expression patterns might be fine. This would apply for example to address data. If you have the word street followed by a colon, followed by some text, it seems fairly obvious that this is the name of a street, and possibly also a house number.

If, however, you have free text, eg the answer to "describe any medical conditions you have", or "which companies have you worked at before?", then you might need to look at named entity extraction (NER) to identify names of medical conditions or companies.

So, some Natural Language Processing (NLP) and information extraction might be required apart from simple pattern matching.


What's the function that SGD takes to calculate the gradient? I'm struggling to fully understand the stochastic gradient descent algorithm.
I know that gradient descent allows you to find the local minimum of a function. What I don't know is what exactly that function IS.
More specifically, the algorithm should work by initializing the network with random weights. Then, if I'm not mistaken, it forward-propagates $n$ times (where $n$ is the mini-batch size). At this point, I've no idea about what function should I search for, with over hundreds of neurons each having hundreds of parameters.

Welcome to AI Stack exchange!

You're right, as the network is initialised randomly, the resultant function is essentially impossible to get your head around. This is because most of the time the network has >4 dimensions (4 can be graphed with some effort and a lot of color), and as such is literally beyond human comprehension via graphing.

So what do we do? Well, conveniently, it is possible to find the gradient of segments of a function, without having to know the entirety of the function itself (it's worth noting that it actually is possible to find the resultant function and with a lot of effort find it's derivative. This proves to be much more work than it's worth though, as we don't need the general derivative that tells us what the gradient is for whatever input we give it, we only need the derivative at the specific input we just fed through the network).

This might be hard to understand, but if you're familiar with the chain rule, it might make a bit more sense. The chain rule essentially allows you to split a function into components, and find the gradient of those specific components. By combining all of that, you end up with some nice gradients at each weight/bias with respect to the loss function. Take the negative of the gradient, and you now have the change required to decrease the loss function.

This is obviously quite hard to understand without an example, so here's the best one I've ever found, that helped me very much.

Also, as a side note, the whole mini batch thing is used to minimise catastrophic forgetting (where the network begins to "unlearn" old inputs). To deal with minibatchs, what you do is take input individually, then find the gradient for all weights and bias' for that specific input and remember the changes you want to make. Do that for all inputs in the minibatch, then finally add all the changes together to get the resultant best change  for each weight and bias. Only then do you update the weights and bias'.

Let me know if you have any further questions


Can the addition of unnoticeable noise to images be used to create subliminals? I was reading this report: https://www.theverge.com/2017/4/12/15271874/ai-adversarial-images-fooling-attacks-artificial-intelligence

Researchers used noise to trick machine learning algorithms to misidentify or misclassify an image of a fish as a cat. I was wondering if something like that can be used to create subliminals.

What I mean by subliminals: United Nations has defined subliminal messages as perceiving messages without being aware of them, it is unconscious perception, or perception without awareness. Like you may be aware of a message but cannot consciously perceive that message in the form of text, etc.

All the reports about the noise trick said the noise was so transparent that humans couldn't detect it. This can be changed to make it noticeable unconsciously but unnoticeable at a conscious level so a human can register the subliminal but not be aware of it.

Is it possible to take an output from a hidden layer to construct such subliminal for humans, with trial and error one can find right combination? Can it be possible to come up with a pixel pattern or noise with ML which allows one to impose subliminals?

I would say no, at least not now. To achieve that you need to have a model that represents the structure of a human brain at a very detailed level, so that inherits the same flaws.


Why should the number of neurons in a hidden layer be a power of 2? I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster.
Is this a fact? If it is, why is this true? Does it have something to do with how the memory is laid down?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster.

I would quite like to see a reference to this suggestion, in case it has been misunderstood.
As far as I know, there is no such effect in normal neural networks. In convolutional neural networks, it might potentially be true in a minor way because some FFT approaches work better with $2^n$ items.

Is this a fact? If it is, why is this true? Does it have something to do with how the memory is laid down?

I would say that this is not a general fact. Instead, it seems like misunderstood advice to search some hyperparameters such as number of neurons in each layer, by increasing or decreasing by a factor of 2. Doing this and trying layer sizes of 32, 64, 128 etc should increase the speed of finding a good layer size compared to trying sizes 32, 33, 34 etc.
The main reason to pick powers of 2 is tradition in computer science. Provided there is no driver to pick other specific numbers, may as well pick a power of 2 . . . but equally you will see researchers picking multiples of 10, 100 or 1000 as "round numbers", for a similar reason.
One related factor: If a researcher presents a result for some new technique where the hidden layer sizes were tuned to e.g. 531, 779, 282 etc, then someone reviewing the work would ask the obvious question "Why?" - such numbers might imply the new technique is not generic or requires large amounts of hyperparameter tuning, neither of which would be seen as positive traits. Much better to be seen using an obvious "simple" number . . .


Should I compute the gradients with respect to the flatten layer in a convolutional neural network? I'm trying to create a convolutional neural network without frameworks (such as PyTorch, TensorFlow, Keras, and so on) with Python.

Here's a description of CNN taken from the Wikipedia article

  In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of deep neural networks, most commonly applied to analyzing visual imagery. They are also known as shift invariant or space invariant artificial neural networks (SIANN), based on their shared-weights architecture and translation invariance characteristics. They have applications in image and video recognition, recommender systems, image classification, medical image analysis, natural language processing, and financial time series.

A CNN has different types of layers, such as convolution, pooling (max or average), flatten and dense (or fully-connected) layers.

I have a few questions.

Should we compute gradients (such as $\frac{\partial L}{\partial A_i}$,$\frac{\partial L}{\partial Z_i}$,$\frac{\partial L}{\partial A_{i-1}}$ and so on) in flatten layer or not?
If no, then how should I compute $\frac{\partial L}{\partial A_i}$ and $\frac{\partial L}{\partial Z_i}$ of first layer of convolutional layer? With $\frac{\partial L}{[\frac{\partial g(A_i)}{\partial x}]}$ or with
$\frac{\partial L}{\partial dA_{i+2}}$(P.S. as you know iteration of BackPropagation is reverse, so I used i+n for denote the previous layer)?
Or can I compute derivatives in Flatten layer with
$$\frac{\partial J}{\partial A} = W_{i+1}^T Z_{i+1}$$(i+1 denotes prev.layer in BackProp)
$$\frac{\partial L}{\partial Z} = \frac{\partial L}{\partial A} *\frac{\partial g(A_i)}{\partial x} $$
and then reshape of Conv2D shape?

P.S. I found questions like mine (names are same), but there're not answer to my question as I asking about formula.

I have found that you should compute derivatives $\frac{\partial L}{\partial A}, \frac{\partial L}{\partial Z}$ in Flatten layer and then reshape Conv2D input shape.


How can we encourage data analysts and decision-makers to adopt AI? Nowadays, there is too much data for humans to work on alone, and it is very normal for data analysts to use AI techniques to treat and process these data so it can lead to a faster and more accurate result. But many data analysts and decision-makers still don't trust AI methods or techniques and are reluctant to use them. How can we encourage them to accept or prefer these AI solutions?

For example, if AU gives advice to solve a problem, then decision-makers must trust the results and data analysts must trust the mechanism, so that decision-makers can be confident in the AI work and also data analysts can concentrate their activities on added value.

How can we encourage data analysts and decision-makers to adopt AI?

Not all of the mistrust aimed at AI systems is unjustified, particularly when it comes to neural networks and other such systems that rely on large training data sets. There are a number of high profile cases, facial recognition being one that has often (understandably) received a lot of flak, where improperly configured training data has resulted in skewed and questionable results.

If you want to foster more trust in the systems, it will require better tools for analyzing how they are approaching problems and reaching decisions, as well as determining if there are holes in the training data. It will require a community working in the field that gives a lot more thought and care to how they approach their training data and what unintended biases they may be introducing by forgetting something than has often been displayed presently.

Of course, some people are just distrustful of new technology, but I think it's more interesting to address the more legitimate concerns.


Analyzing vibration using machine learning I would like a few suggestions on an idea that I have -

I am trying to make a musical instrument (percussion), whilst just having a PVC disc. I am hitting the disc in a variety of styles (in order to produce a variety of sounds correspondingly), just like the way the actual percussion instrument is hit. I am converting the mechanical vibrations on the PVC disc to an electrical signal using a transducer, performing an FFT analysis of the different strokes, and trying to identify the stroke which is hit. Using this technique, I could get an accuracy of only 80 percent. I would like it to be extremely accurate ( more than 95 percent recognition). I was using only frequency as the parameter used to distinguish the sounds.

Now, I am thinking that if I could use other parameters too in order to identify the stroke, I might be able to get the required accuracy. I am thinking of resorting to Machine Learning for this. I am kind of new to this and would like to know what I might need to know before I proceed with this idea.

Any help would be greatly appreciated.

If you want to use machine learning for such a project, you can use vibrations data directly, and treat the problem as a regular audio classification problem.

A simple approach would be to use a Neural Network with Convolutions. This would take care of features extraction for you. And maybe follow these by dense layers at the end.

Given that, it would be easier to make suggestions if you posted samples of your data.

Edit: Also, keep in mind that machine learning usually requires large datasets so if you are collecting data all by yourself in the way you describe, you might not have enough sample to run a good model. In such a case when number of samples is limited one can use transfer learning - using a pre-trained model - but I am not aware of any such pre-trained models for wave data.


Is an algorithm that is no longer actively learning an AI? This question assumes a definition of AI based on machine learning, and was inspired by this fun Technology Review post:

SOURCE: Is this AI? We drew you a flowchart to work it out (Karen Hao, MIT Technology Review)

As the definition of artificial intelligence has been a continual subject of discussion on this stack, I wanted to bring it to the community for perspectives.

The formal question here is:

Is an algorithm that is no longer actively learning an AI?

Specifically, can applied algorithms that are not actively learning be said to reason?

In my view learning is a useful aspect of an AI system, which makes it more resilient towards a changing environment, but it is not a requirement. Early AI systems (Expert systems, theorem solvers) would usually not use learning algorithms, but had a fixed knowledge base.

It also works a lot better with numerical models than with symbolic processing; the proliferation of training data which boosts the most recent AI wave (which is based more on statistical and numerical methods) has made it somewhat necessary for modern systems to have a learning/training component.

However, one needs to distinguish between the initial training, which leads to a production system which then becomes fixed (and will not learn from new input) and those which have a learning component built in as an intrinsic part and will continue to adapt throughout their lifetime.

A cautionary example is Microsoft's "Tay" chatbot. This was still actively learning from conversations and was quickly abused by subjecting it to undesirable input. Switching off the learning component would have been prudent in this case, but it would not have taken the AI out of it.

So, to answer the question: learning is not a required part of an AI system once it is operational. While modern systems generally have some element of learning to build their initial models, older systems were simply hand-coded without any automated learning at all. But one would still regard them as AI system.


How many ways are there to perform image segmentation? I'm new in Artificial Intelligence and I want to do image segmentation.

Searching I have found these ways

Digital image processing (I have read it in this book: Digital Image Processing, 4th edition)
Convolutional neural networks

Is there something else that I can use?

Apart from the multitudes of traditional image segmentation techniques (Watershed, Clustering or Variational methods), newer Segmentation schemes using Deep Learning are actively being used, which provide better results and are better for real-time applications, owing to minimum computation overheads involved.

The following blog provides a detailed review of recent advancements in this field: Review of Deep Learning Algorithms for Image Semantic Segmentation

For the traditional methods, this Wikipedia article provides a nice summary:
Image Segmentation


How to generalise over multiple simultaneous dependent actions in Reinforcement Learning I am trying to build an RL agent to price paid-for-seating on commercial flights. I should reiterate here - I am not talking about the price of the ticket - rather, I am talking about the pricing you see if you click on the seat map to choose where on the plane you sit (exits rows, window seats, etc). The general set up is:

After choosing their flights (for a booking of n people), a customer will view a web page with the available seat types and their prices visible.
They select between zero and n seats from a seat map with a variety of different prices for different seats, to be added to their booking.
The revenue from step 2 is observed as the reward.

Each 'episode' is the selling cycle of one flight. Whether the customer buys a chosen seat or not, the inventory goes down as they still have a ticket for the flight so will get a seat at departure. I would like to change prices on the fly, rather than fix a set of optimal prices throughout the selling cycle.

I have not decided on a general architecture yet. I want to take various booking, flight, and inventory information into account, so I know I will be using function approximation (most likely a neural net) to generalise over the state space.

However, I am less clear on how to set up my action space. I imagine an action would amount to a vector with a price for each different seat type (window seat, exit row, etc). If I have, for example, 8 different seat types, and 10 different price points for each, this gives me a total of 10^8 different actions, many of which will be very similar. In a sense, each action is comprised of a combination of sub-actions - the action of pricing each seat type.

Additionally, each sub-action (pricing one seat type) is somewhat dependent on the others, in the sense that the price of one seat type will likely affect the demand (and hence reward contribution) for another. For example, if you set window seats to a very cheap price, people will be less likely to spend a normal amount for the other seat types. Hence, I doubt the problem can be decomposed into a set of sub-problems.

I'm interested if there has been any research into dealing with a problem like this. Clearly any agent I build needs some way to generalise across actions to some degree, since collecting real data on millions of actions is not possible, even just for one state.

As I see it, this comes down to three questions:

Is it possible to get an agent that can deal with a set of actions (prices) as a single decision?
Is it possible to get this agent to understand actions in relative terms? Say for example, one set of potential prices is [10, 12, 20], for middle seats, aisle seats, and window seats. Can I get my agent to realise that there is a natural ordering there, and that the first two pricing actions are more similar to each other than to the third possible action?
Further to this, is it possible to generalise from this set of actions - could an agent be set up to understand that the set of prices [10, 13, 20] is very similar to the first set?

I haven't been able to find any literature on this, especially relating to the second question - any help would be much appreciated!

If you want to treat the problem as a full Reinforcement Learning problem, I'd recommend to try avoiding the combinatorial explosion of the action space by treating every sub-action as a separate decision point, a separate full action. If you have, for example, already selected 4 sub-actions for a particular customer, you can try to include those in some way in the state representation / input when moving on to the 5th sub-action. By including already-selected sub-actions in the state space, your algorithm can learn to take into account that optimal prices for some seat types will depend on what prices were already selected for others.

I do suspect such a full RL formulation will still be a difficult problem to learn though, will require huge amounts of experience. It may be worth considering to simplify it anyway, treat it as a Contextual + Combinatorial Multi-Armed Bandit Problem. That way you won't be able to learn long-term effects across multiple different customers as you described in the comments, but you will likely at least be able to learn something that works decently well with less experience. Recently, an interesting new book appeared on MAB problems, which is available for free here. You will find many chapters on Contextual MABs there, and also one chapter (chapter 30) on Combinatorial MABs.

Note that with either of these "combinatorial" approaches, you can also try to play around with the order in which you select sub-actions. For example, the sub-action / price point selected for seat type A might have a significant influence on what the optimal remaining policy would be for other seat types, whereas the sub-action for seat type B might have no influence on other seat types. It would then be useful to always prioritize selecting sub-actions for seat type A. You can try to identify these kinds of effects by keeping track of (co)variances in observed returns.

  Is it possible to get an agent that can deal with a set of actions (prices) as a single decision?

The solutions I proposed above do not do this explicitly, they circumvent the issue by taking multiple sequential decisions. There does appear to be some research in RL with vector-valued actions. For example, the paper Clipped Action Policy Gradient briefly mentions vector-valued actions in Subsection 3.2. I am not personally familiar enough with RL + vector-valued actions to make a direct recommendation as to what approaches do or don't work well, but maybe this can at least help you find more relevant literature if this is a direction you'd like to pursue.

  Is it possible to get this agent to understand actions in relative terms? Say for example, one set of potential prices is [10, 12, 20], for middle seats, aisle seats, and window seats. Can I get my agent to realise that there is a natural ordering there, and that the first two pricing actions are more similar to each other than to the third possible action?
  Further to this, is it possible to generalise from this set of actions - could an agent be set up to understand that the set of prices [10, 13, 20] is very similar to the first set?

This kind of generalization should again come naturally from using function approximation with either of the solutions proposed above.


Is there a convention on the order of multiplication of the weights with the inputs in neural nets? Is there a convention on how the input data and the weights are multiplied? The input data can be anything, including the result from the previous layers.
There are two options:
Option 1:
$$\begin{bmatrix}i_1 & i_2\end{bmatrix} \times \begin{bmatrix} w_1 & w_2 & w_3\\w_4 & w_5 & w_6\end{bmatrix} = \begin{bmatrix}i_1*w_1 + i_2*w_4 & i_1*w_2+i_2*w_5 &i_1*w_3+i_2*w_6\end{bmatrix}$$
Option 2:
$$\begin{bmatrix} w_1 & w_4\\ w_2 & w_5\\ w_3 & w_6\end{bmatrix} \times \begin{bmatrix}i_1 \\ i_2\end{bmatrix}  = \begin{bmatrix}i_1*w_1 + i_2*w_4 & i_1*w_2+i_2*w_5 &i_1*w_3+i_2*w_6\end{bmatrix}$$

The conventions I have seen tend to post-multiply rather than pre-multiply, although there are examples in the literature which adopt the opposite convention.
Some examples include:

In Deep Learning: An Introduction for Applied Mathematicians, a layer with input $x \in \mathbb R^n$ and output $f(x) \in \mathbb R^m$ is computed by
$$ f(x) = \sigma(Wx + b)$$
for a matrix $W \in \mathbb R^{m \times n}$ of weights, a vector $b \in \mathbb R^m$ of biases and an activation function $\sigma \colon \mathbb R^m \to \mathbb R^m$.

In Deep Learning by Goodfellow, Bengio and Courville, they compute a layer in chapter 6 (see p. 171) using

$$ f(x) = \max \{ 0, W^T x + c \},$$
where the $\max$ function is applied componentwise and gives the ReLU activation.

In the paper Federated Learning with Matched Averaging, the authors describe a fully-connected layer using pre-multiplication,

$$f(x) = \sigma(xW),$$
where they omit the bias to simplify notation and as before apply $\sigma$ entrywise. In this case of course we need to view $x$ as a row vector and will obtain a row vector as the result.
Ultimately, as long as you are clear it doesn't matter a great deal: in the end, all that changes is the shape and values of the matrix.