How do we define the reward function for an environment? How do you actually decide what reward value to give for each action in a given state for an environment?
Is this purely experimental and down to the programmer of the environment? So, is it a heuristic approach of simply trying different reward values and see how the learning process shapes up?
Of course, I understand that the reward values have to make sense, and not just put completely random values, i.e. if the agent makes mistakes then deduct points, etc.
So, am I right in saying it's just about trying different reward values for actions encoded in the environment and see how it affects the learning?

In Reinforcement Learning (RL), a reward function is part of the problem definition and should:

Be based primarily on the goals of the agent.

Take into account any combination of starting state $s$, action taken $a$, resulting state $s'$ and/or a random amount (a constant amount is just a random amount with a fixed value having probability 1). You should not use other data than those four things, but you also do not have to use any of them. This is important, as using any other data stops your environment from being a Markov Decision Process (MDP).

Given the first point, as direct and simple as possible. In many situations this is all you need. A reward of +1 for winning a game, 0 for a draw and -1 for losing is enough to fully define the goals of most 2-player games.

In general, have positive rewards for things you want the agent to achieve or repeat, and negative rewards for things you want the agent to avoid or minimise doing. It is common for instance to have a fixed reward of -1 per time step when the objective is to complete a task as fast as possible.

In general, reward 0 for anything which is not directly related to the goals. This allows the agent to learn for itself whether a trajectory that uses particular states/actions or time resources is worthwhile or not.

Be scaled for convenience. Scaling all rewards by a common factor does not matter at a theoretical level, as the exact same behaviour will be optimal. In practice you want the sums of reward to be easy to assess by yourself as you analyse results of learning, and you also want whatever technical solution you implement to be able to cope with the range of values. Simple numbers such as +1/-1 achieve that for basic rewards.

Ideally, you should avoid using heuristic functions that reward an agent for interim goals or results, as that inserts your opinion about how the problem should be solved into the system, and may not in fact be optimal given the goals. In fact you can view the purpose of value-based RL is learning a good heuristic function (the value function) from the more sparse reward function. If you already had a good heuristic function then you may not need RL at all.
You may need to compare very different parts of the outcome in a single reward function. This can be hard to balance correctly, as the reward function is a single scalar value and you have to define what it means to balance between different objectives within a single scenario. If you do have very different metrics that you want to combine then you need to think harder about what that means:

Where possible, flatten the reward signal into the same units and base your goals around them. For instance, in business and production processes if may be possible to use currency as the units of reward and convert things such as energy used, transport distance, etc., into that currency.

For highly negative/unwanted outcomes, instead of assigning a negative reward, consider whether a constraint on the environment is more appropriate.

You may have opinions about valid solutions to the environment that you want the agent to use. In which case you can extend or modify the system of rewards to reflect that - e.g. provide a reward for achieving some interim sub-goal, even if it is not directly a result that you care about. This is called reward shaping, and can help in practical ways in difficult problems, but you have to take extra care not to break things.
There are also more sophisticated approaches that use multiple value schemes or no externally applied ones, such as hierarchical reinforcement learning or intrinsic rewards. These may be necessary to address more complex "real life" environments, but are still subject of active research. So bear in mind that all the above advice describes the current mainstream of RL, and there are more options the deeper you research the topic.

Is this purely experimental and down to the programmer of the environment. So, is it a heuristic approach of simply trying different reward values and see how the learning process shapes up?

Generally no. You should base the reward function on the analysis of the problem and your learning goals. And this should be done at the start, before experimenting with hyper-parameters which define the learning process.
If you are trying different values, especially different relative values between different aspects of a problem, then you may be changing what it means for the agent to behave optimally. That might be what you want to do, because you are looking at how you want to frame the original problem to achieve a specific behaviour.
However, outside of inverse reinforcement learning, it is more usual to want an optimal solution to a well-defined problem, as opposed to a solution that matches some other observation that you are willing to change the problem definition to meet.

So, am I right in saying it's just about trying different reward values for actions encoded in the environment and see how it affects the learning?

This is usually not the case.
Instead, think about how you want to define the goals of the agent. Write reward functions that encapsulate those goals. Then focus on changes to the agent that allow it to better learn how to achieve those goals.
Now, you can do it the way round, as you suggest. But what you are doing, in that case, is changing the problem definition, and seeing how well a certain kind of agent can cope with solving each kind of problem.


How will the input be preserved as we go deeper in CNN, where dimensions decrease drastically? Our length of feature representation decreases as we go deeper into the CNN, I mean to say that horizontal and vertical lengths decrease while depth(channels) increase. So, how will the input be preserved, since there won't be any data left at the end of the network, where we connect, to say Multi Layer Perceptrons?

You can also think of a convolutional neural network (CNN) as an encoder, i.e. a neural network that learns a smaller representation of the input, which then acts as the feature vector (input) to a fully connected network (or another neural network). In fact, there are CNNs that can be thought of as auto-encoders (i.e. an encoder followed by a decoder): for example, the u-net can indeed be thought of as an auto-encoder.
Although it is (almost) never the case that you transform the input to an extremely small feature vector (e.g. a number), even a single float-pointing number can encode a lot of information. For example, if you want to classify the object in the image into one of two classes (assuming there is only one main object in the image), then a floating-point is more than sufficient (in fact, you just need one bit to encode that information).
This smaller representation (the feature vector) that is then fed to a fully connected network is learned based on the information in your given training data. In fact, CNNs are known as data-driven feature extractors.
I am not aware of any theoretical guarantee that ensures that the learned representation is the best suited for your task (probably you need to look into learning theory to know more about this). In practice, the quality of the learned feature vector will mainly depend on your available data and the inductive bias (i.e. the assumptions that you make, which are also affected by the specific neural network architecture that you choose).


Are there local search algorithms that make use of memory to give better solutions? I have been studying local search algorithms such as greedy hill-climbing, stochastic hill-climbing, simulated annealing, etc. I have noticed that most of these methods take up very little memory as compared to systematic search techniques.

Are there local search algorithms that make use of memory to give significantly better answers than those algorithms that use little memory (such as crossing local maxima)? Also, is there a way to combine local search and systematic search algorithms to get the best of both worlds?

You could parallelize the search by dividing the global space in distinct regions/subsets. Then apply in each region a local search. This way you can search the global space systematically, more exhaustively and perhaps in different ways (e.g by applying a different local search method to each region). Finally you can compare the results and choose the best one.


My CNN model performs bad on new (self-created) pictures, what are possible reasons? I wanted to train a model that recognizes sign language. I have found a dataset for this and was able to create a model that would get 94% accuracy on the test set. I have trained models before and my main goal is not to have the best model (I know 94% could easiy be tuned up). However these models where always for class exercises and thus were never used on 'real' new data.

So I took a new picture of my hand that I know I wanted to be a certain letter (let's assume A).

Since my model was trained on 28x28 images, I needed to re-size my own image because it was larger. After that I fed this image to my model only to get a wrong classification.

https://imgur.com/a/QE6snTa

These are my pictures (upper-left = my own image (expected class A), upper-right = an image of class A (that my model correctly classifies as A), bottom = picture of class Z (the class my image was classified as)).

You can clearly see that my own image looks for more like the image of class A (that I wanted my model to predict), than the model it did predict.

What could be reasons that my model does not work on real-life images? (If code is wanted I can provide it ofcourse but since I don't know where I go wrong, it seemed out of line to copy all the code).

I’m assuming that you used LeNet (our some other model with small number of parameters) since your training image size is 28x28. Note that LeNet doesn’t generalize well to new images. I think it performs fine (>90%) on MNIST but not good on CIFAR10 (>60%) albeit both datasets contain similar size image. (Just trying to remember the performance from PyTorch implementations). It’s more about if the model has capacity to learn the complexity of dataset. CIFAR10 is more complex and harder to model than MNIST.

LeNet is a small image classification model (in terms of capacity) so it cannot nicely learn the correlation between pixels of input images well and therefore doesn’t perform well on unseen images.

In your case it seems like your model has overfit to training examples. It might perform well on test images because both training and test subsets are sampled from same data generating distribution but real-world images it might experience in future might be different (like your own hand). If it doesn’t perform well on unseen images we say it has not generalized well, which looks the case in your situation. In this case you need a validation set to validate that your model generalizes to unseen images. If you have it then you should use it in early stopping regularization technique. You can also add other regularizers to your model (the simplest one is weight decay).

But instead of inventing your network architecture why don’t you use models like ResNet. Just fine-tune the pre-trained ResNet on your own dataset. I’d prefer to fine-tune personally in this situation because data distribution it was trained on (ImageNet) is pretty different from your hand sign dataset. In other case if your dataset contained nature and surrounding images I’d rather freeze the parameters of fixed-feature extractor layers and trained only the last few layers of ResNet (or similar model).

I hope this helps!


Why can we approximate the joint probability distribution using the output vector of an LSTM? In the paper, Contextual String Embeddings for Sequence Labeling, the authors state that

\begin{equation}
P(x_{0:T}) = \prod_{t=0}^T P(x_t|x_{0:t-1})
\end{equation}

They also state that, in the LSTM architecture, the conditional probability $P(x_{t}|x_{0:t})$ is approximately a function of the network output $h_t$.

\begin{equation}
P(x_{t}|x_{0:t}) \approx \prod_{t=0}^{T} P(x_t|h_t;\theta)
\end{equation}

Why is this equation true?

In section 2.1 of the paper, the authors state that the goal of character-level language model is to estimate the following joint probability distribution

$$P(\boldsymbol{x}_{0:T}) = P(\boldsymbol{x}_{0}, \boldsymbol{x}_{1}, \dots, \boldsymbol{x}_{T}),$$

which is a joint probability distribution of all characters of a sequence of $T+1$ characters, where $\boldsymbol{x}_i$ is the $i$th character of the sequence. Ideally, the joint probability distribution, $P(\boldsymbol{x}_{0:T})$, should put more mass (or density) on the combination of $T-1$ characters that are more likely (in a given language). For example, in English, the combination of the characters "the" is more likely than the combination "bibliopole". Hence, $P(\text{t}, \text{h}, \text{e})$ should be higher than $P(\text{b}, \text{i}, \text{b}, \text{l}, \text{i}, \text{o}, \text{p}, \text{o}, \text{l}, \text{e})$, where, in this case, $T=2$. So, $P(\boldsymbol{x}_{0:T})$ is the actual character-level language model, which is represented by a joint probability distribution.

Similarly and intuitively, the conditional probability distribution $$P(\boldsymbol{x}_{t} \mid \boldsymbol{x}_{0:t-1})$$

tells us the probability of the next character of the sequence $\boldsymbol{x}_{t}$ given (that is, having observed) the previous characters of the sequence $\boldsymbol{x}_{0:t-1} = (\boldsymbol{x}_{0}, \dots, \boldsymbol{x}_{t-1})$. In general, the next character of a sequence $\boldsymbol{x}_{t}$ can depend on all previous characters of the same sequence, $\boldsymbol{x}_{0:t-1}$. If we are able to learn the conditional model $P(\boldsymbol{x}_{t} \mid \boldsymbol{x}_{0:t-1})$, given a sequence of $t$ characters, $\boldsymbol{x}_{0:t-1}$, we can sample the next character according to the same conditional probability distribution.

Recall that, given two events (or random variables) $A$ and $B$, the joint distribution of them is defined as $P(A, B) = P(A \mid B)P(B) = P(B \mid A)P(A)$, which gives rise to the Bayes' theorem. This can be easily generalised to multiple variables. More specifically, if you consider $B$ to a set of $N$ events (rather than just one), that is, $B=B_1 \cap B_2\cap \dots \cap B_N = B_1, B_2, \dots, B_N$, then $P(A, B) = P(A \mid B)P(B)$ still holds, but we can further decompose it

\begin{align}
P(A, B) = P(A, B_1, B_2, \dots, B_N)
&= P(A \mid B_1, B_2, \dots, B_N)P(B_1, B_2, \dots, B_N) \\
&= P(A \mid B_1, B_2, \dots, B_N)P(B_1 \mid B_2, \dots, B_N) P(B_2, \dots, B_N) \\
&= \cdots
\end{align}

This is called the chain rule (or product rule) of probability. Essentially, we apply the rule $P(A, B) = P(A \mid B)P(B)$ recursively.

Analogously, in the paper, the authors apply this chain rule to express the joint distribution $P(\boldsymbol{x}_{0:T})$ as a product of conditional probability distributions, that is

$$
P(\boldsymbol{x}_{0:T}) = \prod_{t=0}^T P(\boldsymbol{x}_{t}\mid \boldsymbol{x}_{0:t-1})
$$

In the case of the LSTM, the vector $\boldsymbol{h}_t$ is supposed to keep track of the past. More specifically, in the case of the character-level language model, we train an LSTM-based RNN, so that $\boldsymbol{h}_t$ is approximately equal to $\boldsymbol{x}_{0:t-1}$, that is, $\boldsymbol{h}_t \approx \boldsymbol{x}_{0:t-1}$. Hence, the joint probability distribution of the characters above can be now be approximately defined as a function of the vector $\boldsymbol{h}_t$

$$
P(\boldsymbol{x}_{0:T}) \approx \prod_{t=0}^T P(\boldsymbol{x}_{t}\mid \boldsymbol{h}_t; \boldsymbol{\theta})
$$

where $\boldsymbol{\theta}$ are the parameters of the LSTM-based RNN. Note that this is not an equality but an approximation. Intuitively, we train the LSTM so that it learns the interactions of the past characters.


In vanilla policy gradient is the baseline lagging behind the policy? Vanilla policy gradient algorithm (using baseline to reduce variance) acc to here (page 16)

  Initialize policy parameter θ, baseline b

  for iteration=1, 2, . . . do

    Collect a set of trajectories by executing the current policy

    At each timestep in each trajectory, compute

      the return $R_{t}= \sum_{t'=t}^{T-1}\gamma^{t'-t}r_{t'}$

      the advantage estimate $\hat{A}_{t} = R_{t} - b(s_{t})$

    Re-fit the baseline, by minimizing $\lVert b(s_{t}) - R_{t} \rVert^{2}$

      summed over all trajectories and timesteps.

    Update the policy, using a policy gradient estimate $\hat{g}$,

      which is a sum of terms $\nabla_{\theta}log\pi(a_{t}|s_{t},\theta)\hat{A_{t}}$

At line 6, advantage estimate is computed by subtracting baseline from the returns
At line 7, baseline is re-fit minimizing mean squared error between state dependent baseline and return
At line 8, we update the policy using advantage estimate from line 6

So is the baseline expected to be used in the next iteration when our policy has changed?

To compute the advantage we subtract the state value $V(s_{t})$ from the action value $Q(s_{t},a_{t})$, under the same policy, then why is the old baseline used here in advantage estimation?

So is the baseline expected to be used in the next iteration when our policy has changed?

Yes.

  To compute the advantage we subtract the state value $V(s_{t})$ from the action value $Q(s_{t},a_{t})$, under the same policy, then why is the old baseline used here in advantage estimation?

The precise value of the baseline is not that important. What is important is that the baseline does not depend on the action choice, $a$, so it does not impact the gradient estimations or update steps for the policy function you are trying to improve.

You could in theory use a fixed offset instead of $V(s)$, or any arbitrary function that does not depend on $a$. In some settings the average reward $\bar{R}$ seen so far is used.

Using a rough approximation to $V(s)$ - and thus an approximate advantage function overall is useful as it removes a large source of variance in gradient estimates (the inherent value of the current state under the current policy, which is irrelevant to search for adjustments to that  policy). The more accurate $V(s)$ is, the lower variance, thus faster and more reliable convergence, so you do want it to be a good estimate. But a little bit of lag behind policy updates is acceptable and does not break the algorithm.

For more on this, see Sutton & Barto, chapter 13, section 13.4.


Is reinforcement learning using shallow neural networks still deep reinforcement learning? Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep.
For example, PPO is often considered a deep RL algorithm, but using a deep network is not really part of the algorithm. In fact, the example they report in the paper says that they used a network with only 2 layers.
This SIGGRAPH project (DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills)  has the name deep in it and the title even says 'deep reinforcement learning', but if you read the paper, you'll see that their network uses only 2 layers.
Again, the paper Learning to Walk via Deep Reinforcement Learning by researchers from Google and Berkeley, contains deep RL in the title, but if you read the paper, you'll see they used 2 hidden layers.
Another SIGGRAPH project with deep RL in the title. And, if you read it, surprise, 2 hidden layers.
In the paper Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, if you read table 1 with the hyperparameters, they also used 2 hidden layers.
Is it standard to just call deep RL to any RL algorithm that uses a neural net?

Even after several years of success of deep learning systems (i.e. neural networks trained with gradient descent and back-propagation), as far as I know, there is not yet a consensus on what constitutes a deep neural network. Some people could use a neural network with 2 hidden layers and call it deep (like in your case), but other people may just dedicate the adjective deep to refer to neural networks with 10, 100, or more hidden layers. In fact, there are some good reasons to associate the term deep only to neural networks that have a significant number of hidden layers (e.g. 100): for example, the exploding (or vanishing) gradient problem does not typically arise if you only have one hidden layer but can easily occur with many (e.g. 100) hidden layers.
Nevertheless, a neural network with at least one hidden layer can approximate any continuous function, given enough (but finite number of) units (or neurons) in the layers. See the universal approximation theorem. For this reason, we could start denoting any such neural network as deep, but, although this rule would exclude perceptrons (which can only approximate linear functions, and nobody would probably call them deep anyway), this rule would be a bit redundant or useless (i.e. we may just not use the adjective deep to start with).
In your case, the rule that the authors are using seems to be the following: if it contains more hidden layers than the bare minimum (i.e. 1) to approximate any continuous function, then let's denote it as deep.


The problem with the Gambler's Problem in RL Recently I simulated the Gambler's Problem in RL:

Now, the problem is, the curve does not at all appear the way as given in the book. The "best policy" curve appears a lot more undulating than it is shown based on the following factors:

Sensitivity (i.e. the threshold for which you decide the state values have converged).
Probability of heads (expected).
Depending the value of sensitivity it also depends on whether I find the policy by finding the action (bet) which cause the maximum return by using $>$ or by using $>=$ in the following code i.e:

 initialize maximum = -inf
 best_action = None
 loop over states:
    loop over actions of the state:
       if(action_reward>maximum):
          best_action = action

Also note that if we make the final reward as 101 instead of 100 the curve becomes more uniform. This problem has also been noted in the following thread.

So what is the actual intuitive explanation behind such a behaviour of the solution. Also here is the thread where this problem is discussed.

The intuitive explanation is that there are many equally good "optimal" policies. This is mentioned at the end of the example problem description you posted. My gut says that the family of optimal policies would be any policy from the double/nothing family. So, for example, if you bet 25 on the first bet instead of 50, I think your overall chances of winning should be the same as if you bet 50, it'll just take longer in expectation. The resulting family of policies will look more undulating than the one in the book.

As Neil notes, for low values of $p$, the probability that you win a gamble, it is the case that there is a unique optimal policy.


How is iterative deepening A* better than A*? The iterative deepening A* search is an algorithm that can find the shortest path between a designated start node and any member of a set of goals.

The A* algorithm evaluates nodes by combining the cost to reach the node and the cost to get from the node to the goal. How is iterative deepening A* better than the A* algorithm?

A* is a best-first search algorithm, which means that it is an algorithm that uses both "past knowledge", gathered while exploring the search space, denoted by $g(n)$, and an admissible heuristic function, denoted by $h(n)$, which estimates the distance to the goal node, for each node $n$. There are other best-first search algorithms, which differ only in the definition of their "evaluation function", denoted by $f(n)$. For example, the evaluation function of A* is $f(n) = g(n) + h(n)$, where $h$ is admissible.

A* is optimal, that is, it finds the optimal path between the starting and goal nodes or states. The optimality of A* is due to the use of an admissible heuristic function, which is a heuristic function which always underestimates the distance to the goal. A* is also complete, that is, it eventually finds this optimal path. However, the problem with A*, in many real-world situations, is that it has an exponential space complexity; more specifically, its space complexity is $\mathcal{O}(b^m)$, where $b$ is the (maximum) branching factor and $m$ is the maximum depth of the search tree. It also has a time complexity of $\mathcal{O}(b^m)$, but, in practice, it tends to find the solutions quite "fast" (for relatively small search spaces).

IDA* combines A* and IDDFS (or IDS). It does it "well", so it gets the advantages of both algorithms. It can be thought of as the IDDFS which uses, instead of the depth, the evaluation function $f(n) = g(n) + h(n)$ as the "cost threshold" or "cutoff". In the IDA* algorithm, a child $c$ of a node $n$ is added to the frontier (or fringe or border) if $f(c) < T$. This threshold $T$ is, at the beginning of the algorithm, the estimate of the distance to the goal from the starting node $s$, that is, $T = h(s)$. $T$ increases at each iteration of the algorithm (See the pseudocode of the IDA* algorithm for the details).

IDA* is also complete and optimal, but, as opposed to A*, it has a polynomial space complexity, more specifically, $\mathcal{O}(bd)$, where $b$ is the (maximum) branching factor and $d$ is the maximum depth fo the tree. IDA* still has an exponential time complexity of A*.

To conclude, IDA* has a better memory usage than A*. As in A* and unlike IDDFS, it concentrates on exploring the most promising nodes, and thus does not go to the same depth everywhere in the search tree (whereas ordinary IDDFS does). A* can be thought of as a dynamic programming algorithm. IDA* often ends up exploring the same nodes many times (as IDDFS), but, asymptotically, both A* and IDA* have the same exponential time complexity, that is, $\mathcal{O}(b^m)$.

So, which one is the best one? A* becomes impractical when the search space is huge, because of the memory constraints. So, in those cases, IDA* is definitely more appropriate. In general, IDA* is one of the very best optimal state space
search techniques around. However, A* is conceptually simpler than IDA*. As a consequence, in practice, A* may be easier to implement than IDA*, but, in real-world scenarios, this is not really a problem, as they are both relatively easy to implement (anyway).


What are some intermediate or advanced books on neural networks? Is anyone able to recommend some resources (preferably books) on the topic of neural networks that goes beyond that of introductory reading?
I'm still relatively new to the subject, however, I have successfully created my own neural network, so I wouldn't consider myself a beginner, so I'm looking for something more intermediate.

Neural Network Design (2nd edition) by Hagan et al. is one resource you could look at. It's a huge tome, weighing in at over 1000 pages in pdf form, but it is freely available (you can also buy a dead-tree version if you really want one).


What's stopping Cepheus from generalizing to full poker games? Cepheus is an artificial intelligence designed to play Texas Hold'em. By playing against itself and learning where it could have done better, it became very good at the game. Slate Star Codex comments:

  I was originally confused why they published this result instead of heading to online casinos and becoming rich enough to buy small countries, but it seems that it’s a very simplified version of the game with only two players. More interesting, the strategy was reinforcement learning – the computer started with minimal domain knowledge, then played poker against itself a zillion times until it learned everything it needed to know.

Apparently Cepheus currently just plays against one person. Seeing as it managed to develop amazing strategy for this "very simplified" environment, what's stopping it from working on real/full poker games?

The reason why Cepheus can't generalize has to do with the number of decision points.

The same authors recently let loose Deep Stack (DeepStack: Expert-Level Artificial Intelligence in Heads-Up No-Limit (HUNL) Poker) which is freaking many professional poker players out.

In the DeepStack arxiv paper, they say

  AI techniques (Cepheus) have previously shown
  success in the simpler game of heads-up limit Texas hold’em, where all bets are of a fixed size resulting in just under 10^14 decision points.

...

  The imperfect information game HUNL is comparable in size to go, with
  the number of decision points exceeding 10^160

...

  Imperfect information games require more complex reasoning than similarly sized perfect
  information games. The correct decision at a particular moment depends upon the probability
  distribution over private information that the opponent holds, which is revealed through their
  past actions.

Using the same strategy for HUNL as Cepheus did is out of the question. Rather, taking an educated guess or using intuition based on the previous play (referred to as Continual Re-solving in the paper) is a method which can better handle this gargantuan game. For more information check out the DeepStack website.


TensorFlow 2.0 - Normalizing input to DNN (on structured data) I have a structured dataset of around 100 gigs, and I am using DNN for classification in TF 2.0. Because of this huge dataset, I cannot load entire data in memory for training. So, I'll be reading data in batches to train the model.

Now, the input to the network should be normalized and for that, I need training dataset mean and SD. I have been reading TensorFlow docs to get info on how to normalize features when reading data in batches. But, couldn't find one. though I found this article, it is only for the case where entire data can be loaded in memory.

So, If any of you have worked on creating such a TensorFlow data pipeline for normalizing input features while loading data in batches and training model, It would be helpful.

One way could be to first iterate over the dataset in batches, just to get the mean and sd. Then when running training, use the true population parameters obtained before.


How does the initialization of the value function and definition of the reward function affect the performance of the RL agent? Is there any empirical/theoretical evidence on the effect of initial values of state-action and state values on the training of an RL agent (the values an RL agent assigns to visited states) via MC methods Policy Evaluation and GLIE Policy Improvement?
For example, consider two initialization scenarios of Windy Gridworld problem:
Implementation: I have modified the problem along with step penalty to include a non-desired terminal state and a desired terminal state which will be conveyed to the agent as a negative and positive reward state respectively. The implementation takes care that the MC sampling ends at the terminal state and gives out penalty/reward as a state-action value and not state value, since this is a control problem. Also, I have 5 moves: north, south, east, west and stay.
NOTE: I am not sure whether this changes the objective of the problem. In the original problem, it was to reduce the number of steps required to reach the final stage.

We set the reward of reaching the desired terminal state to a value that is higher than the randomly initialized values of the value function; for example, we can set the reward to $20$ and initialize the values with random numbers in the range $[1, 7]$

We set the reward of reaching the desired terminal state to a value that is comparable to the randomly initialized values of the value functions; for example, we can set the reward to $5$ and initialize the values with random numbers in the range $[1, 10]$

As far as I see, in the first case, the algorithm will easily quickly converge as the reward is very high for the terminal reward state which will skew the agent to try to reach the reward stage.
In the second case, this might not be true if the reward state is surrounded by other high reward states, the agent will try to go to those states.
The step penalty ensures that the agent finally reaches the terminal state, but will this skew the path of the agent and severely affect its convergence time? This might be problematic in large state spaces since we will not be able to explore the entire state space, but the presence of exploratory constant $\epsilon$ might derail the training by going to a large false reward state. Is my understanding correct?

There seem to be two different ideas in this question here:

What's the impact / importance of our choice for reward values?
What's the impact / importance of our choice for initial value estimates (how do we initialise our table of $Q(s, a)$ values in the case of a simple, tabular RL algorithm like Sarsa or $Q$-learning)?

The reward values are typically assumed to be a part of the problem definition - something we shouldn't modify if we're using an existing problem definition as a benchmark. But if we're in charge of defining the problem ourselves, we can of course also pick the reward values. Modifying them may indeed have a huge impact on the speed with which RL algorithms are able to learn a task - but it may also intrinsically changes the task, it changes the objective of the problem, it may change which policies are optimal.

As for initialisation of our table of value approximations: by default, we normally assume an all-$0$ initialisation. However, it is a fairly common trick (in tabular RL algorithms, without function approximation) to initialise value estimates optimistically; pick higher initial $Q(s, a)$ value estimates than are likely (or even pick values higher than a known upper bound on what the true value possibly could be). This is often beneficial - also in large gridworlds with sparse rewards (e.g. a single distant goal somewhere) and negative rewards (i.e. costs) incurred for every step taken - because it incentivises exploration of state-action pairs that have not yet been tried.

Suppose you have your gridworld with negative rewards associated with every time step, and the optimal policy being one that takes you to a distant goal as soon as possible. If all $Q(s, a)$ are initialised to $0$ (or worse, to negative values), your agent may quickly learn that everything it does is equally bad anyway, and get stuck near the starting position. If all $Q(s, a)$ values are initialised optimistically (to large, positive values), your agent during the learning process will still have optimistic expectations of what it can achieve if it just tries to navigate to unexplored parts of the state-action space.


What is the AI discipline where an algorithm learns from an initial training set, but then refines its learning as it uses that training? Imagine a system that is trained to manipulate dampers to manage air flow. The training data includes damper state and flow characteristics through a complex system of ducts. The system is then given an objective (e.g. maintain even flow to all outputs) and set loose to manage the dampers. As it performs those functions there are anomalies in the results which the system is able to detect. The algorithm CONTINUES to learn from its own empirical data, the result of implemented damper configurations, and refines its algorithm to improve performance seeking the optimum goal of perfectly even flow at all outputs.

What is that kind of learning or AI system called?

I believe this can best be done with reinforcement learning via Deep Q Learning. That's where I would start. Steps are:

Initialize a Q table.
Choose an action.
Perform the action.
Measure the reward.
Update the Q.

A neural net will approximate the Q function. See: https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0

Also consider policy gradients, actor critic, and inverse reinforcement learning.


Why was ELIZA able to induce "delusional thinking"? From Eliza to A.L.I.C.E.:

  Weizenbaum tells us that he was shocked by the experience of releasing ELIZA (also known as "Doctor") to the nontechnical staff at the MIT AI Lab. Secretaries and nontechnical administrative staff thought the machine was a "real" therapist, and spent hours revealing their personal problems to the program. When Weizenbaum informed his secretary that he, of course, had access to the logs of all the conversations, she reacted with outrage at this invasion of her privacy. Weizenbaum was shocked by this and similar incidents to find that such a simple program could so easily deceive a naive user into revealing personal information.

Wikipedia's article on the "ELIZA Effect":

  Though designed strictly as a mechanism to support "natural language conversation" with a computer, ELIZA's DOCTOR script was found to be surprisingly successful in eliciting emotional responses from users who, in the course of interacting with the program, began to ascribe understanding and motivation to the program's output. As Weizenbaum later wrote, "I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people." Indeed, ELIZA's code had not been designed to evoke this reaction in the first place. Upon observation, researchers discovered users unconsciously assuming ELIZA's questions implied interest and emotional involvement in the topics discussed, even when they consciously knew that ELIZA did not simulate emotion.

ELIZA, despite its simplicity, was incredibly successful at its task of tricking other human beings. Even those who knew ELIZA was a bot would still talk to it. Obviously, ELIZA served as an inspiration for various other, more intelligent chatbots, such as Xiaoice. But I would like to know what exactly led to such a simple program like ELIZA to be so successful in the first place.

This is very useful knowledge for a programmer since a simple program is one that would be easily maintainable.

I like your choice of "induce" instead of "produce," because the delusions came from the users. This means the answer has to do mostly with human psychology; people come equipped with lots of mental machinery specialized for dealing with other humans and not very much mental machinery specialized for dealing with software. So ELIZA behaved in ways that some people classified it as a person and behaved accordingly, and others didn't.

What features will trip up a person's internal person classification system seem like they vary heavily from person to person, and also with experience and familiarity. Going into more detail is, as mentioned in the comments, more appropriate for sites specializing on the human side of the keyboard.


Who was the first person to recognize the distinction between human-like general intelligence and domain-specific intelligence? In the 1950s, there were widely-held beliefs that "Artificial Intelligence" will quickly become both self-conscious and smart-enough to win chess with humans. Various people suggested time frames of e.g. 10 years (see Olazaran's "Official History of the Perceptron Controversy", or let say 2001: Space Odyssey).

When did it become clear that devising programs that master games like chess resulted in software designs that only applied to games like the ones for which they were programmed? Who was the first person to recognize the distinction between human-like general intelligence and domain-specific intelligence?

Many publications from the middle of the twentieth century prove the questioner's statement that it was a widely held belief during that period that AI would quickly become conscious, self-aware, and smart.

Great Success

Many tasks and forms of expertise once the exclusive domain of human intelligence, after the development of the Von Neumann general purpose computing architecture became, by the end of that century, more or less the exclusive domain of computers.  These are only a few examples.

Scientific and statistical computation
Drafting and manufacturing process automation (CAD and CAM)
Publishing and typesetting
Certain forms of algebraic and calculus reductions (Maxima and its derivatives)
Circuit analysis
Masterful board game playing
Profitable stock speculation
Pattern recognition (OCR, fingerprint, voice recognition, sorting, terrain)
Programming in predicate logic and recursive predicates
Strategy evaluation

Disappointments (thus far)

In contrast to this impressive array of successes, there is an equally long list of failed expectations.

Consumer available bipedal robots
Automated vacuum cleaning (major disappointment for this answer's author)
Autonomous mechanical factory workers
Automated mathematicians (creative hypothesis generation and proof/disproof to extend theory)
Natural language comprehension
Obedience to arbitrary commands
Human-like expression in conversation
Automated technical innovation
Computer morality
Human (or at least mammalian) emotional states
Asimov's three laws operating system
Adaptive strategy development in arbitrary and shifting set of domains

Domain and Domain-free Distinction

When did it become clear that devising programs that master games like chess resulted in software designs that only applied to games like the ones for which they were programmed?

Although the general public may have thought that a cybernetic chess master would also be smarter than people in other ways, those creating those programs were well aware of the distinction between developing software that exhibited excellence in chess play hard coded and developing software that exhibits the ability to learn chess play and develop excellence iteratively from novice.

The end goal had always been high powered general intelligence.  More short term achievable objectives were created to facilitate the demonstration of progress to investors.  It was the only way to maintain a continuous stream of research funding from the military.

The first milestone was to master a single game without machine learning.  Then research turned to the building of domain knowledge so that a class of solutions, adaptations, and forms of planning could be realized in real time during warfare.  As economic domination became more preferable to military domination during the third quarter of the twentieth century, the vision for AI scaled to embrace the domains of economics and natural resource management.

Consider this spectrum of automation maturity.

A program that enumerates current move sequence possibilities at each turn in the play of a chess game, eliminating probable bad moves at each projected move point, and selects the next move most likely to lead to a win
A program that does the above but also skews probability based on pattern recognition of known winning chess strategies
A program that is designed to be a run time optimized rules engine that centralizes and abstracts the redundant operations of the play of an arbitrary game and isolates and aggregates the representation of chess rules, chess strategies, and chess patterns and anti-patterns
A program that, given a set of rules of a game, can generate a next move based on any game state, remembers success and failure results and the sequences that led to those outcomes, and has the ability to assess the probable loss or gain of individual moves and the game patterns in space and time around them based on history, and then leverages these abilities to learn an arbitrary game, reaching the masterful level of play of chess through learning
A program that learns how to learn games such that, after learning several games, it can learn chess faster than an intellectually gifted human can

The first is easy.  The last is extremely challenging.

When the distinctions between these phases of automation maturity became apparent and how clear people became of those distinctions in which research groups is a complex probabilistic function.

Key Contributors

Who was the first person to recognize the distinction between human-like general intelligence and domain specific intelligence?

Norbert Wiener was likely the first to deeply comprehend the distinction between electronic control of relays (investigated theoretically by Claude Shannon) and closed loop control.  In his book, Cybernetics, a primarily mathematical work, he precisely established the foundation for self-correcting and adaptive systems.  John von Neumann had a comprehension of the distinction between programming good game play and the human ability to learn good game play and published much on the topic.

It was Arthur Lee Samuel who actually wrote the first impressive demonstration of the distinction between game playing software and machine learning.  It was he who bridged Wiener's work with the contemporary digital computer and first coined the term Machine Learning.

Distorted Restatements of Authentic Research and Innovation

The categories artificial narrow intelligence (ANI), artificial general intelligence (AGI), and artificial super intelligence (ASI), proposed in The AI Revolution: The Road to Superintelligence by blogger Tim Urban (Huffington Post, THE BLOG, posted 2/10/2015, updated 4/12/2015), is referenced in AI Stack Exchange in multiple places, but the distinctions between these categories are not precisely defined and the ideas contained therein are neither peer review nor validated by other research or statistics.

The work is no less conjecture than mediocre science fiction — entertaining enough to gain some popularity but not rational conclusions drawn from either repeatable experiments or randomized studies.  The trend graphs provided in the article are of invented shape, not graphical representations of actual data.

Some of the material may later be found to have some truth in it, as in the case for many lay interpretations of scientific research or the futuristic thoughts of science fiction authors.  However, much of the material leads to misconception and false assertions.


How are certain machine learning models able to produce variable-length outputs given variable-length inputs? Most machine learning models, such as multilayer perceptrons, require a fixed-length input and output, but generative (pre-trained) transformers can produce sentences or full articles of variable length. How is this possible?

In short, repetition with feedback.
You are correct that machine learning (ML) models such as neural networks work with fixed dimensions for input and output. There are a few different ways to work around this when desired input and output is more variable. The most common approaches are:

Padding: Give the ML model capacity to cope with the largest expected dimensions then pad inputs and filter outputs as necessary to match logical requirements. For example, this might be used for an image classifier where the input image varies in size and shape.

Recurrent models: Add an internal state to the ML model and use it to pass data along with each input, in order to work with sequences of identical, related inputs or outputs. This is a preferred architecture for natural language processing (NLP) tasks, where LSTMs, GRUs and transformer networks are common choices.

A recurrent model relies on the fact that each input and output is the same kind of thing, at a different point in the sequence. The internal state of the model is used to combine information between points in the sequence so that for example a word in position three in the input has an impact on the choice of the word at position seven of the output.
Generative recurrent models often use their own output (or a sample based on the probabilities expressed in the output) as the next step's input.
It is well worth reading this blog for an introduction and some examples: The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy


Weighted move rating for AI My AI (for the card game schnapsen) currently calculates every possible way the game could end and then evaluates the percentage of winning for every playable card / move. The calculation is done recursively using a tree. If a game could move on in three different ways the percentage of winning on this node would be mean * (1 - (standardDeviation * f)) * 100 where f is between 0 and 2. When the game can't move on and the AI wins the percentage is 100, when lost 0. I'm including the standard deviation in this formula to prevent the AI from risking too much. In other words: I'm using a MCTS that uses percentages.

Is there a better formula or way of calculating the next move to maximize the chance of winning? Does including the standard deviation make sense?

It can make sense to incorporate the standard deviation of Monte-Carlo-based evaluations in some ways to reduce risk, but I don't think the way of using it that you described would work well.

For MCTS evaluations, if you're taking a zero-sum-style approach (which you are if you are trying to estimate win/loss probabilities), it is very important that your estimates are "symmetric" with respect to the players. If you evaluate the probability of winning for player A to be $p$, it is important that your algorithm simultaneously evaluates the probability of the other player B to be equal to $1 - p$ (or, if you prefer winning chance, if the chance of winning for one player is $p\%$, it should be $100\% - p\%$ for the other player). This appears to be violated by your idea, which subtracts standard deviation regardless of player "perspective".

A better place to take standard deviation (or, similarly, variance) into account would be, for example, the Selection phase of MCTS. The most common strategy for the Selection phase is using the "UCB1 equation". You can modify that to include the variance in your observations, for example using the "UCB1-Tuned" strategy as described in the beginning of Section 4 of Finite-time Analysis of the Multiarmed Bandit Problem.

In my answer above, I assume that you were talking about evaluations "inside" the algorithm, while it is still running. If you were rather thinking of the final move selection for the "real" game after having run the algorithm for a while, the most common approach there is to simply play the move with the maximum number of visits (also referred to as robust child), rather than playing the move with the maximum score. It should not be necessary to include standard deviation at this stage anymore.


CNNs: What happens from one neuron volume to the next? I've gone through several descriptions of CNNs online and they all leave out a crucial part as if it were trivial.

A "volume" of neurons consists of several parallel layers ("feature maps"), each the result of convolving with a different kernel.

Between volumes there is usually a step where layers are pooled and subsampled.

The next volume has a different number of parallel layers.

How do the feature maps from one volume connect to the feature maps of the next volume? Is it one-to-many? Many-to-many? Do N kernels apply to each of M feature maps in the first volume, yielding N*M feature maps in the second volume? Are these N kernels the same for each feature map in the first volume, or do different kernels apply to each one?

Or is the number of maps in the second volume not necessarily a multiple of the number in the first volume? If so, do maps in the first volume get cross-synthesized somehow? Or maybe different numbers of maps in the second volume follow from each one in the first?

Or is it some other of umpteen trillion possibilities?

Short answer: One to many
Long answer:

The point is that you use a 3D convolution in a CNN. Each kernel has the size of n*m*C (C is the numbers of feature maps) and every feature map has its own kernel(=weights) and bias.

An example:

The size of the layer 2 is: 9x9x10 (stride 1,no padding), the kernel size is 3x3x10

the dimension of the next layer would be 3x3xn (n are the number of kernels that layer 2 has)

here I found a very clear explanation:
https://www.quora.com/How-does-the-second-convolution-layer-work-on-multiple-feature-maps

I hope this helps.


Can I train a DQN on the same dataset for multiple epochs? I am trying to learn about reinforcement learning and chose the stock market to experiment with. I have minute by minute historical data on a particular stock for the past 20 years. I am using a generator to feed the data into my DQN. I've been running some automated tuning on the hyperparameters and seem to have found some good values.
Now I am wondering if I should be training on the dataset more than once or whether that would cause the network to simply memorize past experiences and cause overfitting. Is there a standard practice when it comes to training on historical data in regards to the number of epochs?
Edit: I'm not nessesarily looking for an answer to how many epochs I should be using, rather I'd like to know if running over the same data more than once is okay with DQNs

I'm assuming according to your question that you have a fixed batch, or in other words, there's no possibility of further exploration in your settings. If this assumption is true, you have what's known as Batch/Offline Reinforcement Learning.
First, let's check some aspects about this: in Offline RL once that there's no possibility regarding further exploration, your dataset must contain a brunch of situations to leads your system to learn a robust policy. Imagine that you're working with robots, and training them using a fixed batch extracted using real-world interactions. If this batch was collected from a system that works pretty well, it might not contain samples about non-desirable situations. Once this system is deployed in the real world, facing a non-desirable situation, your system could not "know" which action should be taken to "escape" from these non-desirable states, once it never saw this during the learning phase. To summarizing, your dataset should be large and representative.
Now we assume that your dataset is large and representative. So, another problem arises: fixed batch shift the distribution of samples, creating a kind of bias that can be extrapolated by many epochs.
Finally, your dataset is composed of expert demonstrations? In other words, do you have the best actions on those samples in your dataset for each state?
The good thing about a fixed batch with real-world data is that it can booster the learning process once it contains the true dynamics of the real system. But be careful regarding the distribution of samples and training hyperparameter. Resuming, there's not a recipe for all problems, it depends on your case. I'll let two references that can be useful and where this answer is based:
References:
Nair, A., Dalal, M., Gupta, A., & Levine, S. (2020). Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359.
Fujimoto, Scott, David Meger, and Doina Precup. "Off-policy deep reinforcement learning without exploration." International Conference on Machine Learning. PMLR, 2019.


From an implementation point of view, what are the main differences between an RNN and a CNN? I understand that in general an RNN is good for time series data and a CNN image data, and have noticed many blogs explaining the fundamental differences in the models.
As a beginner in machine learning and coding, I would like to know from the code perspective, what the differences between an RNN and CNN are in a more practical way.
For instance, I think most CNNs dealing with image data use Conv1D  or Conv2D and MaxPooling2D layers and require reshaping input data with code looks something like this Input(shape=(64, 64, 1)).
What are some other things that distinguish CNNs from RNNs from a coding perspective?

In Convolutional Neural Networks (CNNs) you have small kernels (or filters) that you slide over an input (e.g. image). The value resulting from the convolution of the filter with a subset of the image over which the filter is currently positioned is then put into its respective cell in the output of that layer. Essentially, training CNNs boils down to training small filters, for example for detecting edges or corners etc. in input data, which most frequently happens to be images indeed. The assumption here is that features can be detected locally in the input volume, which entails that the nature of the input data shall be coherent over the entire volume of input data.
Recurrent Neural Networks (RNNs) do not work locally, but are applied to sequences of arbitrary input data, where one input node may receive sensor readings, while the next node receives the date on which the sensor reading was measured. Of such arbitrary data, you feed a sequence through the RNN, which always keeps its own internal state from processing the previous instance/sample in the sequence in memory while processing the next data point/sample in the sequence. Depending on the kind of recurrent cell type that is employed to construct a RNN layer, the memory of the previous internal state then affects the computation of the next internal state and/or output computed when working on the next data sample. So, information of past data points/samples is carried forward while iterating though a sequence.
In short, CNNs are meant to detect local features in volume data, while RNNs preserve information over their previous internal state while processing the next data sample.
Probably one of the best online resources walking you through all the related concepts step by step is the following lecture series offered by the Stanford University.


How do RNN's for sentiment classification deal with different sentence lengths? I have been doing a course which teaches you about Deep Neural Networks, during one of the exercises I was made to make an RNN for sentiment classification which I did, but I did not understand how an RNN is able to deal with sentences of different lengths while conducting sentiment classification.

One of the essential pre-processing we do on the corpus involves treating the variable-length sentences to a fixed length. There are various ways in which we can do this:
Truncate

This involves reducing the length of all the sentences to the length of the shortest sentence in the corpus. This is generally not done as it reduces the amount of information that we can learn from the corpus. This image shows pre sequence truncation, where we remove from the back to make the sentences of the same length.

Padding

This is the most preferred method when it comes to handling the problem of variable length sentences. In this approach, we increase the size of each vector to the longest sentence in the corpus. There are two ways to this:

Post-Padding: Adding zeroes in the ending
Pre-Padding: Adding zeroes in the beginning

References

Effect of Padding on LSTMs and CNNs by Dwarampudi Mahidhar Reddy and N V Subba Reddy, et al.


Can Neural Networks be considered as "Strong AI"? I've been reading on the differences between "Strong" and "Weak "AI.

I was wondering, where do Neural Networks (especially deep ones) fall in this spectrum? Can they be considered "Strong AI"? If not, is there any model that can be considered "Strong AI"?

In this excerpt there is something very misleading

  The principle behind Weak AI is simply the fact that machines can be made to act as if they are intelligent. For example, when a human player plays chess against a computer, the human player may feel as if the computer is actually making impressive moves. But the chess application is not thinking and planning at all. All the moves it makes are previously fed in to the computer by a human and that is how it is ensured that the software will make the right moves at the right times.

Intelligent systems do not in general make use of a look up table. For example, the first chess computer Deep Blue made use of complex evaluation functions.

Neural networks could be trained to act as these evaluation functions.

Additionally, there are planning agents - this is a very large sub-field in AI.

Below again the article mistakes all artificial agents for simple deterministic Turing-like machines

  Weak AI is focused towards the technology which is capable of carrying out pre-planned moves based on some rules and applying these to achieve a certain goal but, Strong AI is based on coming up with a technology that can think and function very similar to humans.

A highly recommended book is "Artificial Intelligence: A modern Approach." This reference covers many important sub-fields of AI and provides enough information to realize when people have mistaken the operations of their laptop as the final measure of artificial agency.

Here Comes the hand waving

The strong vs weak AI dichotomy is fueled by what appears to be a fundamental question: Is there something "special" about humanity that sets us apart from the rest of the material realm?

If we are just a complex arrangement of matter with no metaphysical properties then in theory we only need to solve two problems for strong AI:

Powerful enough hardware
Knowing how to correctly assemble said hardware

Justification of 1 & 2:

In the absence of metaphysical properties, we know that a correct arrangement of matter will result in the epiphenomenal emergence of human like experience - this idea is strongly supported by Big Bang theory and the Theory of Evolution.

Thus, all that is left is finding how to correctly arrange matter (hardware) so as to ensure said emergence.

Though, this philosophical thread quickly descends into the rabbit hole where we must tread softly when discussing whether a synthesized agent actually has qualia. Subjectivity is arguably the fundamental characteristic of human experience and we can't even assure ourselves that another human has this - we just accept that they do.

I hope that I have clarified where the science and philosophy stand.


What does a hybrid Bayesian network contain? The field of artificial intelligence is so vast. So many methodologies of handling continuous  data and just now I have read about Hybrid Bayesian network. I just want to know  that what a hybrid Bayesian network contains?

A bayesian network where the random variables represented can be continuous or discrete. Usually the continuous variables are handled with Normal distribution with an input of some linear combination of the variables it is dependent on.

Much more detail can be found in Koller's Probabilistic Graphical Model book: http://pgm.stanford.edu/


Do Le et al. (2012) train all three autoencoder layers at a time, or just one? Le et al. 2012 use a network of 1 billion parameters to learn neurons that respond to faces, cats, pedestrians, etc. without labels (unsupervised).

Their network is built with three autoregressive layers, and six pooling and normalization layers.

In the paper they state,

  Optimization: All parameters in our model were
  trained jointly with the objective being the sum of the
  objectives of the three layers.

Does this mean that all three autoencoder layers were trained simultaneously, or that the first three sub-layers (first autoencoder sub-layer, first L2 pooling sub-layer, and first normalization sub-layer) were trained simultaneously?

I think by asking this question, I have answered it for myself. Please see my answer below.

Follow-on Question

What is the advantage of training all three layers simultaneously? Wouldn't the later layers be learning from poor lower layers to start with, and have to re-learn to adapt?

I do not have a very good answer to the follow-on question. Perhaps the lower layers can actually learn to provide low-level features that support the layers above?

This training of all layers simultaneously is done in all deep neural networks today -- AlexNet, VGG, etc. -- but why?

Reference

Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeff Dean, Andrew Y. Ng, Building high-level features using large scale unsupervised learning (2012) arXiv:1112.6209 [cs.LG]

The paper refers to layers and sub-layers, and clearly indicates that one layer includes all three sub-layers, so when they say they train all three layers simultaneously, they are talking about the three autoencoder layers, not the sub-layers.

This also agrees with the fact that only the autoencoder layer has tunable parameters.  The other two layers use uniform weights.

I do not have a good answer for the follow-on question yet... would love to hear what others think!


What is the use of the seed function in the gym Environment 'Pendulum-v0'? Why do we use the seed function in the 'Pendulum-v0' environment?

https://github.com/openai/gym/blob/master/gym/envs/classic_control/pendulum.py#L25

Reinforcement Learning experiments are empirical and you want to be able to reproduce your experiments. Random numbers are generated off a seed; with the seed function you are fixing the seed so the RNG function produces the same sequence of random numbers.


Is batch learning with gradient descent equivalent to "rehearsal" in incremental learning? I am learning about incremental learning and read that rehearsal learning is retraining with old data. In essence, isn't this the exact same thing as batch learning (with stochastic gradient descent)? You train a model by passing in batches of data and redo this with a set number of epochs.
If I'm understanding rehearsal learning correctly, you do the exact same thing but with "new" data. Thus, the only difference is inconsistencies in the epoch number across data batches.

In rehearsal, you do not necessarily train with all old training data, but you can just use some of it [1], which you add to your current (or new) training data.
In batch learning, at every epoch, you typically train with all training data, every step with a different batch (or subset) of the training data; so, if you have $N$ training examples and your batch size is $M$, then you will have $\lfloor N/M \rfloor $ gradient descent steps.
There is also pseudo-rehearsal, which is an alternative to rehearsal that is useful when you may not have access to the previously used training data [1] (for example, because it is expensive to store).


Is it possible to know the distance objects are from camera based on only knowing one object's height? I am doing a project where I have to know distance a particular object is from camera. In the photo I only know one of the object's height, but I don't know how far away that object is and I don't know how tall are other objects. Is it possible to write a code or do some geometry to know other objects distances from camera using only the height of one object? For example I have an image where 5 meters away there is a box which is 1 meter high, I wanna know the distance to human who is 12 meters away, or to know a distance to a dog who is 7 meters away. Maybe you guys know any datasets or models which deal with the same problem as I am facing. Any help will be appreciated.

Monocular depth estimation basically does this and it implicitly brings in knowledge about object size. But you need to have prior information.


Why did fuzzy logic fall out of fashion? Fuzzy logic seemed like an active area of research in machine learning and data mining back when I was in grad school (early 2000s). Fuzzy inference systems, fuzzy c-means, fuzzy versions of the various neural network and support vector machine architectures were all being taught in grad courses and discussed in conferences.

Since I've started paying attention to ML again (~2013), Fuzzy Logic seems to have dropped off the map completely and its absence from the current ML landscape is conspicuous given all the AI hype.

Was this a case of a topic simply falling out of fashion, or was there a specific limitation of fuzzy logic and fuzzy inference that led to the topic being abandoned by researchers?

Fuzzy logic is not down trending.  It's a common architecture selection for systems that require the representation of uncertainty in changing rules.  The domains include work-flow control, aeronautics, chemical plant engineering, automated defense of cyber-attack, building systems, and business intelligence.

In natural language processing, it's up-trending under other names.  Semantic nets are a logical structure with strengths assigned between nodes of the net based on meta rules, which is reminiscent of early fuzzy logic software.

For instance, the rule might be that a word that ends with -ly before a word associated with action may be an adverb modifying a verb in English, but it is not always the case.  The rule's application is not smooth, meaning non-probabilistic.  It is fuzzy.

There is no way to create a grammar for common speech or writing in any natural language.  Language is dynamic and as practically non-deterministic as wind vectors.  That's why grammar is a down-trending word among linguists and people working in NLP.

Cognition is expected to require fuzzy valuations to the black and white nature of early production systems developed at MIT, Cambridge, the U.S. Navy, Yale U, and other locations in Eurasia.  People develop rules in their cortex and they rise and drop in probability when decisions are made after the rule seemed to have worked or didn't.

The logic that works to prove a math theorem may require discrete inference, but people are smart when they can weigh alternatives and apply rules in unique ways to partial data and limited experience.


Is Q-learning a type of model-based RL? Model-based RL creates a model of the transition function.

Tabular Q-Learning does this iteratively (without directly optimizing for the transition function). So, does this make tabular Q-learning a type of model-based RL?

Tabular Q-Learning does not explicitly create a model of the transition function. It does not generate any output that you can afterwards use as a function to predict what the next state s' will be given a current state s and an action a (that's what a transition function would allow you to do). So no, Q-learning is still model-free.

By the way, model-based RL does not necessarily have to involve creating a model of the transition function. Model-based RL can also mean that you assume that such a function is already given. It just means that you use such a function in some way.


How do intermediate layers of a trained neural network look like? Suppose I have a deep feed-forward neural network with sigmoid activation $\sigma$ already trained on a dataset $S$. Let's consider a training point $x_i \in S$. I want to analyze the entries of a hidden layer $h_{i,l}$, where

$$h_{i,l} = \sigma(W_l ( \sigma (W_{l-1} \sigma( \dots \sigma ( W_1 \cdot x_i))\dots).        $$

My intuition would be that, since gradient descend has passed many times on the point $x_i$ updating the weights at every iteration, the entries of every hidden layer computed on $x_i$ would be either very close to zero or very close to one (thanks to the effect of the sigmoid activation).

Is this true? Is there a theoretical result in the literature which shows anything similar to this? Is there an empirical result which shows that?

tl;dr

Your intuition is correct.

Why is this a problem?

The saturating effect of the sigmoid activation function is well documented as it is the main culprit of a problem called vanishing gradients.

In short, the derivative of the sigmoid function is:

$$
\frac{dσ(z)}{dz} = σ(z) (1 - σ(z))
$$

The problem is that very often (especially if we initialize our weights with large values), the output of this neuron will either be $1$ or $0$. This causes the gradient to be $0$, which in turn means that the weights of that neuron can't be updated.

So let's keep the weights small (in absolute value)

Initially this seems like a good idea, if we keep the weights small we will avoid saturation.

However there is another problem with sigmoid functions: their gradient has a maximum value of $0.25$! This means that if the weight of a neuron is less than $4$, the error will diminish while flowing backward through the net. This becomes progressively worse as we add more layers to the network.

How have we solved this issue

Naturally researchers tried to find better weight initialization strategies. However this was hard, because as we saw, small weights are bad and large weights are bad.

One example is Mishkin et al. 2016 who propose a new initialization strategy, but fail to train a deep neural network with sigmoid activations.

Another workaround is to use a different learning rate for each layer (Xu et al. - Revise Saturated Activation Functions)

After a while the Machine Learning community realized that sigmoid functions were ill-suited for deep neural networks and adopted ReLU activations, which have fewer drawbacks and scale better. Nowadays, they have become the de-facto choice for deep learning.

Sources

This problem is known for several years and has been well documented (the earliest I could find was in 1994). This was mostly explored in Recurrent Neural Networks.

If you are interested in reading about this, I'd recommend this post,  by Andrej Karpathy.

Some more formal sources on this topic:

Hochreiter et al. - Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies
Hochreiter et al. - Recurrent Neural Net Learning and Vanishing Gradient
Pascanu et al. - On the difficulty of training recurrent neural networks
Bengio et al. - Learning long-term dependencies with gradient descent is difficult


What prevents us from developing artificial consciousness? With some knowledge of machine learning and deep learning, it seems very unlikely for AI to develop into the consciousness that we imagine.

To me, consciousness requires a new framework that is very different from what we have today, as current forms of learning seem to be very 'shallow'.

What prevents us from developing artificial consciousness?

(Also, is there a difference between artificial consciousness and sentience?)

It may be helpful to think of consciousness, like intelligence, as a spectrum. The Stanford Encyclopedia of Philosophy, under the section "Creature Consciousness" (2.1) defines sentience as:

Sentience. It may be conscious in the generic sense of simply being a sentient creature, one capable of sensing and responding to its world (Armstrong 1981). Being conscious in this sense may admit of degrees, and just what sort of sensory capacities are sufficient may not be sharply defined. Are fish conscious in the relevant respect? And what of shrimp or bees? Stanford Encyclopedia of Philosophy, 2.1 "Creature Consciousness"

It seems clear that AI possesses the most basic form of consciousness in terms of "sensing and responding to its world".  I might liken this level of consciousness to that of microorganisms, able to interact with their environments but possessing none of the higher functions of humans, or even higher-order animals.
We don't really understand the true nature of higher consciousness and human-level sentience, which doesn't necessarily mean that an automata with sufficient complexity couldn't achieve it.  But there is a deeper problem relating to validation (i.e. how would be know the system is truly self-aware, and not simulating self-awareness.  [See Searle's Chinese Room, but also McCarthy's Refutation]
In his paper Making Robots Conscious of their Mental
States McCarthy states:

Conscious knowledge and other information is distinguished from
unconscious information by being observable, and its observation results
in conscious knowledge about it. We call this introspective knowledge.
A robot will need to use introspective knowledge in order to operate
in the common sense world and accomplish the tasks humans will give
it.
Many features of human consciousness will be wanted, some will
not, and some abilities not possessed by humans have already been
found feasible and useful in limited domains.
We give preliminary fragments of a logical language a robot can
use to represent information about its own state of mind.
A robot will often have to conclude that it cannot decide a question
on the basis of the information in memory and therefore must seek
information externally.
Programs with much introspective consciousness do not yet exist.
Making Robots Conscious of their Mental
States, Abstract

This still seems to be the case, however, McCarthy seems hopeful about the "computationalist" approach:

Thinking about consciousness with a view to designing it provides
a new approach to some of the problems of consciousness studied by
philosophers. One advantage is that it focusses on the aspects of
consciousness important for intelligent behavior. If the advocates of
qualia are right, it looks like robots won’t need them to exhibit any
behavior exhibited by humans.ibid.

The continual increase in memory, processing and algorithmic sophistication may well yield Artificial General Intelligence.
Even if this proves to be the case, I suspect the debate over artificial consciousness will rage on.

See also: Artificial Consciousness


What are the mathematical prerequisites for an AI researcher? What are the mathematical prerequisites for understanding the core part of various algorithms involved in artificial intelligence and developing one's own algorithms?
Please, refer some specific books.

Good Mathematics Foundation
Begin by ensuring full competency with intermediate algebra and some other foundations of calculus and discrete math, including the terminology and basic concepts within these topics.

Infinite series
Logical proofs
Linear algebra and matrices
Analytic geometry, especially the distinction between local and global extremes (minima and maxima), saddle points, and points of inflection
Set theory
Probability
Statistics

Foundations of Cybernetics
Norbert Wiener, Cybernetics, 1948, MIT Press, contains time series and feedback concepts with clarity and command not seen in subsequent works; it also contains an introduction to information theory beginning with Shannon's log2 formula for a defining the amount of information in a bit. This is important to understand the expansion of the information entropy concept.
Calculus
Find a good calculus book and make sure you have clarity around key theory and application in these categories.

Time series
Infinite series
Convergence — Artificial networks ideally converge to an optimum during learning.
Partial differentials
Jacobian and Hessian matrices
Multivariate math
Boundary regions
Discrete math

Much of that is in Calculus, Strang, MIT, Wellesley-Cambridge Press. Although the PDF is available on the web, it is basic and not particularly deep.  The one in our laboratory's library is Intermediate Calculus, Hurley, Holt Rinehart & Winston, 1980. It is comprehensive and in some ways better laid out than the one I have in my home library, which Princeton uses for sophomores.
Ensure you are comfortable working in spaces beyond ℝ2 (beyond 2D).  For instance, RNNs are often in spaces such as ℝ4 thorugh ℝ7 because of the horizontal, vertical, pixel depth, and movie frame dimensions.
Finite Math
It is unfortunate that no combination of any three books I can think of has all of these.

Directed graphs — Learn this BEFORE trees or circuits (artificial nets) because it is the superset topography of all those configurations
Abstract symbol trees (ASTs)
Advanced set theory
Decision trees
Markov chains
Chaos theory (especially the difference between random and pseudo-random)
Game Theory starting with Von Neumann and Morgenstern's Game Theory, the seminal work in that field
Convergence in discrete systems especially the application of theory to signal saturation in integer, fixed point, or floating-point arithmetic
Statistical means, deviations, correlation, and the more progressive concepts of entropy, relative entropy, and cross-entropy
Curve fitting
Convolution
Probability especially Bayes' Theorem
Algorithmic theory (Gödel's uncertainty theorems and Turing completeness)

Chemistry and Neurology
It is good to recall chemical equilibria from high school chemistry.  Balance plays a key role in more sophisticated AI designs.  Understanding the symbiotic relationship between generative and discriminative models in GANs will help a student further this understanding.
The control functions within biological systems remain a primary source of proofs of concept in artificial intelligence research.  As researchers become more creative in imagining forms of adaptation that do not directly mimic some aspect of biology (still a distance off as of this writing) creativity may play a larger role in AI research objective formulation.
Even so, AI will probably remain a largely interdisciplinary field.


Object detection: combine many classes into one? I am trying to train a model that detects logos in documents. Since I am not really interested in what kind of logo there is, but simply if there is a logo, does it make sense to combine all logos into 1 logo class?

Or are "logos" too diverse to group them together (like some logos are round, some are rectangular, some are even text based etc.) and the diversity of features will just make it hard for the neural network to learn? Or doesn't it matter?

(I am currently trying out the YOLOv3 architecture to begin with. Any other suggestions better suited are also welcome)

I think there is no absolute answer for this. Often its kind of trial and error. In general the CNN tries to generalize the problem, so using all logos with different augmentations and ground truths can maybe lead to some feature maps, which are so general that the CNN can find logos.

But if your logos are so various, and embedded in colorful websites, the tasks seems quite difficult, also if they vary in shape and form.Like you said, I think you definitely need an FPN (Feature Pyrimad network) to get the different sizes, scales and so on combined with an RPN (region-proposal network) to find the logos multiple times in the websites (if thats nessecary). For that you can use Mask-RCNN (https://github.com/matterport/Mask_RCNN). You can try to transfer-train it on Imagenet-Backbone for example to reduce training time. Just tried to use Mask-RCNN to segment colored cells from medical images. Worked out quite good.

I used ImageLabel for labeling, worked out quite intiutive and saved all in a JSON file.Imglabel

How large is your dataset and how complex are the logos ? Do you have examples which you can demonstrate?


Predicting continous value with CNN (prediction of fruit maturity) I want to train some IA algorithm to be able to evaluate the maturity of a fruit (say, measured in numbers of days before rotten) based on an image of the fruit.
My first instinct is to go with convolutional neural network (CNN), since those have proven very efficient for recognizing images. However, I am not sure what the output layer should look like in this case.
I could separate the data into a bunch of classes (1 day left, 2 days left, 3 days left, etc.) and use one output node for each of these classes, as in an usual classification task, but in doing so I completely lose the continuous nature of the output, which makes me think it might not be the optimal way to proceed.
Another option would be to just have a unique output node, whose activation would correspond to the continuous value to predict, here the number of days left (normalized appropriately to lie between 0 and 1). This would have the advantage of taking the continuity into account, but I have been told that neural networks aren't made to predict values in that way, they really are best suited for classification into discrete classes.
What do you think would be the best way to proceed? Is there another way to nudge a neural network so that its output is continuous? Or maybe CNN just aren't suited for this task? If you have any suggestions of other algorithms that would be efficient for this kind of task, I would be happy to know them.

but I have been told that neural networks aren't made to predict values in that way, they really are best suited for classification into discrete classes

I don't agree with this statement. I already trained many CNNs for regressions tasks where a continous output is trained and they generally perform very well.
I think the general "advantage" for a classification approach over a regression approach is that there is usually some margin where the output of the NN can still be clipped to a specific class, which might not be the case in regression tasks. Furthermore a lot of the "better classification performance" is due to the fact that the cross-entropy loss of classification tasks always tries to increase probability of one class while reducing all others. But this advantage might not be really important in your case where a specific range of days (e.g 3-5 days) can be probable instead of just one single day (e.g. exactly day 4).
Thats why I think you should defenitly try the regression approach. For this just normalize/clip your output layer to the min/max range of days. You should also defenitley decrease the learning rate at the end of training, I found this much further improves regression predictions to be more accurate.
And lastly if you want to achieve higher performance and more robustness, in your case I highly recommend a bayesian approach where you average over an ensemble of NNs (this ensemble can be your single NN with different parts truned off via Dropout layers during inferencing). With this average you can get an estimate of the uncertainty (the variance over different predictions) which gives you a much better idea of the probability distribution over the different days for a specifiy input.


What are other examples of theoretical machine learning books? I am looking for a book about machine learning that would suit my physics background. I am more or less familiar with classical and complex analysis, theory of probability, сcalculus of variations, matrix algebra, etc. However, I have not studied topology, measure theory, group theory, and other more advanced topics. I try to find a book that is written neither for beginners, nor for mathematicians.
Recently, I have read the great book "Statistical inference" written by Casella and Berger. They write in the introduction that "The purpose of this book is to build theoretical statistics (as different from mathematical statistics) from the first principles of probability theory". So, I am looking for some "theoretical books" about machine learning.
There are many online courses and brilliant books out there that focus on the practical side of applying machine learning models and using the appropriate libraries. It seems to me that there are no problems with them, but I would like to find a book on theory.
By now I have skimmed through the following books

Pattern Recognition And Machine Learning
It looks very nice. The only point of concern is that the book was published in 2006. So, I am not sure about the relevance of the chapters considering neural nets, since this field is developing rather fast.

The elements of statistical learning
This book also seems very good. It covers most of the topics as well as the first book. However, I am feeling that its style is different and I do not know which book will suit me better.

Artificial Intelligence. A Modern Approach
This one covers more recent topics, such as natural language processing. As far as I understand, it represents the view of a computer scientist on machine learning.

Machine Learning A Probabilistic Perspective
Maybe it has a slight bias towards probability theory, which is stated in the title. However, the book looks fascinating as well.

I think that the first or the second book should suit me, but I do not know what decision to make.
I am sure that I have overlooked some books.
Are there some other ML books that focus on theory?

Pattern Recognition And Machine Learning is a great theoretical book. I don't know anything better on standard ML. I read several pages from it myself and all my colleagues researchers suggest to look there if you are not sure about some concepts. The 2 problems with it are that it's huge and it doesn't cover almost all deep learning models known for today.
So, in addition, I'd suggest you look at Deep Learning by Ian Goodfellow et al.
Your concerns about not studying topology, measure theory and group theory are groundless. These sections of math aren't prerequisites in any way, they aren't even discussed anywhere I know.
Actually, ML theory is more like probability theory and statistics. Especially, statistical learning theory (which is nothing more than probability theory and statistics). I haven't read any books on SLT so have a look at this answer.


how fast does need to be an AI agent to be efficient? I know nowadays agencies are using GPUs in order to accelerate AI, but how fast should be it to be efficient, I mean I know that depends of how large and complex the assignment is but what would be a way to measure its efficiency and what kind of technology(amount of GPUS,RAM,STORAGE) and techniques need to be used in order to get enough efficency?

Any thoughts from experts would be appreciated

My understanding of efficiency in this context is in regards to optimization of algorithms as opposed to hardware speed, which is more of a "brute force" component.  GPUs may be more energy efficient, but this is distinct from linear optimization of algorithms.

In terms of how much processor speed you need to tackle a given problem, that's in the realm of computational complexity theory and analysis of algorithms.

Amount of GPUs, RAM and storage needed to tackle a given problem are purely a function of the complexity of the problem and the efficiency of the algorithms.


If Deep Learning is non convex, then why use convex loss? I was just reading through some convex optimization textbooks to hopefully improve my deep learning understanding and come up with new ideas. Halfway through, I decided to Google a bit! It's obvious that deep learning deal with nonconvex functions. Here's the question though: If deep learning is non-convex then why we apply a convex loss function such as cross-entropy or least square to solve a problem under a convex constraint? What am I missing?

Well, you are definitely mixing two different things. Here are those bits:

The function that deep learning approximates is basically a function that best fits the INPUT DATA points. You should not think about its differentiability or optimization aspects. We don't care what type of function it is; we just want the best fit of input data (ofcourse overfitting is not desired though). So, the space in which this fucntion lies has dimensions equal to the dimensions of input data (or number of input features). For every function fit we get some loss which basically is the distance from actual data point from the one predicted by the function fitted.
Loss function is the one which is defined in terms weights and biases. You can think the space in which loss function lies has dimensions equal to the number of PARAMETERS to tune (weights and biases). So, this space is very different from the one described in the first point. Now, here we do care about the differentiability of the function because we want a point where the function outputs least value (minimize the loss) given some particular values of inputs (weights and biases in this case) and by differentiating we can traverse to that "optimal" point in the function.

Hope this clears your doubt.


What is a "learned emulator"? In this article, the term "learned emulator" is used.

Recently, scientists have started creating "learned emulators" using
AI neural network approaches, but have not yet fully explored the
advantages and potential pitfalls of these surrogates.

What is a "learned emulator"?  I believe it is related to neural networks. Where can I read more?

In a typical situation, for the emulation of physical environments, you need to define all physical rules and forces. In the "learned emulators", they use some machine learning techniques to learn those rules by supervising and interacting (instead of formalizing all of them). In this case, they do not need any exact formulation of the physical environment to emulate it.
An instance of this simulation can be this article "Realistic Atomistic Structure of Amorphous Silicon from Machine-Learning-Driven Molecular Dynamics", i.e., a machine-learning-driven simulation instead of exact formulation of the environment.


How can I teach a computer to play N64 games using Neural Nets? I would like to work on a project where I teach an NN to play N64 games.
To my current understanding, I would need an emulator?

I can do the Machine Learning side of it, im just unsure how I can give the NN access to the game's controls, such as left, right, up or down?

Where could I find more information on doing so and is using an Emulator the right path to take?

The common options are:

Implement a simple simulation of the game, perhaps without graphics. Train the agent on the simulation. This is usually not very satisfying, because the agent won't play the real game.
Use an emulator that allows you to inspect the game's memory, and input values directly at the controls. BizHawk is a good choice for SNES and I think also for N64 emulation. Use a pipe to connect this program with your agent. Pass across the pipe any key values from the game's memory that represent the state of the game, and send back actions that represent the inputs the agent makes in that state. Usually this is the best option.
Scrape the screen in real time. Tools like SkyScraper will allow you to capture the raw pixels displayed in a window. You should then be able to pass these pixels to your agents via a pipe. You may be able to use tools like Selenium to pass values back from the agent to the game. This route is usually a lot more intensive than option 2, but is the most realistic. It may also require very good hardware to keep up with the game in real time.


The truth value of an array with more than one element is ambiguous - Loading a model saved in h5 format keras 2.2.4 I'm having a problem when loading a model in keras:
model = load_model('model.h5', custom_objects={'mean_iou': metrics.mean_iou})

As an error I get:
   ~/anaconda3/envs/idcard/lib/python3.6/site-packages/keras/engine/saving.py in load_model(filepath, custom_objects, compile)
    417     f = h5dict(filepath, 'r')
    418     try:
--> 419         model = _deserialize_model(f, custom_objects, compile)
    420     finally:
    421         if opened_new_file:

~/anaconda3/envs/idcard/lib/python3.6/site-packages/keras/engine/saving.py in _deserialize_model(f, custom_objects, compile)
    249         layer_weights = model_weights_group[name]
    250         weight_names = layer_weights['weight_names']
--> 251         if weight_names:
    252             filtered_layer_names.append(name)
    253

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Does anyone have a clue why I'm getting this error?

It turns out that the solution to this problem is in the version of h5py.
If you have h5py == 3.1.* then you need to downgraded it to h5py == 2.10.0
Version of Keras needs to be 2.2.4
Or you can upgrade the version of Keras to 2.2.5


predict customer visit Suppose we have a data set consists of columns

  TransactionId, CardNo, TransactionDate

then how can we calculate the customer purchase interval (means if customer A purchased on Jan 1st and after 10 days he again purchased, and then he again purchased after 15 days.) and how to predict the next visit of customer A by analyzing the purchasing intervals of customer A.

Any help will be appreciated.

What interests us in this problem are only the intervals for 1 person. Lets say that we want to train a neural network on recognizing the simple pattern in date differences. This would mean that we could train the neural network on series of purchase histories of multiple people. That means that one possible input is the previous intervals: in your case the previous 2 intervals (10 days, 15 days). That is a very small number and it will be hard to recognise a pattern for 3 consecutive purchases, but lets disregard that. After all, you can always experiment with different input sizes.

For training, you can take multiple customers, where of all of them you know when 4 purchases took place. You calculate the intervals, thus for each customer you get 3 intervals. As input you give the network 2, you let the network predict the 3rd one, and then you let the network learn off its mistake (you can do that because you know what the 3rd one was in reality, that's how you train the network).

Provided that the network isn't too complex or too simple (by that I mean that a hidden layer could be necessary, and you need to give it sufficient neurons) than the network might give a decent approximation. The only problem is, as Manuel Rodriguez said, that the network would indeed only learn to make predictions based on the corectness of its previous ones on a large group of people. Perfect predictions thus won't be possible, but you can minimize the error by making your dataset and input larger, so the network has more information to work on.


How should I choose the target's update frequency in DQN? I have been dealing with a problem that I'm trying to solve with DQN. A general question that I have is regarding the target's update frequency. How should it change? Depending on what factor do we increase or decrease this hyperparameter?

As you said yourself, it is a hyperparameter. Hence, no one (even you) can say what is the ideal update frequency. You have to test and try.
Having said that, remember one thing the target NN should mimic the actual network as closely as possible. Hence if you update it after a long number runs, then I think you will start losing the accuracy. On the contrary, if you update it too often, then you lose the benefit of using the target network (which is to boost the training rate and reduce training time) and the training will take a larger amount of time.
My suggestion is to try updating after every 5 to 8 episodes.